[feature] Extensible stage router known tools

[slopp]

Problem

The stage router currently derives progress and difficulty signals from a built-in vocabulary oriented primarily toward coding agents. It recognizes tools such as file reads, edits, writes, shell commands tests, and planning operations.

Agent harnesses in other domains often expose different tools. For example, Tau3 conversation tasks use tools such as:

• KB_search and get_user_information_by_name for observation
• log_verification and send_payment_request for mutation
• send_message_to_user and start_conversation for protocol progress
• end_conversation for completion

Because the stage router does not recognize these names, they are generally classified as unknown tool activity. The router therefore misses evidence that the agent is observing state, making changes, or progressing through a task.

This has two practical impacts:

1. With a capable-first policy, ambiguous turns disproportionately route to the expensive model.
2. With an efficient-first policy and an LLM fallback, ambiguous turns invoke the fallback judge unnecessarily.

Any non-coding harness with domain-specific tool names—including customer-service, browser, database, and computer-use benchmarks—can lack the signals expected by the current stage-router vocabulary.

Proposed solution

Allow users to extend the stage router’s known tool semantics through additive route configuration.

  [routes.router]
  id = "switchyard/router"
  type = "stage_router"
  capable_target = "capable"
  efficient_target = "efficient"
  picker = "efficient_first"
  confidence_threshold = 0.3

  [routes.router.tool_semantics]
  observe = [
    "KB_search",
    "get_user_information_by_name",
    "get_current_time",
    "get_credit_card_accounts_by_user",
  ]

  mutate = [
    "log_verification",
    "send_payment_request",
    "transfer_to_human_agents",
  ]

  progress = [
    "start_conversation",
    "send_message_to_user",
    "unlock_discoverable_agent_tool",
    "give_discoverable_user_tool",
  ]

  complete = [
    "end_conversation",
  ]

The categories would map into existing stage-router signals:

• observe: count as read/investigation operations.
• mutate: count as write/production operations.
• progress: indicate forward protocol progress without pretending that a read or mutation occurred. This should suppress false stall signals.
• complete: indicate terminal progress and remain separately identifiable for future completion-specific routing policy.

The extension should have the following properties:

• Additive: configured names extend the built-in vocabulary rather than replacing it.
• Case-insensitive exact-name matching by default.
• Built-in semantics retain precedence if a configured name conflicts with a packaged tool.
• Empty or absent configuration preserves current behavior.
• The configuration is scoped to an individual route.

Library-mode construction could expose the same concept as a ToolSemantics value passed into the stage-router configuration:

Alternatives considered

Rely exclusively on the LLM fallback judge

This works functionally but adds latency, cost, and another source of nondeterminism to every ambiguous turn. The fallback judge remains useful for genuinely ambiguous turns, but it should not compensate for tool vocabulary the user can describe deterministically.

Attach semantic hints to tool definitions

Tool schemas could carry routing metadata such as observe, mutate, progress, or complete, allowing semantics to travel with the tool instead of being duplicated in route configuration. This is attractive for frameworks that control tool registration, but tool-schema metadata is not consistently preserved across providers, MCP adapters, and agent harnesses. Route-level configuration is more dependable at the Switchyard boundary and also works when upstream tool definitions cannot be modified.

Improve unknown-tool handling heuristically

The router could treat successful use of a previously unknown tool as evidence of progress, while interpreting repeated calls to the same tool with similar arguments or results as evidence of stalling. This would generalize to new tools without explicit configuration and could be a better hueristic than treating all unknown tools identically. However, repetition is not always failure—pagination, polling, iterative search, and multi-record mutations may legitimately reuse a tool—so explicit semantics remain more predictable and auditable; improved unknown-tool heuristics could complement them as a fallback.

Scope notes

• This is an extension of the existing stage-router request-side signal extraction, rather than a new chain role.

• The primary affected component is the existing request processor that extracts tool-result and progress signals before stage-router tier selection.

• Server configuration parsing and validation would need to accept the new tool_semantics block.

• Library-mode stage-router construction should accept the equivalent configuration.

• It should not require changes to LLMBackend, ResponseProcessor, or ResponseTranslator.

• It does not need to touch a public API listed in switchyard/init.py.all unless ToolSemantics is intentionally promoted as a public library-mode type. A plain mapping accepted by the existing stage-
  router constructor could avoid adding a new public symbol.

• Backward compatibility should be straightforward:

  • An absent or empty tool_semantics block preserves current behavior.
  • Existing built-in classifications retain precedence.
  • Existing route configuration remains valid.
  • Unknown tools continue to behave as they do today.

• Configuration validation should reject a tool name appearing in multiple user-defined categories, or document deterministic precedence if overlap is allowed.

Additional context

The issue arose while evaluating Switchyard as the model endpoint for the LangChain Deep Agents unified evaluation suite, particularly its Tau3 conversation tasks.