No way to bind host MCP tool implementations for execution — scenario evals score 0 and agents fabricate tool results

[TonsOfFun]

The engine can attribute a host application's MCP tools but cannot execute them, so scenario evaluations against any agent whose tools come from the host always score 0.

The gap

AgentExecutionService#execute_tool routes every call through the engine's own toolbox:

AgentToolbox.call(name, **kwargs)

Agent::AVAILABLE_TOOLS is:

terminal playwright filesystem code database slack fetch search edit translate memory agents

A host whose agent uses its own MCP tools gets an empty intersection. In our install (Sparkle → Clara):

agent.tools & ActionAgent::Agent::AVAILABLE_TOOLS  # => []

agent.tools holds 14 real tool names (healthcheck, count_records, find_records, …) served by the host's MCP registries.

Consequence — worse than a zero score

Duplicating the observed agent to make it executable (as require_executable_scenario_agent! suggests) does not help. The run succeeds with no tools bound, so the model fabricates the answer. A scenario asking Clara to run a healthcheck produced:

"Running healthcheck... Here's the summary: Database connection: Healthy, Cache: Healthy, Search service: Healthy…"

with tool_calls: [] and zero tool spans. Nothing touched the host. A confidently invented answer is a worse failure mode than an error, and it scores as expected_tool_not_called, which reads as an agent-configuration problem rather than an engine limitation.

Request

An execution-time counterpart to the existing ActionAgent.mcp_catalog registration — that one supplies attribution metadata (which server a tool belongs to), and it works well; the report correctly labels missing tools by server. What's missing is a way to register callable implementations so execute_tool can dispatch to the host:

ActionAgent.register_tool_executor("sparkle-diagnostic") do |name, **kwargs|
  MCP::Diagnostic.call(name, **kwargs)
end

Shape is up to you — per-server executor, a resolver object, or extending mcp_catalog entries with a callable. The requirement is that AgentExecutionService consult host-registered executors before falling back to AgentToolbox, and that a declared-but-unresolvable tool fail loudly instead of letting the model answer without it.

Interim behaviour worth fixing regardless

Independent of the feature: when an agent declares tools that resolve to nothing, the run should error rather than proceed tool-less. That alone would have turned a fabricated healthcheck into an actionable failure.

Workaround

Scoring recorded interactions (EvaluationRunnerService with rule + telemetry criteria) works today and needs no tool binding — that's what we've adopted meanwhile. It measures the agent as it actually ran, but cannot test new prompts.

[TonsOfFun]

Narrowed and superseded by #419.

Re-reading the code, the callback shape proposed here is not needed: the engine already persists Agent#mcp_servers, already normalises its stored shapes in EvaluationToolResolver, already enumerates a server's tools in ToolDiscovery, and already has a working Streamable-HTTP MCP client in PlaywrightMCPClient. The only missing piece is that AgentExecutionService#execute_tool never consults any of it before falling through to AgentToolbox.

So rather than a host registering executor lambdas, the host declares mcp_servers on the agent — which the builder already writes — and the engine dispatches to them. #419 carries that, plus failing loudly on a tool that resolves to nothing, and dropping the observed-agent execution ban once tools can bind.

(Generated using Claude Code)