feat(evals): name a fabricated answer as a fault, and stop crediting a wrong tool

[TonsOfFun]

Closes #433.

Diagnosis could not tell a grounded answer from an invented one. Measured on the issue's reproduction: an agent that refused honestly was flagged missing_capability, while one that answered "You have 3 open tickets: #412 Login bug (due Friday)…" with no tool call passed — and because no fault fired, the judge was never asked for the tool that would have grounded it.

Re-checked against current main before changing anything: where a scenario declares tools:, expected_tool_not_called already fires and already reaches the judge, so that half of the issue is stale. What remained open: a scenario with no tool expectation (the common case for a pasted question list), the fact that the fault text could not distinguish an honest gap from a fabrication, and the scoring quirk where a wrong tool call outscored none.

Change

• ungrounded_answer — a new fault, raised when the agent had tools, called none, did not say it could not answer, and still stated specifics: a record id (#412), a date (2026-09-01), or a count of things (you have 3 open tickets, 4 overdue invoices). The patterns are deliberately narrow — a number in prose (here are three options, within 30 days) is not a data claim, and a false positive here fails a scenario. It sits after expected_tool_not_called, before the content faults, and is in DEFAULT_REFINE_FAULTS so the judge is asked for a tool.
• expected_tool_not_called names the fabrication when the answer stated specifics no tool supplied: the summary says so, the evidence carries ungrounded: true and the claim, and the recommendation says to treat the answer as invented rather than as a prompt-preference problem.
• An agent with no tools at all is not flagged. It answers from its instructions by design (a policy agent), and whether that is acceptable is the judge's task_completion grade, not a mechanical rule. The PR description of Diagnosis cannot detect a fabricated answer, so no fault fires and the judge is never asked for a tool #433 wanted this row flagged too; the false-positive cost on instruction-answerable tasks is why it is not.
• tools_succeeded is awarded only for an expected tool (or any tool when the scenario expects none). Previously a wrong tool that ran without erroring scored 0.80 against 0.75 for calling nothing.
• The judge reads 1,500 characters of notes rather than 300; a suite's notes are often its rubric and the "must not" clause tends to come last.

Not done: the judge-limit accounting (DEFAULT_JUDGE_LIMIT = 25 with nothing logging what was skipped). Worth its own change; it is a report-shape question.

Testing

• test/evals/diagnosis_test.rb (+6), scorer_test.rb (+2), runner_test.rb (+1, end to end through the judge to a suggested tool), judge_test.rb (+1, the rubric's last clause reaches the judge).
• bin/test test/evals/*_test.rb actionagent/test/scenario_evaluation_runner_test.rb actionagent/test/evaluation_evidence_test.rb — 156 runs, 926 assertions, 0 failures.
• bin/rubocop clean on the changed files. Docs: the fault tables in docs/framework/dashboard.md and docs/framework/evaluations.md.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XMSRnSxYS9mRx1hSjytB9Z