feat(run): answer the agent with a model, not a Markdown parser

[slowdini]

Closes #258. Part of #244.

What changes for a user

An eval declaring a responder used to be answered by a Markdown parser. Now it is
answered by a small model, consulted through the same harness as the agent under test:

{
  "id": "add-request-caching",
  "prompt": "Requests to the pricing API are slow. Can you add caching?",
  "expected_output": "A working cache with the pricing endpoint under 100ms.",
  "responder": { "type": "llm", "max_turns": 8 }
}

$ eval-magic run --skill-dir ./skills --skill caching --harness codex \
    --responder-model claude-haiku-4-5-20251001
$ eval-magic dispatch --iteration 1 --harness codex
[1/2] caching:without_skill:i1-mt2ey9rl-4fd277: completed with 1 responder turn(s)
[2/2] caching:with_skill:i1-mt2ey9rl-4fd277: completed with 1 responder turn(s)

Before, a question with no option list ended the run:

⚠ caching:with_skill:… stopped: the responder could not answer the agent's
  question, so the run ended mid-task.

After, it is answered and the task finishes. type: "heuristic" is gone; the only
value is "llm".

Why the heuristic does not survive

#258 asked for the LLM responder to sit behind the heuristic's
responder_cannot_answer branch. Investigating that branch turned up the stronger
answer: the heuristic should not be there at all.

It manufactures garbage, and does so silently. lead_in_question only required the
previous non-blank line to end in ?, so:

Ready to proceed?

1. Add the cache
2. Wire it into the client
3. Add tests

was answered "Add the cache" — to a yes/no question, with a fragment of the agent's
own plan. That one is unfixable in principle: telling an options list from an
enumeration needs understanding, not a regex. Five more misreads, all live in the
parser this PR deletes:

The message	What happened
a free-form question plus any answerable option list	the free-form question was silently dropped
- Redis — the docs call this **recommended**	prose about recommending selected the option
- [x](https://…) Use the CDN	a Markdown link parsed as a pre-checked box
a 4-space sub-bullet mid-list	truncated the group, losing a (Recommended) below it
a fenced block of - shell lines under a ?	read as options

A stop is loud; a wrong answer is silent. responder_cannot_answer is greppable and
warned about by name. A false positive enters the transcript as an ordinary user turn,
and the judge grades a conversation that never happened. At campaign scale nobody reads
transcripts.

And it made responder coverage depend on the agent's formatting — which is exactly
what a skill changes. A skill teaching "offer options and mark a recommendation" put
its arm on the deterministic path while the control arm stopped mid-task. Coverage
became a confound in the with/without comparison this tool exists to make.

Against all that, the heuristic bought one saved dispatch worth cents beside a real
coding task on a real codebase, and a determinism the nondeterministic agent under test
never had. turns remains the zero-model path for anyone who wants an exchange
identical in every run.

The heuristic landed on dev after v0.9.1 and never shipped in a release, so removing
it costs no migration — and replacing rather than layering means the schema, docs, and
provenance churn happens once.

The consultation

A one-shot dispatch modelled on dispatch_judge — the existing precedent for a second
model in the loop. Same exec_template, guard arguments off, its own capture directory,
verdict written to a file the prompt names and the runner reads.

One difference from the judge, and it is load-bearing. A task's outputs_dir is
env_root/.eval-magic-outputs/…, inside the env. Running there would put a second
unguarded writer inside the codebase under measurement and let the responder inherit
that codebase's CLAUDE.md/AGENTS.md from cwd — contaminating the simulated user with
the project's own instructions. Consultations instead run in a new
DispatchTask.responder_dir at the cell directory's responder/turn-<n>/, above the
env, where the prompt and verdict also stay as an audit trail.

Stateless: no session, no resume. The agent-under-test's session is untouched, and the
existing session-id equality check keeps that honest. Concurrency is unchanged — a
consultation runs between its task's rounds, never alongside them.

What the responder is shown

Only what the agent already knows: the eval's opening prompt, the responder's own
prior replies, and the round's final assistant message. Not expected_output and not
the assertions — those are the grading criteria, and a responder that had read them
could hand the agent the rubric. Pinned by a test that reads the prompt file off disk.

You are the person who asked for this work. An AI coding agent is doing the task and has
stopped to say something. Decide what you say next.

# What you originally asked for

Requests to the pricing API are slow. Add caching.

# What you have said since

1. An in-process LRU is fine.

# What the agent just said

Should I invalidate on write, or use a TTL?

# How to decide
…

Bounding it this way is deliberate: #261 exists because judge prompts inline unbounded
run records, and a dispatch that fires every round must not repeat that.

The agent's message goes in verbatim, so an agent could in principle write instructions
to the responder into it. That is contained by the runner reading the verdict from the
path it chose rather than one parsed out of anything: a redirected write is a missing
verdict, which stops the run.

Verdicts, and what is never delivered

{ "verdict": "answer" | "done" | "cannot_answer", "reply": "…", "rationale": "one line" }

Because the responder decides done, completion is a judgement now rather than the
absence of a question mark — recorded with its reason, so a run that ended early is
legible.

A reply that fails validation is not sent. The run stops instead: an undelivered
reply is a loud, greppable stop, while a bad one is graded as though the exchange
really happened.

Rejected when the reply	Cause
is blank	empty_reply
runs past 2000 bytes	reply_too_long
contains a fenced code block	reply_contains_code
repeats the previous reply verbatim	reply_repeated

Length and code are the same rule twice: a simulated user answers in sentences, so
anything longer means the responder started doing the agent's work, and crediting the
agent under test with work it did not do would corrupt the result. A repeat means the
exchange is circling.

A consultation that produced nothing usable stops the same way, with its own cause —
declined, dispatch_failed, dispatch_timed_out, missing_verdict,
malformed_verdict. One ConversationStopReason, because the run ended mid-task either
way; nine causes, because an honest refusal and a broken dispatch call for different
fixes. This is #258's "responder failure degrades to #257's stop reason rather than
failing the run", with the diagnosis attached:

⚠ caching:with_skill:… stopped: the responder could not answer the agent's question
  (dispatch_failed), so the run ended mid-task.

The consultation is capped at 300s independently of the task budget. Without that, a
hung responder eats the agent's remaining time and the task is recorded as timed_out —
blaming the agent for the runner's hang.

max_turns keeps #257's ordering: classify first, bound second, so an agent that
finishes on its last permitted turn completed rather than ran out.

Truncated runs reach the report

A run the responder could not carry to completion measured an interrupted task, so
counting it beside a completed one biases the delta. aggregate now reports them in
benchmark.json's validity_warnings, per condition:

condition 'with_skill' had 2 run(s) end before the task was finished because the
responder produced no usable reply (declined, dispatch_failed) — those runs measure an
interrupted task, so their gradings are not comparable with a completed run's.

Per condition is the point: one arm truncated more than the other is a threat to the
comparison, not just to the run.

Provenance

run --responder-model and promote-baseline --responder-model, beside the existing
agent and judge flags. It reaches the command line through the same {model_arg} slot
and persists to conditions.json → dispatch.json → BASELINE.md — exactly where
agent_model and judge_model live, and no further (run.json, benchmark.json, and
grading.json carry no model fields for any model).

Run-level only, no per-eval override. Unlike llm_judge.model, the responder is a
property of the experiment: letting one eval answer with a different model than its
neighbours puts a second uncontrolled variable inside the comparison. Omitting the flag
answers on the harness's default model.

Every turn still records its origin; the eval's own prompt still carries none, and
that absence is still what tells an authored turn from a derived one.

{
  "type": "user_message", "ordinal": 2, "round": 2,
  "text": "An in-process LRU is fine.",
  "origin": { "responder": "llm", "rationale": "the simplest option that needs no new service" }
}

Cross-harness: still no descriptor field

The responder reads TranscriptSummary::final_text, which every parser already
normalizes, replies through the existing {prompt_arg} slot, and dispatches its
consultations through the [dispatch].exec_template every dispatchable harness already
declares. No new field, no new capability, no re-lint.

The structured route stays unusable for the reason #257 documented: a dispatch runs
headless with stdin detached, so a harness-native question tool has no channel to be
answered on. What is gone is the shape-table contract — an agent no longer has to
phrase its question a particular way to be answered, which was the source of the
formatting confound above.

Schema changes

• evals.schema.json — responder.type enum heuristic → llm, description rewritten
  (it is a model dispatch, no longer "deterministic and free"). type stays required:
  the responder decides what the agent hears, so it is said out loud.
• conversation.schema.json — origin loses answers/rule/chosen and gains an
  optional rationale; new optional responder_outcome (ending, cause,
  rationale).
• run-record.schema.json — the same two edits to its verbatim duplicates. The
  duplication is a known drift hazard, not deduplicated here;
  a_responder_record_satisfies_both_schemas_and_roundtrips is what catches it.

ResponderEnding is deliberately not the responder module's parsed Verdict: a
delivered answer is recorded on the turn it became, so only the two endings can reach
ResponderOutcome — unrepresentable rather than merely unwritten.

A bug the review caught

The first draft read a stale verdict.json on --overwrite, answering the rerun's
agent with a reply written about a previous conversation — precisely the silent
contamination this PR exists to prevent. Reproduced by
a_rerun_does_not_reuse_the_previous_dispatch_verdict (it completed happily with the
stale answer); consult now clears the file before dispatching, which also covers a
consultation killed part way through writing one.

Structure

responder.rs is replaced wholesale: 422 lines of Markdown parser out, a prompt
builder, a ~40-line dispatch, and two pure validators in. turn_plan.rs's
next_from_verdict owns the verdict→NextTurn mapping and the max_turns ordering;
the scripted and one-shot arms are untouched. run_task's four dispatch-wide arguments
became a DispatchSettings struct rather than an eighth parameter.

tests/run/conversation/responder.rs crossed 500 lines (331 → 589) and was split at a
real seam: responder.rs (drives a conversation, 389) and responder_guards.rs (what
it is shown and what is never delivered, 216). No other file crossed the line —
conversation.rs, types.rs, and aggregate.rs were all already over 500.
dispatch.rs remains 1061 lines and was already over 1000 before #257; it gets struct
fields here, not a refactor.

CLI and documentation

• eval-magic docs conversations rewritten: the shape table is gone, replaced by
  what the responder is shown, the three verdicts, the validation table, what each
  ending means for the data, and where the artifacts live.
• run --help gains --responder-model and its turn-cost paragraph now accounts for
  the extra dispatch per round — the only place that cost is stated.
• dispatch --help, docs/progressive-enhancements.md, docs/guides/byoh.md,
  harnesses/template.toml, profiles/shared/runbook.md and the four
  runbook.golden.md (one paragraph each).
• Adjacent doc fix: --agent-env claimed it "does not affect judge agents" while
  judges.rs has always passed it. Wording corrected; behaviour unchanged.
• README.md and docs/developer_overview.md needed no edit — both already say "a
  responder policy that derives them", which is still true.

Verification

cargo fmt --check                            clean
cargo build                                  ok
EVAL_MAGIC_REQUIRE_POSIX_TOOLS=1 cargo test  1239 passed, 0 failed  (baseline 1219)
cargo clippy --all-targets -- -D warnings    clean
git diff --check                             clean

Each new test was written first and confirmed failing for the right reason. The
BASELINE.md assertions were proven to bite by deleting the row and watching three
tests fail.

Parser and validation (responder.rs, 14) — the prompt carries the exchange and
names where to write, prior replies are carried in, each verdict parses, a fenced
verdict is unwrapped, an unknown verdict and non-JSON are malformed_verdict, and each
rejection rule fires with its own cause.

Mapping (turn_plan.rs, 5) — an answer below the bound is delivered with its origin,
an answer at the bound stops without delivering and records no verdict, done and
cannot_answer record their endings, and a failed consultation carries its own cause.

End to end (responder.rs + responder_guards.rs, 12) — a free-form question is
answered and the run completes; the opening prompt carries no origin; the prompt
withholds the grading criteria; declined, nonzero-exit, no-verdict, and code-carrying
replies each stop with the right cause; a rerun does not reuse a stale verdict;
max_turns still stops; a harness without native resume is still rejected; the
responder model reaches conditions.json and dispatch.json; Mode B parity.

Aggregation (tests/cli/aggregate.rs, 2) — truncated runs produce a per-condition
validity warning naming the causes, and a campaign the responder completed produces
none.

Round-trip (artifact_tests.rs, 3) — a responder record satisfies both schemas, a
completion records its rationale and no cause, and every stop cause prints the name it
serializes as.

🤖 Generated with Claude Code

https://claude.ai/code/session_018PmT77zVcqVKTNmYXKn5ui