feat(run): derive conversation turns from a heuristic responder

[slowdini]

Closes #257. Part of #244.

What changes for a user

An eval could only drive a multi-turn conversation by scripting it — an ordered
turns array whose author had to predict what the agent would ask, and in what
order. A realistic task against a real codebase is not predictable that way.

An eval can now declare a responder instead, and the runner derives each
follow-up from what the agent just said:

{
  "id": "add-request-caching",
  "prompt": "Requests to the pricing API are slow. Can you add caching?",
  "expected_output": "A working cache with the pricing endpoint under 100ms.",
  "responder": { "type": "heuristic", "max_turns": 8 }
}

$ eval-magic dispatch --iteration 1 --harness codex
[1/2] caching:without_skill:i1-mt2ey9rl-4fd277: completed with 1 responder turn(s)
[2/2] caching:with_skill:i1-mt2ey9rl-4fd277: completed with 1 responder turn(s)

Dispatched 2 task(s): 2 completed, 0 stopped, 0 timed out, 0 failed, 0 skipped

turns and responder are alternatives, not layers; declaring both is a
validation error. Declaring neither still means one-shot dispatch. Retiring
turns is not this PR.

The heuristic

Per #244's decision 3, heuristics only — the LLM answering agent is #258, kept
separate so this loop is proven before a second model enters the attribution
picture.

It reads the round's final assistant message as Markdown. A bullet or
numbered list counts as a question only when the line directly above it ends
with ?:

Which cache should I use?

- An in-process LRU (Recommended)
- Redis

Ending, not merely containing. That distinction is load-bearing and was found by
a test: a closing summary is also mostly a bulleted list, and

Why a decorator? It keeps the call sites untouched. Here is what changed:

- Wrapped the client
- Added the cache

would otherwise have been "answered" — derailing a finished task by replying
"Wrapped the client" to nobody's question.

Given a list, the choice is mechanical:

The list	Recommendation marked	Rule	Chosen
plain (-, *, 1.)	yes	recommended_option	first recommended option
plain	no	first_option	first option
checkboxes (- [ ])	yes	recommended_option	every recommended option
checkboxes	no	no_selection	nothing — None of these.

Plain lists ask for exactly one; checkboxes ask for zero or more. That syntax is
the only signal used to tell them apart. An option is recommended when it carries
a standalone recommended in parens, brackets, or bold — or is a pre-checked
- [x] box. A message asking several questions is answered in one numbered turn.

How a conversation ends

Recorded as	When
completed	The last message asked nothing — the agent is done, so the run stops instead of burning its remaining turns.
stopped / responder_cannot_answer	A question with no option list. The greppable entry point for #258.
stopped / max_turns_reached	Still asking at the bound.

Both responder stops are recorded results — dispatch exits 0 and ingest still
records the run — but both end with the task unfinished, so each is warned
about by name rather than passing silently into the data:

Dispatched 2 task(s): 0 completed, 2 stopped, 0 timed out, 0 failed, 0 skipped
⚠ caching:with_skill:… stopped: the responder could not answer the agent's
  question, so the run ended mid-task. Read the last assistant message under
  its outputs before trusting this data point.

The design is deliberately conservative in one direction: a stray question mark
in an otherwise-finished message stops rather than claiming completion. That
costs a dispatch; the alternative — recording a run as complete while the agent
was still waiting — would cost the result's credibility. Pinned by a test.

Provenance

Every synthesized turn names the rule that produced it and the options it read.
The opening prompt carries no origin at all — that absence is what tells an
authored turn from a derived one.

{
  "type": "user_message", "ordinal": 2, "round": 2,
  "text": "An in-process LRU",
  "origin": {
    "responder": "heuristic",
    "answers": [{
      "question": "Before I start — which cache should I use?",
      "options": ["An in-process LRU (Recommended)", "Redis", "Cache at the CDN"],
      "rule": "recommended_option",
      "chosen": ["An in-process LRU"]
    }]
  }
}

run.json carries the whole ConversationRecord, so this reaches the report with
no extra plumbing; the policy that drove the run is recorded on the task in
dispatch.json. (Recording a responder model in conditions.json is #258's
criterion — there is no model here.)

Cross-harness: no harness-specific code, and none possible

No new descriptor field and no named capability. The responder reads
TranscriptSummary::final_text, which every harness's parser already normalizes,
and replies through the existing {prompt_arg} slot in resume_exec_template.
Any harness that can already run scripted turns runs a responder eval unchanged;
cline can run neither, for the reason its descriptor already documents.

The structured route — Claude Code's AskUserQuestion, whose args
TranscriptEvent::ToolInvocation already captures in full — is not merely
avoidable here but unusable: a dispatch runs headless with stdin detached, so
a tool asking the user has no channel to be answered on, and the runner can only
send free text as the next user turn. Text is the only mechanism that fits the
architecture, and it happens to be the portable one.

What is borrowed from one harness is the convention: (Recommended) and
checkbox lists are how Claude Code's own question UI renders choices. So the
recognized shapes are documented as a harness-neutral contract in
eval-magic docs conversations, not as "what Claude does" — an agent that offers
options that way is answered identically whatever harness runs it, and one that
phrases them differently stops with responder_cannot_answer: a documented gap in
the shape table, not a missing descriptor field. Widening the table is a runner
change that benefits every harness at once.
docs/progressive-enhancements.md records this under "Native conversation
resume", so a harness maintainer reading that section learns they need nothing.

Deliberately not done: telling the agent in its dispatch prompt to mark
recommendations. It would fire the heuristic far more often, and would inject the
very asking-behavior some skills under test are meant to be measured on. #258 is
the real fix.

Adjacent fix: ingest failed on every timed-out task

run-record.schema.json's embedded conversation definition never learned the
timed_out status, timed_out_in_round, or the relaxed events.minItems that
conversation.schema.json gained with per-task timeouts (#256). record_runs
clones the whole ConversationRecord into the run record and validates it there,
so a single hung task failed the whole ingest. Proven by reverting the fix under
the new test:

/conversation/status "timed_out" is not one of "completed" or "stopped"
/conversation Additional properties are not allowed ('timed_out_in_round' was unexpected)

Pre-existing, but responder runs against a real codebase are exactly the ones that
hit a deadline. The run-record copy now mirrors the conversation schema's four
conditional rules.

Schema changes

• evals.schema.json — new definitions/responder (type required, heuristic
  only; optional max_turns, minimum 1, default 8), an eval.responder property,
  and an allOf rejecting responder alongside turns. The clash is also
  checked before the schema, because a bare not reports the whole eval as
  disallowed and never names the two fields — the same reason the codebase rules
  are pre-checked.
• conversation.schema.json — stop_reason gains responder_cannot_answer and
  max_turns_reached; userMessage gains the optional origin object.
• run-record.schema.json — the same two edits to its embedded copy, plus the
  timed_out parity fix above.

Structure

run_task's scripted for loop becomes one loop that asks a TurnPlan for the
next turn, so one-shot, scripted, and responder tasks share a single delivery path
and a single round counter — keeping the delivered_followups + 1 == rounds
identity three ingest sites reconstruct rounds from. The scripted arm keeps
unmet_gate and its joined-assistant-text input exactly as they were.

The driver was heading past 500 code lines, so the new policy is carved out rather
than piled on: conversation.rs (518 code) runs and records rounds,
conversation/turn_plan.rs (141) decides what comes next, and
conversation/responder.rs (214) is the parser. dispatch.rs remains 1054 lines
and was already over 1000 before this PR — the 11 lines added here are struct
fields that cannot be carved from their struct; flagging it as needing its own effort.

CLI and documentation

• New shipped guide: eval-magic docs conversations
  (docs/guides/conversations.md) — the two ways to drive a conversation, the
  recognized question shapes as a contract, what each ending means for the data,
  and the cross-harness position.
• dispatch and init help rewritten around multi-turn rather than "scripted".
• docs/progressive-enhancements.md, docs/developer_overview.md,
  docs/guides/byoh.md, harnesses/template.toml, profiles/shared/runbook.md,
  README.

Goldens: only the four runbook.golden.md files moved — one intended paragraph
each, explaining that a responder stop is recorded but ended mid-task. Nothing
else re-blessed.

Verification

cargo fmt --check                                  clean
cargo build                                        ok
EVAL_MAGIC_REQUIRE_POSIX_TOOLS=1 cargo test        1219 passed, 0 failed  (baseline 1194)
cargo clippy --all-targets -- -D warnings          clean
git diff --check                                   clean

25 new tests, each written first and confirmed failing for the right reason.

Parser (conversation/responder.rs) — a_recommended_option_is_chosen,
a_plain_list_with_no_recommendation_takes_the_first_option,
a_checkbox_list_with_no_recommendation_selects_nothing,
a_checkbox_list_selects_every_recommended_option,
a_pre_checked_box_counts_as_a_recommendation,
a_separator_left_by_a_trailing_marker_is_cleaned_up,
two_option_groups_are_answered_in_order,
a_closing_summary_with_a_bulleted_list_is_not_a_question,
a_stray_question_mark_stops_rather_than_claiming_completion,
a_free_form_question_is_unanswerable, a_message_with_no_question_reads_as_done.

Driver (tests/run/conversation/responder.rs) —
a_responder_eval_answers_a_recommended_option_and_completes,
the_opening_prompt_carries_no_responder_origin,
a_responder_run_that_reaches_max_turns_is_recorded_not_failed,
a_question_the_responder_cannot_classify_stops_the_run,
a_responder_eval_is_rejected_on_a_harness_without_native_resume,
revision_mode_runs_a_responder_eval (Mode B parity).

Config + artifacts — four validation cases, plus
a_responder_record_satisfies_both_schemas_and_roundtrips and
a_timed_out_conversation_satisfies_the_run_record_schema proving the Rust types
and both schemas agree, and
records_a_run_whose_conversation_timed_out_in_a_later_round /
a_responder_task_without_its_completion_artifact_is_skipped_as_incomplete at the
ingest boundary.

Also driven end to end by hand against a stub harness — run → dispatch → ingest
— for the answering path (origin recommended_option reaching run.json) and the
stop path (exit 0, the warning above, responder_cannot_answer in run.json).

Reviewer's attention

responder_cannot_answer is the seam for #258. #258 triggers on exactly this
branch and degrades back to it on failure, so the name is the consequence rather
than the shape — it stays accurate when an LLM responder also cannot answer.

max_turns counts synthesized follow-ups, not rounds. The opening prompt is
not one, which is what keeps it identical to delivered_followups.

Classification happens before the bound. An agent that stops asking on its last
permitted turn completeds; it did not run out of budget.

TaskOutcome::Stopped carries Option<ConversationStopReason>. The schema gate
in write_conversation guarantees a reason exists, but filling an absent one with
an arbitrary variant would mislabel the outcome if that invariant ever moved.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Caup9LtqB1s8gKx9RTkY2c