feat(run): drive every dispatch from the runner

[slowdini]

Closes #256. Part of #244.

What changes for a user

run still prepares a campaign, but it is no longer the last step before you
start pasting commands. eval-magic dispatch runs the whole plan — one-shot
tasks, scripted conversations, and judges alike.

Before — RUNBOOK.md handed you a pipeline per plan shape, and a third one
for judges:

JOBS=${JOBS:-4}
jq -r '.tasks[] | .eval_root, .dispatch_prompt_path, .outputs_dir' dispatch.json \
  | tr -d '\r' \
  | tr '\n' '\0' \
  | xargs -0 -P "$JOBS" -n 3 sh -c '
    eval_root="$1"; prompt_path="$2"; outputs_dir="$3"
    mkdir -p "$outputs_dir"
    unset GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE …
    codex --ask-for-approval never exec --cd "$eval_root" … \
      > "$outputs_dir/codex-events.jsonl" 2> "$outputs_dir/codex-stderr.log"
  ' sh

After:

$ eval-magic dispatch --iteration 1 --harness codex
[1/4] one-shot:without_skill:i1-mt225rr5-189dc6: completed
[2/4] one-shot:with_skill:i1-mt225rr5-189dc6: completed
[3/4] scripted:with_skill:i1-mt225rr5-189dc6: completed with 1 scripted follow-up turn(s)
[4/4] scripted:without_skill:i1-mt225rr5-189dc6: completed with 1 scripted follow-up turn(s)

Dispatched 4 task(s): 4 completed, 0 stopped, 0 timed out, 0 failed, 0 skipped

Rerun it and finished work is left alone:

Dispatched 4 task(s): 0 completed, 0 stopped, 0 timed out, 0 failed, 4 skipped

The command

eval-magic dispatch [--jobs 4] [--timeout 1800] [--task-index N]… [--overwrite] [--judges]

• --jobs (default 4, matching the recipes' JOBS=${JOBS:-4}) is a bounded
  std::thread::scope pool — no new dependency. Each task already owns a private
  environment, so they are independent.
• --timeout (default 1800s, 0 disables) gives each task a deadline
  spanning all its rounds. An overrun is killed and recorded as a timed_out
  conversation, not a crash.
• Failures are recorded, not fatal. The batch finishes, the summary names
  what failed, and the command exits nonzero. A gate stop (agent_did_not_ask)
  is valid eval data and exits zero.
• Resume is the completion artifact. A task with a conversation.json is
  skipped; a failure writes none, so a plain rerun retries exactly the failures.
  --overwrite redispatches regardless.
• --judges runs the judge tasks ingest emitted, skipping verdicts already
  on disk, printing N/M verdicts present, and exiting nonzero while any are
  missing.

dispatch-task is removed — dispatch --task-index N covers it.

Judges need no judge template

A judge is a one-shot task whose prompt happens to be a rubric, so it reuses the
harness's exec_template with its placeholders bound differently:

Placeholder	Task mode	Judge mode
<eval-root>	the task's private env	the iteration directory
<dispatch_prompt_path>	the task prompt	the judge prompt
<outputs_dir>	outputs/turn-1	<response_path> minus .json, as a directory
{guard_args}	per the run	always empty — judges run outside every guarded env
{model_arg}	the run's agent_model	the task's own model

The per-task capture directory matters: several assertions share one
judge-responses/ directory, so binding captures there would have each judge
overwrite the previous one's transcript.

What this deletes

render_parallel_dispatch_recipe, render_judge_dispatch_recipe, the
parallel_command_template and judge_command_template descriptor fields,
check_judge_command_template, the probe's render-only checks, POSIX_RECIPE_TOOLS,
and require_posix_toolchain. jq is no longer a requirement anywhere, so
POSIX_TOOLING_REQUIREMENT and AFTER_HELP now ask only for a POSIX shell.

Net: +2762 / −2740 across 85 files — a net deletion despite adding the driver,
the judge dispatcher, and 12 tests.

Schema changes

• schema/conversation.schema.json — status gains timed_out; new
  timed_out_in_round; events.minItems relaxed to 1 (a round-1 timeout has
  only the seeded user message), with the two-event floor moved into a
  conditional for completed/stopped.
• schema/harness-descriptor.schema.json — drops the two removed template fields.

CLI changes

• New dispatch subcommand; dispatch-task removed.
• run, ingest, --agent-model, --judge-model, and --harness help
  rewritten around runner-driven dispatch.

Documentation

profiles/shared/runbook.md, dispatch-manifest.md, all four harness
descriptors' next_steps_template / manifest_template, harnesses/template.toml,
README, AGENTS.md, docs/developer_overview.md, docs/progressive-enhancements.md,
docs/guides/byoh.md, docs/guides/isolation.md, and the per-harness notes.

Goldens: runbook, manifest, and next-steps re-blessed; the five judge-recipe*
goldens deleted, along with manifest-noguard / manifest-nomodel (the manifest
no longer renders a conditional recipe, so guard state and model cannot change it)
and the next-steps-model / next-steps-nomodel pair (collapsed to one per
harness, with the test now asserting model-invariance).

Decisions worth a reviewer's attention

Dispatched children get null stdout/stderr rather than inheriting them.
Killing the shell at a deadline leaves the harness grandchild holding the
inherited pipe, which kept the caller blocked ~5s past a 1s timeout (measured:
5.18s → 1.16s). Every shipped exec_template already redirects both into the
outputs directory, so nothing is lost — but a BYOH template that does not
redirect now discards its output instead of leaking it to the terminal.

The flat one-shot transcript path stays. Every task now carries a
conversation_path, so it no longer distinguishes a scripted task; record_runs
keys its "incomplete conversation" skip on turns instead. Routing every task
through the conversation path would also move tool_invocations from the
transcript to conversation.json, which 16 ingest tests do not fabricate.
#266 now records
what removing it involves.

One-shot transcripts move to outputs/turn-1/ — the layout ingest already
read for scripted rounds, so the two-shape branching collapses.

Windows

This is the last PR that has to keep the windows-latest leg green; #275 removes
it. .github/workflows/ci.yml is untouched — the choco install jq step is now
unnecessary but harmless, and removing CI legs is #275's job. No Windows-native
accommodation was added, and the accommodations this rewrite deletes were not
ported forward. Per the #256 comment, posix_shell() stays: harness
exec_templates are still POSIX command lines.

docs/developer_overview.md's Platform-support section now points the remaining
removal at #275 rather than gating it on this ticket.

Verification

cargo fmt --all -- --check                                    clean
cargo build                                                   ok
cargo clippy --all-targets --all-features -- -D warnings      clean
EVAL_MAGIC_REQUIRE_POSIX_TOOLS=1 cargo test --all-targets     1202 passed, 0 failed
git diff --check                                              clean

Run under the flag CI sets on both runners, so no test skips.

New tests, each written first and confirmed failing:

• the_driver_runs_a_task_that_declares_no_scripted_turns — a task with no
  turns leaves a turn-1 transcript and a zero-follow-up completion artifact.
• dispatch_drives_every_task_in_the_plan
• rerunning_dispatch_skips_completed_tasks_and_retries_the_rest — proven with a
  counter the stub appends to per invocation.
• a_failing_task_is_recorded_and_the_rest_of_the_batch_still_runs
• a_task_that_outruns_the_timeout_is_recorded_and_the_batch_finishes
• jobs_runs_tasks_concurrently — four 1s dispatches finish in ~1.3s against a
  4s serial floor.
• revision_mode_dispatches_through_the_same_command — Mode B parity.
• dispatch_judges_runs_missing_verdicts_and_skips_present_ones,
  each_judge_task_captures_its_transcript_separately,
  dispatch_judges_exits_nonzero_while_a_verdict_is_missing,
  a_judge_dispatch_carries_no_guard_arguments
• the_runbook_names_exactly_one_task_dispatch_command
• a_shell_command_* (4) covering the shared timeout-aware spawn helper.

FixtureArgs gains --sleep-ms so a timeout is testable identically on both
runners without a sleep binary.

Also driven end to end by hand against a stub harness:
run → dispatch → ingest → dispatch --judges → finalize → teardown, confirming
codebase/skill_source provenance still reaches run.json and benchmark.json.

🤖 Generated with Claude Code