feat(cli): coder-eval execute + detached grading via evaluate <run_dir>

.claude/commands/coder-eval-review.md

@@ -24,7 +24,8 @@ The run layout (`runs/<run_id>/<variant_id>/<task_id>/<NN>/…`, `<NN>` a zero-p
 1. Read `<run_path>/run.json` if present (for context — `run_id`, `start_time`).
 2. Glob `<run_path>/*/*/*/task.json` and read each one.
 3. Read `<run_path>/analysis.md` if present — it already diagnoses many failures; lean on its findings rather than re-deriving them.
-4. A task counts as **failed** if `final_status != "SUCCESS"` **or** `weighted_score < 0.9`. Skip passing tasks for now (we may extend to passing tasks later — the schema supports it).
+4. Skip any task whose `final_status` is `"NOT_GRADED"` — `coder-eval execute` produced it, no criterion ran, and `weighted_score` is `null`. It is neither a pass nor a failure, and comparing `null < 0.9` would book every ungraded row as a failure to review.
+5. Of the rest, a task counts as **failed** if `final_status != "SUCCESS"` **or** `weighted_score < 0.9`. Skip passing tasks for now (we may extend to passing tasks later — the schema supports it).
 
 If no `task.json` files exist, write an empty `review_index.json` (`{"reviews": []}`) and exit.
 

.claude/shared/run-layout.md

@@ -12,6 +12,9 @@ runs/<run_id>/<variant_id>/<task_id>/<NN>/{task.json, task.log, artifacts/}
 - `<NN>` — zero-padded replicate index (e.g. `00`, `01`).
 - `task.json` — the persisted per-replicate result (the consumer contract; carries the large `iterations` array — still accepted under its former name `turns` when reading, but not what current runs write).
 - `task.json.malformed` — present only on the docker degrade path: when an existing `task.json` fails to parse (schema skew from a stale `:latest` image, or a truncated/torn write), the docker runner moves the unparseable original aside to this sidecar and writes a synthetic `final_status=ERROR` `task.json` in its place. Diagnostic-only; `rglob("task.json")` consumers do not match it.
+- `task.execute.json` — present only after a DETACHED grade (`coder-eval evaluate <run_dir>` or `coder-eval run --resume` over a `NOT_GRADED` row). The pre-grade snapshot of `task.json`, written once and never overwritten by a later grade, so "this run was executed separately from grading" stays auditable. Diagnostic-only; `rglob("task.json")` consumers do not match it.
+- `task.json.graded` — present only after `coder-eval execute --driver docker` refused a container's verdict: the runtime image predated `execute` and graded anyway, so the runner quarantines the graded record here rather than leaving it readable as `task.json`, where a later `--resume` / `aggregate` would fold in exactly the row it declined to publish. Diagnostic-only; `rglob("task.json")` consumers do not match it.
+- `grade.log` — present only after a DETACHED grade over this directory (`coder-eval run --resume`). The grading pass's own log. It is a separate file because the log handler truncates whatever file it opens, so writing to `task.log` would destroy the agent trajectory log the run already paid for.
 - `task.log` — the human-readable task log; `artifacts/` — files the agent produced.
 
 **Scope-marker files** (used to detect what a given path represents):

.github/workflows/verify-published-action.yml

@@ -541,6 +541,18 @@ jobs:
                     "uploaded run dir before re-running; this is an unattended paid job.")
               sys.exit(1)
 
+          # NOT_GRADED means a task ran but was never scored. This job invokes the
+          # published action, which runs `coder-eval run` (graded), so reaching it is
+          # impossible unless the action started dispatching `coder-eval execute` --
+          # in which case every score gate below silently measures nothing and the job
+          # goes green having verified no verdict at all. Hard-fail, don't tolerate.
+          ungraded = [s for s in statuses if s == "NOT_GRADED"]
+          if ungraded:
+              print("::error::task(s) reported NOT_GRADED -- the published action ran without "
+                    "grading. `coder-eval run` always grades, so the action is dispatching the "
+                    "wrong command and every score gate in this job is measuring nothing.")
+              sys.exit(1)
+
           # Exit-contract check, conditional on the model having actually performed.
           # Ignoring the step's exit code entirely (see the continue-on-error rationale
           # above) would also hide a REGRESSION in the action's own exit logic -- e.g. a

docs/REPORT_SCHEMA.md

@@ -23,6 +23,7 @@ read). Times are ISO-8601.
 | --- | --- | --- |
 | `run.json` / `run.md` | `RunSummary` | Every run (and rebuildable via `coder-eval aggregate`) |
 | `<variant>/<task_id>/<NN>/task.json` | `EvaluationResult` | One per replicate |
+| `<variant>/<task_id>/<NN>/task.execute.json` | `EvaluationResult` | Pre-grade snapshot, written once by a detached grade (`evaluate <run_dir>` / `run --resume`). Deliberately **not** matched by `rglob("task.json")`, so it never enters an aggregation. |
 | `<variant>/<suite_id>/suite.json` / `.md` | `SuiteRollup` | Dataset-backed suites only |
 | `experiment.json` / `.md` | `ExperimentResult` | Every run (experiment layer) |
 | `<variant>/variant.json` / `.md` | `VariantAggregate` | Per variant |
@@ -44,7 +45,9 @@ run-level summary; full per-replicate detail lives in each `task.json`.
 | `start_time` / `end_time` | `datetime` | Run window. |
 | `total_duration_seconds` | `float` | Wall-clock. |
 | `tasks_run` | `int` | Total replicates executed. |
-| `tasks_succeeded` / `tasks_failed` / `tasks_error` | `int` | Category counts. **Invariant:** the three sum to `tasks_run`. |
+| `tasks_succeeded` / `tasks_failed` / `tasks_error` | `int` | Category counts. **Invariant:** these three plus `tasks_not_graded` sum to `tasks_run`. |
+| `tasks_measured` | `int` | Rows carrying a criteria verdict (`weighted_score is not None`). NOT a bucket and not part of the task-count invariant: it is the evidence that gates `pass_rate` / `error_share`, because the four category buckets cannot tell a graded `FAILURE` from a `TIMEOUT` no criterion ever saw. Defaults to `0`, inert on pre-`execute` `run.json` (where `tasks_not_graded` is `0`). |
+| `tasks_not_graded` | `int` | Tasks run by `coder-eval execute` — executed, deliberately unscored. Excluded from **both** sides of `pass_rate`. Defaults to `0`, so pre-`execute` `run.json` still parses. |
 | `tasks_token_budget_exceeded` / `tasks_cost_budget_exceeded` | `int` | Sub-counters of `tasks_failed` (not part of the invariant). |
 | `skipped_tasks` | `list[{path, reason}]` | Load failures / `skip: true` opt-outs. |
 | `max_parallel` | `int` | Concurrency used. |
@@ -59,8 +62,9 @@ publishing different numbers for the same run.
 
 | Key | Type | Meaning |
 | --- | --- | --- |
-| `pass_rate` | `float \| None` | `tasks_succeeded / tasks_run` — errors are in the denominator, counted as misses. `None` on an empty run (0/0 is unknown, not 0%). |
-| `error_share` | `float \| None` | `tasks_error / tasks_run`. Diagnostic only; never adjusts the rate. |
+| `pass_rate` | `float \| None` | `tasks_succeeded / tasks_graded` — errors are in the denominator, counted as misses; ungraded tasks are in neither. `None` when the run is empty, **or when no row produced a verdict at all** — an `execute` night whose only non-ungraded rows are crashes was never measured, and reporting `0.0%` there reads as a total failure. |
+| `error_share` | `float \| None` | `tasks_error / tasks_graded`. Diagnostic only; never adjusts the rate. `None` under exactly the same condition as `pass_rate` — a run that measured nothing has no error share either. |
+| `tasks_graded` | `int` | `tasks_run - tasks_not_graded`. The denominator of both rates above. |
 | `total_cost_usd` | `float \| None` | **The bill**: agent + judge + simulator, summed over the rows. `None` when nothing could be priced. |
 | `agent_cost_usd` | `float \| None` | Subject-agent spend alone. The harness-vs-harness comparison figure — judge spend is a property of the suite's criteria and identical across harnesses, so leaving it in would make two harnesses look closer than they are. |
 | `eval_overhead_cost_usd` | `float \| None` | Judge + simulator spend. The other half of `total_cost_usd`. |
@@ -232,7 +236,9 @@ the same weighted armed gate as a native fail),
 ## `variant.json` — `VariantAggregate`
 
 A single aggregate (not wrapped): `variant_id`, `tasks_run`, `tasks_succeeded`,
-`tasks_failed`, `tasks_error` (same sum-to-`tasks_run` invariant), `average_score`,
+`tasks_failed`, `tasks_error`, `tasks_not_graded` (same sum-to-`tasks_run` invariant),
+`average_score` (`float | None` — the mean over **measured** rows: an errored row counts
+as `0.0`, an ungraded one leaves both sides; `null` when nothing was measured),
 `average_duration`, `total_tokens`, `replicate_count`, `tasks_token_budget_exceeded`,
 `tasks_cost_budget_exceeded`.
 
@@ -246,6 +252,11 @@ The cross-variant summary:
   where each `VariantResult` carries `{variant_id, task_id, weighted_score,
   final_status, duration_seconds, total_tokens, iteration_count,
   total_assistant_turns, reference_similarity, replicate_index, replicate_count}`.
+  `weighted_score` is `float | None`: `null` only when every replicate was
+  **ungraded**. An **errored** replicate counts as `0.0` — same rule as
+  `VariantAggregate.average_score` above. That is deliberate: dropping errored
+  rows would let a nightly where one image build failed report a *higher*
+  headline score than a clean one.
 - `variant_aggregates: dict[str, VariantAggregate]` — keyed by variant id.
 - `total_duration_seconds`.
 - `per_replicate_scores: dict[variant_id -> dict[task_id -> list[float]]]`.
@@ -262,8 +273,9 @@ Written for dataset-backed suites; its `passed` flag drives the CI exit code.
 | Key | Type | Meaning |
 | --- | --- | --- |
 | `suite_id` / `variant_id` | `str` | Identity. |
-| `rows_total` / `rows_passed` / `rows_failed` / `rows_error` | `int` | Row counts. |
-| `pass_rate` | `float` | `rows_passed / rows_total`. |
+| `rows_total` / `rows_passed` / `rows_failed` / `rows_error` / `rows_not_graded` | `int` | Row counts. **Invariant:** the four category counts sum to `rows_total`. `rows_not_graded` defaults to `0`. |
+| `pass_rate` | `float \| null` | `rows_passed / rows_graded` — ungraded rows leave both sides, matching `RunSummary.pass_rate`. `null` when nothing was graded (0/0 is unknown, not 0%) **and** when no row produced a verdict, via the same shared `nothing_was_measured` helper the other two rates use. |
+| `rows_graded` | `int` | `rows_total - rows_not_graded`. The denominator above, serialized so a consumer never has to re-derive it. |
 | `average_weighted_score` | `float \| null` | Mean row score. |
 | `criterion_stats` | `list[{criterion_type, rows_evaluated, average_score, error_count}]` | Per-criterion summary. |
 | `failed_samples` | `list[FailedRowSummary]` | Capped at 20 (`{row_id, task_id, final_status, weighted_score, failure_reasons, error_message, task_json_relpath, replicate_index}`). |
@@ -308,10 +320,20 @@ String enum values and their reporting category:
 | `COST_BUDGET_EXCEEDED` | failed | `$` |
 | `ERROR` | error | `!` |
 | `BUILD_FAILED` | error | `B` |
+| `NOT_GRADED` | ungraded | `?` |
 
 > **Gotcha:** `BUILD_FAILED` (a failed Docker image build) categorizes as **error**,
 > not failed — easy to miscount downstream.
 
+`NOT_GRADED` is produced only by [`coder-eval execute`](USER_GUIDE.md#coder-eval-execute--run-without-grading):
+the task ran and its full trajectory was captured, but no criterion was checked, so
+`weighted_score` is `None` (**not** `0.0` — that would be indistinguishable from a task
+that was graded and scored zero). `ungraded` is a fourth reporting category, not a fold
+into one of the other three: counting it as failed would depress every pass rate, and
+counting it as succeeded would invent a verdict. Execution facts still win over it — a
+crash, timeout, or budget breach under `execute` reports `ERROR` / `TIMEOUT` /
+`TOKEN_BUDGET_EXCEEDED` as usual.
+
 `TOKEN_BUDGET_EXCEEDED` and `COST_BUDGET_EXCEEDED` are produced by the cumulative budget caps under
 `run_limits:` (`max_input_tokens` / `max_output_tokens` / `max_total_tokens`, and `max_usd`
 respectively), checked after each completed agent turn — see

docs/TASK_DEFINITION_GUIDE.md

@@ -1544,6 +1544,10 @@ post_run:
 
 Commands run sequentially with `cwd` set to the sandbox directory. stdout and stderr are captured on the `post_run_results` field of the evaluation result (truncated to 100KB each).
 
+**`post_run` runs in the grading phase.** "After evaluation completes" is a phase, not a clock reading, and it matters because these commands may mutate the workspace the criteria read — `rm -rf node_modules` is the usual case. So `coder-eval execute`, which checks no criteria, **defers** them; they run when `coder-eval evaluate <run_dir>` or `coder-eval run --resume` grades the row, and exactly once. Under a plain `coder-eval run` nothing changes: criteria first, then `post_run`, as always.
+
+The consequence worth knowing: a run you `execute` and never grade never runs its `post_run`, so its preserved sandbox keeps whatever the cleanup would have removed.
+
 **Experiment-level defaults:**
 
 Set `defaults.post_run` in an experiment YAML to run cleanup or extraction after every task.