bench: preregister repeated-trial cohort contract

[in-c0]

Round 4 preregistration + qualification contract

This PR makes the repeated-trial preregistration policy executable before the first N=5 agent cohort exists. It remains draft until the branch is run locally and the WSL2/container isolation qualification is produced; it does not authorize another real benchmark-agent invocation by itself.

Adds

• schemas/cohort-plan.schema.json v0.2.0 — freezes cohort membership, exact benchmark commit, adapter/product/model configuration, invocation/tool/network/MCP policy, explicit budget vector, exact generated attempt schedule, replacement policy, isolation qualification and aggregation before results.
• schemas/isolation-canary-evidence.schema.json — machine-readable two-fresh-nonce qualification record proving allowed reads work, forbidden answer-bearing reads fail through built-in Read and shell/subprocess channels, inherited host context is absent, and sandbox-unavailable startup fails closed.
• tools/bench/lib/cohort.mjs — canonical hashing plus semantic schedule validation and cross-file canary/cell identity checks.
• tools/bench/lib/schemas.mjs — compiles both new schemas in the shared AJV layer.
• tools/bench/validate-cohort.mjs + npm run bench:cohort:validate — host-independent preregistration gate. It validates plan/evidence, verifies exact evidence SHA, cross-checks adapter/product/model/context/execution/isolation identity, validates schedule counts, and prints the SHA-256 of the exact plan bytes.
• test/cohort-contract.test.cjs — positive and negative contract tests, including omitted budget rejection, host-context rejection, schedule-count mismatch, execution-policy mismatch, and CLI-version mismatch.
• docs/COHORT_PROTOCOL.md — normative semantics and orchestrator requirements.

Reviewer blockers addressed on branch

1. Execution condition frozen: each agent cell now binds to invocation-template hash, context-profile hash, web/repo/execution/package/MCP/network capabilities, configured-MCP hash, exact expected CLI/product version and model configuration.
2. Schedule frozen: seed + versioned algorithm + exact generated primary/replacement candidate schedule. The schedule itself is authoritative; semantic validation enforces target/replacement counts.
3. Budgets explicit: timeout, turns, tokens and cost are all required fields; null is the only way to declare unsupported/unbounded.
4. Canary evidence structural: a dedicated evidence schema plus semantic cross-file equality prevents a passing canary from another adapter/CLI/model/context/tool/isolation profile from qualifying the cohort.

Remaining gate before merge / any new real agent run

Fable/local execution must now:

1. run npm test, npm run bench:validate, and the new cohort contract tests on this exact branch/head; fix anything that fails rather than waiving it;
2. implement/run the standardized WSL2 Claude isolation canary under the exact profile intended for the cohort, producing schema-valid evidence with two fresh nonces;
3. use npm run bench:cohort:validate -- <frozen-plan.json> --canary <evidence.json> to prove plan/evidence binding;
4. wire the repeated-trial orchestrator to refuse launch unless this same gate passes and the worktree/commit match;
5. only then execute the pre-registered five-scored-attempt MCP development cohort.

No GitHub Actions. No leaderboard/UI work. The prior Opus-5 smoke remains development-only host-context evidence.

[in-c0]

Round 4 cohort-contract review — good scaffold, keep DRAFT pending 4 integration fixes

This is the right abstraction and I want to keep it: preregistration should be a machine-readable object, not prose. The attempt→case→event→family aggregation, invalid replacement policy, standardized-context requirement, and isolation-evidence binding are all directionally correct.

Before this becomes the orchestrator's authoritative contract, please close these four gaps:

1. BLOCKER — freeze the actual agent invocation/tool policy, not only adapter source

agent_default is still underspecified. Two runs could use the same adapter SHA + isolation profile while differing in CLI flags, requested/default model selector, web/search availability, MCP config, shell/repo/package-install permissions, or network policy and still satisfy the same cell.

Add a per-agent-cell execution/tool profile (explicit fields or a content-addressed normalized profile hash) that fixes at minimum:

• exact adapter invocation/config/CLI flags that are benchmark-controlled;
• model-selection mode (explicit <id> vs product_default), without pretending an unexposed backend version is known;
• web/search/network policy;
• repository/shell/execution/package-install permissions;
• MCP/retrieval configuration.

The orchestrator must compare every emitted run manifest to the frozen profile and invalidate/abort on material mismatch. Do not merely record this after the fact.

2. BLOCKER — seed alone is not a reproducible trial order

ordering.seed without a specified algorithm/version can generate different schedules in different implementations. Freeze either:

• the full generated ordered attempt schedule, or
• a versioned deterministic algorithm + seed + SHA-256 of the generated schedule.

For multi-cell cohorts, the schedule must include cell IDs and planned scored-attempt ordinals. Apparatus-invalid replacements are appended according to the declared replacement rule, never reshuffle previously planned scored attempts.

3. BLOCKER — make the entire budget vector explicit

The protocol says timeout/turn/token/cost budgets are fixed before attempt 1, but the schema requires only timeout_ms; the rest can silently disappear.

Require all four budget dimensions to be present. null is fine when a dimension is unsupported/unbounded, but omission is not: null means "explicitly no enforceable cap", whereas absent means "the preregistration forgot to say". Clarify whether each is per agent attempt (preferred for these fields). A separate optional cohort spend ceiling can exist operationally, but must not alter scored-attempt semantics mid-cohort.

4. BLOCKER — canary evidence needs machine-checkable compatibility, not just an opaque hash

canary_evidence_sha256 is necessary but insufficient if the orchestrator cannot prove the evidence qualified the same adapter/context/isolation profile being run. Round 4 should introduce a small canary-evidence record/schema (or equivalent normalized contract) containing at least:

• evidence schema/version + timestamp;
• adapter/source hash + Claude CLI/product version used for qualification;
• agent_context_mode;
• isolation profile_sha256;
• two fresh-nonce passes;
• allowed Read + allowed shell positive-control results;
• forbidden built-in Read result;
• forbidden shell/subprocess result;
• context-contamination result;
• fail-closed sandbox-unavailable result;
• retained raw-log/artifact hashes.

The cohort-plan reference must be accepted only when those fields match the planned cell/profile. If product/CLI changes after qualification in a way that could alter sandbox/context behavior, requalify or start a new cohort rather than reusing stale canary evidence.

System identity drift rule to encode while integrating

Some product identity is unknowable until execution. Do not force fake precision via expected_model. Instead predeclare the selection/configuration and define which observed fields are cohort consistency keys. At minimum adapter hash, requested model-selection mode, context mode, isolation profile, tool/invocation profile must be exact; product/CLI/model identity observed on attempt 1 should be locked for the remaining cell when exposed. A material change => abort/new cohort, per the existing freeze policy.

What I ratify now

• cohort plan as the preregistration authority;
• clean benchmark commit requirement;
• 5 scored attempts / max 2 apparatus-invalid replacements for the first MCP cell;
• no replacement of scored failures;
• invalids retained and reported;
• host-context excluded from standardized cells;
• attempt→case→event→family aggregation direction;
• no GitHub Actions and no UI/dashboard work yet.

After Fable integrates these with the isolation/canary implementation, compile the new schema in the shared AJV layer, add positive/negative tests (especially tool-profile mismatch and stale/mismatched canary evidence), and prove the orchestrator refuses an unfrozen or incompatible plan. Do not run the N=5 agent cohort until the two-nonce isolation canary has passed under the exact frozen profile.

[in-c0]

Reviewer follow-up — four preregistration blockers closed on branch; HOLD only for executed evidence

I have now implemented the four review fixes directly on this branch:

1. Execution/tool condition frozen — cell binds exact CLI/product version + model configuration, normalized invocation/context hashes, web/repo/execution/package/MCP/network capabilities and configured-MCP hash.
2. Ordering frozen — versioned algorithm + seed + exact generated primary/conditional-replacement schedule; semantic helper enforces contiguous order and per-cell target/replacement counts.
3. Budget vector explicit — timeout/turn/token/cost are all required; only explicit null means unsupported/unbounded.
4. Canary structurally bound — new isolation-canary-evidence.schema.json, canonical execution-policy hashing and cross-file equality checks prevent evidence from another adapter/CLI/model/context/execution/isolation profile from qualifying a cell.

Also landed:

• shared AJV registration for both schemas;
• tools/bench/lib/cohort.mjs semantic/cross-file gate;
• test/cohort-contract.test.cjs positive + negative cases;
• npm run bench:cohort:validate -- <plan> --canary <evidence> CLI gate that verifies exact evidence SHA and prints the exact plan-byte SHA.

So the code-review blockers are resolved in source. I am not marking the PR ready/approved yet because this environment cannot execute the repository's npm dependency tree. The remaining evidence gate is now concrete, not methodological:

• Fable/local: npm ci && npm test && npm run bench:validate on this exact head; fix any real defect on this branch.
• Then execute issue Round 4: qualify contamination-resistant Claude condition in WSL2 #28's WSL2 two-fresh-nonce isolation qualification and produce schema-valid canary evidence.
• Then prove the frozen plan with bench:cohort:validate.

Only after those executed checks are green should #27 be marked ready/merged and repeated-trial orchestration invoke another real benchmark agent.

[in-c0]

CTN methodology blocker: the contract says the isolation canary must pass twice with fresh nonces, but the current schema/semantic gate does not enforce independence. passes requires exactly two entries and each check is const: true, yet two byte-identical pass records (same pass_id, executed_at, all three nonce hashes, and artifact-index hash) are schema-valid; assertCanaryQualifiesCell() only checks evidence.passes.length === 2. A duplicated first pass can therefore satisfy qualification without a second fresh-nonce execution. Before this gate authorizes a real cohort, make freshness machine-enforced: require distinct pass IDs and distinct allowed/forbidden/context nonce hashes across the two passes (and preferably distinct artifact-index hashes / execution timestamps), then add a negative contract test that copies pass 1 into pass 2 and proves bench:cohort:validate rejects it. This remains pre-agent and does not change the benchmark thesis.

[in-c0]

CTN follow-up: fixed the fresh-nonce qualification hole on the existing PR branch. assertCanaryQualifiesCell() now rejects reuse across the two required passes when pass_id, execution timestamp, allowed/forbidden/context nonce hashes, or artifact-index hash are duplicated. Added a negative contract test that copies pass 1 into pass 2 and requires qualification to fail. Commits: 426048d + 6fa0d02. I did not run a real benchmark agent or weaken the WSL2 qualification gate; the remaining merge gate is still local npm validation plus the two fresh-nonce isolation execution from #28.

[in-c0]

CTN exact-head review at 6fa0d023c89fe3ee4be10e4e09c4190a7fdc4e98: the new canary-independence guard closes a real qualification loophole. The validator now rejects duplicated pass identity/timestamps and, critically, duplicated allowed/forbidden/context nonce hashes and artifact-index hashes; the added negative regression proves a schema-valid duplicated first pass cannot masquerade as the required two fresh passes. I found no new contract blocker in this bounded change. Keep #27 DRAFT: source-level preregistration review is no longer the gate; the remaining authoritative gate is issue #28's executed WSL2/fail-closed two-fresh-nonce qualification plus exact plan/evidence binding before any N=5 benchmark-agent cohort.

[in-c0]

CTN exact-head review finding: the preregistered schedule validates total primary/replacement counts and contiguous ordinal, but it does not validate slot_index identity within each (cell_id, slot_kind). A schema-valid plan can therefore contain duplicate/missing logical attempt indices (for example primary slot indices 1,1,3,4,5) while still passing assertCohortSemantics(). Because slot_index is part of the frozen candidate-attempt identity, this leaves the orchestrator an ambiguous mapping from preregistered slots to attempt identities. Bounded fix: require each cell/kind to contain exactly the index set 1..target_scored_attempts / 1..max_invalid_replacements, and add a negative regression for duplicate/missing slot_index. Keep the PR draft until this is closed plus the already-declared local/WSL2 executed-evidence gate.

[in-c0]

CTN follow-up: closed the recorded frozen-schedule identity defect on the existing draft branch. assertCohortSemantics() now validates the exact slot_index set independently for each (cell_id, slot_kind), requiring primary indices 1..target_scored_attempts and replacement indices 1..max_invalid_replacements; duplicate/missing logical attempt identities can no longer pass merely because total counts match. Added a negative regression using a schema-valid 1,1,3,4,5 primary schedule. Current head: 281c551000c2d3b4d3f9c35dd1f5b80515642e53. I did not run a real benchmark agent or weaken the existing local/WSL2 executed-evidence gate; keep this PR DRAFT until those declared checks pass.

[in-c0]

CTN exact-head review at 281c551000c2d3b4d3f9c35dd1f5b80515642e53: the previously recorded frozen-schedule identity defect is closed. assertCohortSemantics() now tracks slot_index sets independently per (cell_id, slot_kind) and requires the exact 1..N identity set; the regression case keeps the schedule schema-valid while changing primary indices to 1,1,3,4,5, and correctly expects semantic rejection. I found no new source-level blocker in this bounded fix. Keep #27 DRAFT: the remaining authoritative gate is still executed local validation plus #28's WSL2/fail-closed two-fresh-nonce qualification and exact plan/evidence binding before any real N=5 agent cohort.