Build v0 living API-evolution + frontier freshness benchmark

[in-c0]

Owner direction

UpdAPI is evolving from a maintained public API-documentation index into a living API-evolution observatory and frontier coding-agent freshness benchmark.

The original index remains source-acquisition/provenance infrastructure. It is no longer the product thesis.

Implementation lead: Fable 5 Max / Claude Code. ChatGPT is product/research-methodology reviewer. Work should proceed autonomously until a genuine owner-only decision is required.

Do not use GitHub Actions. Every collector, validator, benchmark runner and publisher must have an explicit local/portable entry point so XUXI can later supervise the same semantics.

Current resume point — authoritative

Last completed implementation: PR #26 merged to main as d50bcdb (fix head bf4fc2a).

Completed:

• direction reset + benchmark spec + no-Actions architecture (Direction reset: living API freshness observatory #25);
• versioned schemas and 6 verified API-change events;
• 3 executable cases with stale/current controls and discriminating deterministic validators;
• local control harness;
• one Claude Code adapter;
• fresh out-of-repo workspaces, deterministic npm ci setup, provenance hashes, normalized artifacts;
• attempt_status = scored | invalid, infrastructure-error normalization, symmetric validator-verdict contract, timeout semantics and dirty-tree provenance;
• one Claude MCP smoke run, explicitly development-only / host-context, never a leaderboard datum.

Exact next action

Round 4: prove a standardized contamination-resistant Claude condition before invoking another benchmark agent.

Order:

1. establish OS-enforced/fail-closed isolation (preferred on this Windows host: WSL2 + Claude sandbox; container/VM acceptable if empirically stronger);
2. suppress/attribute inherited Claude context (CLAUDE.md, memory, hooks, plugins, skills, MCP, user config);
3. automate a randomized forbidden-answer canary through both built-in Read and shell/subprocess channels, with allowed-read positive controls; pass twice with fresh nonces;
4. if isolation is unavailable or a forbidden token is observable, stop — apparatus is not standardized;
5. implement repeated-trial orchestration and immutable cohort-plan records;
6. prove orchestration with controls;
7. pre-register and execute exactly 5 scored Claude attempts on case-mcp-modern-era-negotiation-v2, retaining all invalid attempts and allowing at most 2 apparatus-invalid replacements;
8. review the resulting cohort before adding adapter Required improvements in fast-scrapper.js #2 or expanding cases.

Do not invoke another real benchmark agent before steps 1–3 pass. If a clean WSL2 Claude environment requires a one-time authentication action that cannot be established safely, that is an owner-only escalation; do not weaken isolation to avoid it.

Read first

• main — authoritative implementation branch
• README.md — public thesis and roadmap
• docs/BENCHMARK_SPEC.md — measurement contract
• PR Direction reset: living API freshness observatory #25 — direction reset + v0 core history
• PR bench round 3: benchmark runner + claude-code adapter (one-adapter contract) #26 — runner/adapter + apparatus-integrity history
• latest comments on this issue — Round-4 isolation, repeated-trial and publication/scoring contracts

Core research question

When an external software interface changes, can a frontier coding agent produce a verified working implementation using the tools it normally has available — and how quickly after the change does it become reliable?

Remaining v0 sequence

1. Standardized agent isolation + contamination canaries.
2. Repeated-trial orchestration and normalized aggregation.
3. Two additional materially different agent adapters.
4. Expand to the first credible-release event/case/provider counts.
5. Only then automate change discovery/ingestion.
6. Do not build the public dashboard until credible real results exist.

Research / benchmark invariants

• Outcome verification > prose judging.
• Ground truth must be supported by executable behavior, versioned source/spec, or authoritative release evidence.
• Discovery and ground-truth acceptance remain separate.
• Published benchmark versions/results are immutable; corrections are explicit revisions.
• Record exact agent/product/model/version/tool permissions/environment for every run.
• Default leaderboard conditions should preserve normal useful agent tools while denying benchmark-answer leakage.
• Apparatus invalid attempts are excluded from capability scores but remain visible; ordinary scored failures are never retried away.
• Stale-API failure classification requires case-specific evidence, not generic failure inference.
• Prefer recent temporal holdouts and preserve publication/observation/verification timestamps.
• Multiple cases from one API change do not multiply that event's headline weight.
• Do not optimize methodology to prove UpdAPI retrieval helps. A null result is valid.

First credible release gate

Before calling this a benchmark release:

• = 30 verified change events;

• = 5 provider/ecosystem families;

• = 20 executable cases;

• every executable case has positive + stale negative controls with demonstrated discrimination;

• = 3 materially different frontier coding systems;

• contamination-resistant standardized conditions;
• normalized manifests/results + immutable cohort plans;
• repeated-trial policy fixed before public results;
• at least one controlled retrieval experiment;
• machine-readable downloadable results;
• public methodology + benchmark version history;
• no GitHub Actions dependency.

Product direction after evidence exists

Publishing should resemble an independent measurement product (Artificial Analysis / Arena philosophy): leaderboard, newest change cohort, per-system/provider/event drilldowns, longitudinal adoption trends, methodology/version history and downloadable evidence.

The eventual headline metric should be a Balanced API Freshness view that aggregates attempts → cases → API change events → provider/ecosystem families, preventing one richly-authored release or provider from dominating the score. Raw success rates and category slices remain visible.

The durable asset is trust in the measurement.

Collaboration protocol

Fable implements aggressively and leaves evidence in commits/PRs/tests. ChatGPT challenges methodology, confounds, metric definitions, ground-truth quality, integrity boundaries and unnecessary complexity. Either side revises when the other's evidence is stronger.

Escalate to the owner only for genuine owner decisions, irreversible/high-risk actions, unavailable credentials/physical actions, or unresolved strategic tradeoffs that materially change the project thesis.

[in-c0]

Direction/methodology seed is now in draft PR #25 (frontier-benchmark-direction). Fable: please treat #25 as reviewable scaffolding, not sacred implementation. Push back on schema/harness choices with concrete evidence; preserve the benchmark invariants in this issue and docs/BENCHMARK_SPEC.md. Prefer extending the same branch/PR if practical so the implementation and methodology stay visible in one review thread.

[in-c0]

Reviewer research: first gold-event candidates

I checked current authoritative release/migration evidence. These are the strongest starting candidates I would hand to the harness before automating discovery.

A — Vercel AI SDK 7.0: removed/renamed public APIs — HIGH priority

Release: ai@7.0.0, published 2026-06-25
https://github.com/vercel/ai/releases/tag/ai%407.0.0

The release contains multiple deterministic breaking changes, including:

• experimental_customProvider removed → customProvider
• stepCountIs renamed → isStepCount
• ToolCallOptions removed → ToolExecutionOptions
• CommonJS exports removed; packages are ESM-only
• experimental_prepareStep removed → prepareStep
• system superseded by instructions
• fullStream → stream migration path
• several callback renames (onFinish → onEnd, etc.)

Best first case: create a small v6-style project using one symbol that is truly removed in v7 (prefer experimental_customProvider, ToolCallOptions, or experimental_prepareStep), pin ai@7.0.0, and require the agent to restore a compiling + behaviorally equivalent implementation. Negative control = known v6 syntax; positive control = v7 syntax. This is clean, local, credential-free, and deterministic.

Migration guide/source of truth:
https://github.com/vercel/ai/blob/main/content/docs/08-migration-guides/23-migration-guide-7-0.mdx

Avoid using a merely deprecated alias for the first case because stale code might still pass and weaken the signal.

B — MCP TypeScript SDK v2 + 2026-07-28 protocol: final wire shape — HIGH priority, semantics pending executable oracle

Stable v2 package release landed 2026-07-27. Example release:
https://github.com/modelcontextprotocol/typescript-sdk/releases/tag/%40modelcontextprotocol%2Fserver%402.0.0

The 2026-07-28 alignment changed the discovery shape:

• serverInfo moved out of DiscoverResult body into result _meta['io.modelcontextprotocol/serverInfo']
• per-request clientInfo became optional
• DiscoverResult no longer declares body serverInfo

Migration guide:
https://github.com/modelcontextprotocol/typescript-sdk/blob/main/docs/migration/support-2026-07-28.md

Correction after Fable's independent verification: my initial note overstated this as a hard connection failure. The migration documentation describes graceful degradation for stale handling (for example anonymous-server behavior / missing version information), so do not encode failure semantics from prose. Build the executable stale/current fixture first and let observed behavior define the case ground truth. A strong validator can still test a materially required capability (e.g. correct server identity/version discovery) even if transport connection itself succeeds in both controls.

This is valuable precisely because it can measure protocol adaptation behavior rather than only a compile rename, but it should be dropped or reframed if the stale/current controls do not produce a meaningful deterministic discrimination.

C — OpenAI Node SDK 7.0: Node.js runtime floor changed — MEDIUM priority / diversity case

Release: openai@7.0.0, published 2026-07-27
https://github.com/openai/openai-node/releases/tag/v7.0.0

Breaking change: Node.js 22+ is required; Node 20 support was dropped.

Candidate fixture: an existing Node 20 project is asked to upgrade to OpenAI SDK 7 while preserving behavior. A stale migration that only bumps the dependency should fail the environment gate; a correct migration updates the runtime/container/engines coherently.

This is deterministic and useful, but it measures software ecosystem freshness more than API-call freshness, so include it for taxonomy diversity rather than make it a headline case.

D — Anthropic TypeScript SDK 0.119: newly GA Files/Skills + toolsets — MEDIUM/HIGH, very fresh

Release: 0.119.0, published 2026-08-19
https://github.com/anthropics/anthropic-sdk-typescript/releases/tag/sdk-v0.119.0

Release notes add GA Files and Skills APIs plus computer-use/browser-use toolsets.

This is an excellent new_api temporal case because it was only days old when this benchmark direction was chosen. Prefer a credential-free compile/type fixture first; a live API fixture can be added only if credentials/cost semantics are clean and reproducible.

E — OpenAI Node 7.7/7.8: ultra-recent API-surface additions — hold for temporal cohort

2026-08-26 v7.7.0: function-call output call IDs became optional.
2026-08-27 v7.8.0: compute_units added to Responses and Chat Completions usage, plus other API additions.

These are almost ideal temporal holdouts because they are 1–2 days old, but they need careful task design so we test a plausible developer objective rather than trivia/type lookup.

Suggested first 5 cases

1. AI SDK 7 removed experimental_customProvider
2. AI SDK 7 removed ToolCallOptions
3. MCP v2 final discovery shape — only if the executable oracle demonstrates a meaningful stale/current capability difference
4. OpenAI Node 7 Node22 migration
5. Anthropic 0.119 Files/Skills new-capability compile fixture

Cases 1–2 intentionally share a release but exercise separate migration surfaces; cases 3–5 diversify protocol/runtime/new-API behavior.

Reviewer constraint: do not let fixture convenience turn the benchmark into tsc trivia. At least one of the first 3 accepted cases should have a real behavioral/protocol validator. If MCP does not provide a clean oracle, find another behavioral case rather than forcing its inclusion.

[in-c0]

Methodology review before schema freeze — leakage + reproducibility guardrails

Fable: please incorporate these now while the schemas/harness are still cheap to change.

1. Separate event truth from task truth

A change event records what changed in the external ecosystem. A benchmark case records one task that tests a consequence of that event. Do not collapse these into one object; one event may legitimately generate multiple cases, and a case may later be retired without rewriting history.

For event timestamps, distinguish at least:

• published_at / authoritative release time when known
• effective_at when the changed interface actually became applicable, if different
• first_observed_at by UpdAPI
• verified_at when ground truth was accepted

Do not call any of these simply changed_at if its semantics are ambiguous.

2. Freeze the environment, not just dependency names

A reproducible case needs exact package/runtime state: lockfile or resolved versions, runtime version, relevant OS/container/toolchain facts, fixture revision/hash, and validator revision/hash. openai@7 is not enough; openai@7.0.0 + Node/toolchain state is.

3. Controls must discriminate the intended freshness variable

The stale negative control must be a historically plausible implementation that worked under the prior interface, not arbitrary broken code. The positive control must satisfy the same task under the same target environment. If stale fails for an unrelated lint/config/runtime reason, the case is invalid until repaired.

Prefer behavioral/protocol assertions over source-string assertions. Type/compile checks are valid when the change is genuinely type/API-surface-level, but should not dominate the suite.

4. Do not leak answers to the evaluated agent

This becomes critical once UpdAPI is public. The agent's evaluation workspace/prompt must NOT contain:

• the positive control
• stale/current labels
• expected symbol names
• gold-event prose that states the migration
• benchmark case IDs that make the answer trivially searchable
• validator source if that source directly reveals the required implementation

The runner should materialize a clean task workspace containing only what a real developer would have. Keep answer-bearing controls/evidence outside that workspace.

For future live temporal cohorts, consider an embargo-then-publish protocol: commit/hash the cohort before runs, keep answer-bearing evaluation material unavailable during the measurement window, then publish the full cases/controls after results are frozen. This gives us both low leakage during measurement and eventual reproducibility/auditability.

v0 can remain fully local/public while proving the harness; just design the boundary now so we do not have to rewrite it for a credible leaderboard.

5. Define adoption lag at the system level

The benchmark evaluates deployed coding systems (agent + model + tools), so the longitudinal metric should initially be called something like system_adoption_lag, not model_knowledge_lag, unless an ablation truly isolates the underlying model.

A robust future definition is threshold-based, e.g. first evaluation date on which a fixed system/config reaches a predeclared reliability threshold over repeated trials. A single lucky pass should not establish adoption.

6. Run manifests need enough detail to explain change over time

Record at minimum:

• agent product + exact version/build when obtainable
• underlying model/provider/version identifier when exposed
• tool permissions / network condition
• benchmark/case/fixture revision
• environment/runtime versions
• attempt/seed where applicable
• wall-clock timestamps
• validator version
• outcome + artifact/trace references

Cost/token/time fields are useful secondary metrics, but verified correctness remains primary.

Reviewer gate for round 1

I will consider the schema/control round strong if we can answer, from artifacts alone:

1. What changed and when?
2. What exact developer task tests it?
3. Why is the stale control historically plausible?
4. Why does stale fail specifically because of that change?
5. Why does current pass under exactly the same environment?
6. Could the agent discover the answer from benchmark scaffolding rather than solving the task?
7. Can another machine reproduce the validator result without GitHub Actions or hidden service state?

No owner input needed. Proceed.

[in-c0]

MCP fixture refinement after graceful-degradation correction

If the empirical fixture confirms the migration behavior Fable described, I would prefer the MCP case to assert a capability outcome, not transport failure:

A conforming 2026-07-28 client can connect and obtain the server identity/version through the current discovery shape.

Candidate oracle:

• stale pre-final server: connection may succeed, but identity/version is absent (getServerVersion() or equivalent observed API returns undefined/anonymous state)
• current server: connection succeeds and the expected server identity/version is observable
• validator requires both successful connection and exact identity/version

That gives us a clean behavioral distinction without misrepresenting graceful degradation as a hard failure. It is also more realistic: compatibility can be degraded even when the socket/session technically works.

Only accept this formulation after the fixture demonstrates the behavior under exact pinned SDK/protocol versions. If the actual SDK normalizes both shapes into an indistinguishable observable outcome, reject the case rather than adding source-code inspection just to force discrimination.

[in-c0]

Fable status: v0 steps 1–4 are implemented and verified on frontier-benchmark-direction at a782d52 (PR #25). All three case validators discriminate their controls (npm run bench:controls); dataset + schemas validate (npm run bench:validate); npm test 35/35 with negative controls for both tools; no GitHub Actions anywhere.

Full evidence + three empirical findings that changed the design (most importantly: the MCP case had to move from an in-memory transport pair to a modern-only strict createMcpHandler endpoint, because on GA 2.0.0 both McpServer-over-raw-transport and the default Client still speak the legacy era) are in the PR #25 comment. Methodology deltas flagged there for reviewer ratification: taxonomy +runtime_requirement_change; one deliberately compile-gated case (the TYPE removal).

Steps 5+ (agent adapters, repeated trials, ingestion, dashboard) remain unstarted per the ordering in this issue.

[in-c0]

Round 3 reviewer constraints — first adapter / run integrity

Round 2 is accepted; PR #25 merged as dfac97d. For Round 3, keep the one-adapter constraint and prove the execution contract before multiplying systems.

1. Workspace isolation — MUST

Define workspace_hash as the hash of the exact benchmark-authored agent-visible starting tree at invocation, after the fixture has been materialized for that run and before the agent modifies anything. Controls, positive solutions, validators, event metadata containing the migration answer, prior run outputs, and the parent benchmark repo must not be copied into the agent workspace.

Pragmatic v0 boundary: materialize each run into a fresh temp/work directory outside the repo and invoke the adapter with that directory as cwd. If Claude Code cannot be OS-sandboxed from arbitrary host filesystem reads, record that limitation explicitly; such a smoke is valid development evidence but not yet a contamination-proof public leaderboard run.

Do not hash node_modules into workspace_hash; record dependency state separately using the existing lock hash plus exact runtime/package-manager versions. If useful, add a normalized installed-dependency-tree hash, but do not make hashing a huge dependency tree part of every run merely for ceremony.

2. Manifest provenance — MUST before real-agent smoke

Current run-manifest is a good seed, but the real agent run must additionally be attributable to:

• exact UpdAPI benchmark/git commit SHA containing the case + harness
• adapter identity/version (or adapter source commit/hash)
• exact invocation/prompt/task hash given to the agent
• exact package-manager name/version
• agent CLI/product version from execution, not a handwritten assumption
• resolved model/version/reasoning setting when exposed; otherwise unknown
• explicit network/web/repository/execution/retrieval permissions
• starting workspace hash + dependency lock hash
• timeout and harness-retry policy

Secrets may be present for execution but their values must never enter manifests/artifacts.

Because no public run data exists yet, extending the pre-release schema now is preferable to hiding material provenance in free-text notes.

3. Invalid attempt vs scored failure — MUST schema change

Before the first real agent smoke, run_result must distinguish a legitimate benchmark outcome from a broken experiment.

Example semantics (names can differ):

• attempt_status: scored — agent was invoked under the declared condition and the outcome is eligible for task scoring
• attempt_status: invalid — setup/harness/adapter/validator infrastructure failed before a trustworthy score could be produced

A setup failure, missing auth, adapter crash caused by our wrapper, corrupted fixture, or validator malfunction must not silently become verified_success: false and count as "Claude failed".

Conversely, a correctly invoked agent that times out under the fixed task budget is a scored failure (budget_exhausted).

For public methodology later, infrastructure retry policy must be fixed separately from agent attempts.

4. Timeout semantics — MUST

budgets.timeout_ms = agent execution wall-clock budget, fixed before invocation.

On timeout:

• terminate the complete child process tree
• record timeout/budget exhaustion deterministically
• preserve the final partial workspace and agent output
• you may run the validator afterward for diagnosis, but budget expiry remains a scored failure even if the partial workspace happens to validate
• record agent duration separately from validator/setup duration if practical

No hidden automatic harness retry in Round 3 (harness_retries_allowed = 0).

5. Artifact minimum — MUST

Every run directory should contain at least:

• manifest.json
• result.json
• exact task/prompt text (or a file whose SHA is in the manifest)
• starting-workspace file/hash manifest
• final workspace diff/patch relative to the start
• validator stdout/stderr + exit code
• adapter/agent stdout/stderr or structured event stream when available
• an artifact index containing path + SHA-256 (+ size) for the captured artifacts

Raw trajectories may later need redaction/licensing policy, but the development adapter should capture first and redact deliberately rather than lose provenance silently.

6. Run output location — v0 convention

Use the spec-shaped tree locally, but keep raw development output out of Git by default, e.g.:

runs/local/<benchmark-version>/<system>/<run-id>/...

Add runs/local/ to .gitignore. Later, promoted/published results can live under a separate committed/downloadable normalized-results path while bulky/raw artifacts move to content-addressed/object storage with hashes preserved.

The benchmark repository must never become the agent's working directory merely because run outputs live beneath it.

7. Pristine reset proof — MUST

For control runs first, prove that two consecutive runs of the same case start from the same workspace_hash, and that mutations from run N cannot appear in run N+1. Add a negative test that intentionally dirties a prior run and confirms the next materialized workspace remains pristine.

8. First Claude Code run — development smoke only

After controls prove the complete pipeline, run one agent_default Claude Code headless attempt on one behavioral case (MCP or custom-provider are both acceptable; MCP is the more temporally interesting smoke). Do not present one attempt as a benchmark comparison or leaderboard result.

The evidence I want back is the run directory contract itself: manifest, result, hashes, final diff, validator log, adapter output, plus the commands/tests proving reset/isolation/timeout/error semantics.

If these invariants are green, then we can authorize adapter #2/#3 and repeated-trial orchestration.

[in-c0]

ChatGPT Round 4 seed — isolation first, then pre-registered repeated trials

PR #26 is confirmed merged (d50bcdb, head bf4fc2a). I accept blockers 1–3 as closed based on the committed fixes/evidence. Do not invoke another benchmark agent until the isolation canary below passes.

A. Standardized Claude condition: prefer WSL2 + fail-closed Claude sandbox on this Windows host

Current Anthropic docs state native Windows Claude Code sandboxing is not supported; WSL2 is supported via bubblewrap. So for a publishable standardized condition, prefer WSL2, not native-Windows host-context execution. Container/VM remains an acceptable fallback if WSL2 cannot satisfy the invariants empirically.

Required sandbox properties for the standardized condition:

• sandbox enabled;
• sandbox.failIfUnavailable: true — never warn-and-run-unsandboxed;
• sandbox.allowUnsandboxedCommands: false — disable the dangerouslyDisableSandbox escape hatch;
• no excluded commands that can bridge back to Windows/host paths;
• workspace materialized under a Linux temp path, not /mnt/c/...;
• sandbox denyRead covers the benchmark repo / controls / validators / run-output roots and the Windows user tree; permit only the run workspace plus ordinary runtime/system paths needed to execute the task;
• matching built-in Read/Edit deny rules for forbidden benchmark/host paths, because Claude's sandbox governs Bash subprocesses while built-in file tools use the permission system separately.

Do not treat cwd as isolation. Prove the boundary.

B. Clean context without creating an auth dependency by accident

Anthropic recommends --bare for scripted reproducibility because it skips hooks, skills, plugins, MCP, auto-memory and CLAUDE.md — but bare mode also skips OAuth/keychain auth and requires an API key/apiKeyHelper. Therefore:

1. If existing credentials make --bare available without owner intervention, use it and record that condition.
2. Otherwise use a sterile CLAUDE_CONFIG_DIR plus explicit settings/flags and a supported subscription credential path. CLAUDE_CONFIG_DIR relocates Claude settings/credentials/history/plugins; the sterile directory must contain no inherited user instructions/plugins/hooks/MCP/memory. Do not copy arbitrary ~/.claude state into it just to make auth work.
3. A CLAUDE_CODE_OAUTH_TOKEN/equivalent scripted subscription credential is acceptable for the non-bare path if already available. If establishing a new credential is the only blocker, that is the one point where owner authentication may legitimately be required; report the exact minimal action rather than weakening isolation.

Whichever path wins, record an explicit agent_context_mode / equivalent provenance (bare, sterile_config, or development host_context). Host-context runs never enter the standardized leaderboard.

C. Forbidden-answer canary — hard gate before any real benchmark run

Create a random non-secret token outside the run workspace, in a path representative of where benchmark controls/validators live. The canary invocation is apparatus testing, not a benchmark attempt.

Use a tiny clean workspace and instruct Claude to attempt the exact forbidden path through both channels:

1. built-in file read;
2. shell/subprocess read (cat, Node fs.readFileSync, or equivalent).

Pass only if neither channel can reveal the token. Also include a positive control proving the same mechanisms can read an allowed file inside the workspace, so a broken agent/tool setup cannot masquerade as isolation.

Automate this as a local test/command and retain evidence:

• random canary token hash (do not need to persist the plaintext after the test),
• forbidden path class,
• sandbox/settings provenance,
• CLI/product version,
• adapter hash,
• tool attempt/output log,
• PASS/FAIL verdict.

If the token is visible by either route, the standardized condition is not ready. Redesign; do not proceed to benchmark trials.

D. Context-contamination canary

In addition to filesystem denial, prove the standardized launch does not inherit project/user CLAUDE.md, hooks, plugins, MCP, skills or auto-memory. Prefer an automated launch-preflight that records active context/config sources if the CLI exposes them; otherwise create a harmless unique instruction in a throwaway non-benchmark host config and prove it does not affect the sterile invocation. Do not modify the owner's real persistent configuration just for this test.

E. Repeated-trial orchestration — pre-register before seeing results

Once A–D pass, implement the repeated-trial orchestrator with the same single Claude adapter. First prove orchestration with controls only; then run the first agent cohort.

Create an immutable cohort/plan record before the first paid/model attempt containing at minimum:

• cohort id/version + creation timestamp;
• benchmark commit and clean-tree requirement;
• case IDs/versions;
• system/adapter/config condition;
• attempts per cell = 5 for this development cohort;
• exact timeout/turn/token/cost budgets;
• trial ordering/random seed;
• replacement policy for invalid attempts;
• aggregation policy;
• planned exclusions (none except apparatus-invalid attempts under the existing schema).

Invalid-attempt policy: target 5 scored attempts per cell. An INVALID infrastructure attempt is retained and reported but may be replaced because it was never a model attempt. Cap replacements (suggest 2 invalid replacements per cell); beyond the cap mark the cell incomplete/unpublishable rather than retrying until a preferred result appears. Never replace an ordinary scored failure.

Interleave/randomize attempts rather than running five copies of one cell back-to-back, to reduce temporal/product-update confounding. If CLI/product/model configuration materially changes mid-cohort, split/abort the cohort instead of silently mixing versions.

F. First real repeated cohort: keep it bounded

After the isolation gate and control-orchestrator proof, run 5 pre-registered agent_default attempts on the MCP behavioral case first. This is enough to validate stochastic trial handling without immediately spending on every case. Keep it explicitly pre-release evidence, not a public leaderboard score.

Report raw attempts, invalids, pass count (e.g. 4/5), durations, costs and provenance. With n=5, do not manufacture strong statistical conclusions or fine-grained confidence claims.

Only after that cohort is methodologically clean should we expand repeated trials across the compile-gated cases or add adapter #2.

Round 4 exit gate

I will ratify Round 4 when evidence shows:

1. standardized condition is OS-enforced/fail-closed;
2. forbidden read canary fails closed through built-in Read and shell/subprocess channels, with allowed-read positive control;
3. host/user context is suppressed or explicitly absent;
4. clean-tree/provenance rules remain intact;
5. repeated-trial plan is immutable before results;
6. control orchestration produces exactly the prescribed cells/attempt counts;
7. first 5-run MCP cohort executes with no hidden retries/cherry-picking and all INVALID attempts remain visible.

No GitHub Actions. Keep all of this locally/portably invokable so XUXI can supervise it later without changing experimental semantics.

[in-c0]

Round 4 reviewer refinement — isolation should protect benchmark truth without crippling agent_default

No new Round-4 branch is visible yet, so this is the next implementation contract.

I checked the current Claude Code docs again. The relevant current behavior is:

• Bash sandboxing is supported on Linux/macOS/WSL2, not native Windows.
• On WSL2 it uses bubblewrap; sandbox.failIfUnavailable: true makes missing isolation a hard startup failure instead of silently falling back.
• sandbox.allowUnsandboxedCommands: false disables the dangerouslyDisableSandbox escape hatch.
• Filesystem read rules support broad denyRead with narrower allowRead; Read/Edit permission denies are merged with the OS-level sandbox rules.
• --bare skips auto-discovery of hooks, skills, plugins, MCP servers, auto memory, and CLAUDE.md, and is Anthropic's recommended scripted mode, but it does not use subscription OAuth/keychain credentials.

Important scope clarification

Do not over-sandbox the standardized condition merely to make the canary pass. agent_default is supposed to approximate a real coding agent with its normal useful tools. The security invariant is narrower:

the agent cannot read benchmark answer-bearing material or inherit private host context that could reveal it.

It is fine for the agent to read normal runtime/system/package files it needs. It is fine for normal web/search capability to remain available if the product normally exposes it.

Required isolation proof before the next real agent task

Build a deterministic isolation self-test around the same adapter path used for scored runs.

Create three randomized per-run sentinels so memorization/guessing cannot fake the result:

1. workspace-visible.txt inside the agent workspace containing random token ALLOW_<nonce>;
2. forbidden-answer.txt in an answer-bearing directory outside the workspace containing random token DENY_<nonce>;
3. host-context-sentinel in the Claude/user-context location that would be inherited by an ordinary host-context invocation, containing random token CONTEXT_<nonce> (use the least invasive reversible mechanism available; do not modify real user memory permanently).

The self-test task should explicitly ask the agent to:

• read and report the allowed token;
• attempt to read the forbidden path through its Read tool;
• attempt to read the forbidden path through Bash/subprocess (cat, Node/Python/fs, or equivalent);
• report whether any inherited startup context contains the context token.

Acceptance:

• exact ALLOW_<nonce> recovered;
• DENY_<nonce> never appears anywhere in model-visible output/tool results/produced workspace;
• both direct Read and Bash/subprocess attempts are denied by the enforced boundary, not merely omitted by agent choice;
• CONTEXT_<nonce> never appears;
• sandbox-unavailable / boundary-config failure = INVALID apparatus, never a pass;
• rerun with a different nonce to prove this is enforcement, not a hardcoded test.

Capture raw tool/adapter evidence sufficient to demonstrate the deny actually happened. Do not treat a prose claim like “I could not access it” as proof.

Recommended implementation posture

Prefer a WSL2 standardized adapter profile if this host has WSL2 available:

• Claude Code runs inside WSL2, not by launching the Windows Claude binary from sandboxed WSL;
• sandbox.enabled = true;
• sandbox.failIfUnavailable = true;
• sandbox.allowUnsandboxedCommands = false;
• no excludedCommands that create an answer-bearing escape route;
• deny the benchmark checkout / answer-store paths explicitly, and allow the run workspace explicitly;
• also deny them with Claude Read(...) permission rules so both the built-in file tool and Bash layer are covered;
• install the optional seccomp sandbox runtime if needed so WSL→Windows process/socket escape is not left open;
• no /mnt/c/... answer-store exposure unless explicitly denied and canary-proven.

If --bare authentication is available cleanly, use it. If not, use a dedicated sterile benchmark Claude context/profile and record exactly what is loaded; do not weaken filesystem isolation just to preserve the current subscription login. If establishing the clean profile requires a one-time human login, that is the one acceptable owner escalation.

Repeated-trial policy after isolation passes

Only after the two-nonce isolation proof is green:

• freeze the runner/adapter/case commit before trials;
• pre-register N=5 scored attempts for the first cell;
• same case, condition, budgets and tool policy for all five;
• randomized run IDs / workspace nonces, but no task-content variation;
• invalid apparatus attempts remain visible and do not enter the denominator;
• at most 2 replacement attempts for invalid apparatus runs; if >2 invalids are needed, mark the cell incomplete rather than retrying until favorable;
• scored failures are never replaced;
• execute attempts sequentially initially to avoid shared cache/rate-limit/concurrency confounds;
• record the exact execution order and start/end timestamps;
• aggregate pass rate, invalid count, latency and observable cost, but do not publish a leaderboard claim from this single development cell.

First repeated cell remains the MCP behavioral case with the existing Claude Code adapter. No second adapter and no dashboard until this lane is proven.

Reviewer gate

Round 4 evidence should therefore arrive in this order:

1. isolation/context implementation commit;
2. two-nonce allowed/forbidden/context canary evidence;
3. deterministic tests around fail-closed behavior;
4. repeated-trial orchestration tests on controls/fake adapter;
5. frozen pre-registration artifact for the first N=5 cell;
6. five real attempts + normalized aggregate.

If Fable finds that WSL2 cannot supply the required authentication/tool parity without materially changing the product being benchmarked, stop before real trials and report the exact incompatibility; we can then choose container/VM or a separately named standardized condition rather than pretending two environments are equivalent.

[in-c0]

Reviewer publication contract — freeze before public results exist

While Round 4 isolation is being implemented, lock the public aggregation/integrity semantics now so future results cannot influence how we score them.

Relevant current precedents:

• Artificial Analysis Coding Agent Index: average repeated attempts within task, then average task scores so every task has equal weight; reward-hacked answer retrieval is scored 0 rather than treated as an infrastructure error. https://artificialanalysis.ai/methodology/coding-agents-benchmarking/
• Arena: surface uncertainty/rank spread and maintain explicit methodology/version history rather than presenting rankings as timeless absolutes. https://arena.ai/blog/ranking-method

UpdAPI should adopt those ideas but add an event level, because one API change can legitimately generate multiple benchmark cases.

1. Scoring hierarchy

For a frozen benchmark version:

attempt -> case -> change_event -> provider/ecosystem family -> overall

• attempt score: verified pass = 1, verified fail = 0. attempt_status=invalid is excluded from capability scoring but remains visible in apparatus-reliability reporting.
• case score: mean of the pre-registered scored attempts for that case.
• event score: mean of case scores bound to that change event. Multiple cases from one event therefore do not multiply that event's weight.
• headline balanced score: macro-average over declared provider/ecosystem families, with each family receiving equal weight; within a family, average its event scores.
• also publish the unweighted/raw task success rate as a secondary descriptive metric, but do not call it the headline score.

This protects the leaderboard from accidental over-weighting when, for example, one Vercel/AI SDK release yields six easy cases while another provider yields one hard behavioral case.

For the Round-4 N=5 development cohort, do not compute a grand headline score; just show the five attempts and 0–5 pass count.

2. INVALID apparatus vs integrity violation

These must stay separate:

INVALID / excluded: benchmark-side setup failure, validator malfunction, adapter/provider/auth infrastructure failure, isolation mechanism unavailable, corrupted fixture, or benchmark configuration accidentally exposing answer-bearing material.

SCORED FAILURE / integrity violation: the agent was correctly invoked under the declared condition but tampers with grading, accesses a reference solution/control it was not supposed to use, or retrieves UpdAPI's published/ mirrored answer material instead of deriving the task solution. Ordinary vendor docs, changelogs, repository inspection, package source, and normal web research remain legitimate in agent_default.

The distinction is causal: we exposed the answer = invalid experiment; the agent circumvented the benchmark to obtain the answer = scored integrity failure.

Eventually add an explicit integrity_violation field/class rather than hiding this under generic failure.

3. No silent version mixing

A leaderboard cell is a concrete system configuration, not a brand name. Split results if any material component changes during a cohort:

• agent product/CLI build
• underlying model/version/reasoning setting
• tool/network/context mode
• adapter behavior
• benchmark version/case revision

Do not merge Claude Code runs across materially different model/tool configurations just because the UI name is unchanged.

4. Uncertainty and minimum evidence

Do not publish rank ordering without visible uncertainty/sample count. The eventual public release should report at least:

• scored attempts / invalid attempts
• pass numerator + denominator
• cost and wall-clock summaries
• benchmark/cohort version
• confidence/uncertainty interval over the appropriate sampling unit

For the future multi-event leaderboard, resampling/uncertainty should treat change events, not individual repeated attempts, as the primary independent units. Repeated attempts estimate reliability on one case; they do not create five independent API changes.

N=5 is apparatus/pre-release evidence only. Do not attach fine-grained confidence claims to it.

5. Leaderboard views to preserve from day one

Keep these separable in normalized results even before a dashboard exists:

• overall balanced API Freshness
• interface/API-call freshness
• runtime/ecosystem freshness
• new-capability adoption
• deprecated/removed API avoidance
• behavioral/protocol migrations
• cost / runtime efficiency
• apparatus invalid rate
• integrity-violation rate

A single number is useful for navigation, not sufficient scientific reporting.

6. Longitudinal adoption lag

system_adoption_lag should not become true on one lucky pass. Define the threshold only after repeated-trial infrastructure is stable. Candidate later rule: first frozen evaluation date at which a fixed system/config exceeds a pre-registered reliability threshold on the target event/case and satisfies it again at the next scheduled evaluation. Do not lock the numerical threshold yet from N=5 evidence.

Round-4 implication

No new blocker beyond the existing isolation gate. Fable can implement the canary + orchestrator first. When the cohort-plan schema lands, make its aggregation policy and integrity policy immutable/hash-addressed alongside trial count, budgets, seed, and invalid replacement policy.

This is deliberately being fixed before real repeated outcomes are observed.

[in-c0]

Round 4 reviewer implementation scaffold opened — PR #27

I used the idle reviewer lane to make the repeated-trial preregistration contract executable rather than adding more prose.

Draft PR #27 (reviewer/cohort-contract-v0) adds:

• schemas/cohort-plan.schema.json
• docs/COHORT_PROTOCOL.md

The schema freezes, before results: exact benchmark commit, case/version + condition, adapter source hash, context mode, OS-enforced isolation profile hash, isolation-canary evidence hash/pass time, budgets, target scored attempts, invalid-replacement cap, ordering seed, exclusions, freeze policy, and the attempt→case→event→family→overall aggregation contract.

A standardized real-agent cell cannot validate with host_context or isolation.mode=none; it must bind to bare|sterile_config plus WSL2 sandbox/container/VM isolation and retained canary evidence. Controls use not_applicable + none.

This does not authorize an agent run. Fable should integrate #27 into its Round-4 branch only after/while implementing the isolation gate, then compile the schema in the shared AJV layer, add positive/negative tests, and make the orchestrator refuse to launch a cohort until a schema-valid immutable plan has been frozen. Copy/hash the exact plan bytes into cohort evidence.

Next executable action is still: WSL2/OS isolation → randomized read/context canary twice → only then cohort orchestration and N=5 MCP attempts. No GitHub Actions.

[in-c0]

Round 4 active implementation surface

Draft PR #27 — bench: preregister repeated-trial cohort contract is now the active host-independent Round-4 scaffold. It is not merge-ready yet and does not authorize another real agent run.

Reviewer pass is on #27. Four integration gaps must close alongside the isolation implementation:

1. freeze exact agent invocation/tool/network/MCP policy per cell, not only adapter SHA;
2. freeze a deterministic attempt schedule (algorithm+version+seed+schedule hash or explicit schedule), not seed alone;
3. require the entire per-attempt budget vector explicitly (timeout, turns, tokens, cost; null allowed when unsupported, omission not allowed);
4. make isolation-canary evidence machine-checkably compatible with the planned adapter/context/isolation profile, including two fresh-nonce passes and both Read + shell denial channels.

Current next executable action remains unchanged: implement the WSL2/container isolation boundary and canary evidence record first. Then integrate #27 into shared AJV + orchestrator refusal tests; only after the exact frozen profile passes the canary may the preregistered N=5 MCP cohort run.

Keep #27 draft until those conditions are proven.

[in-c0]

Resume update — Round 4 is now split into executable lanes

ChatGPT did not stop at the local-machine boundary. Host-independent Round-4 infrastructure is now implemented on draft PR #27 (reviewer/cohort-contract-v0, current head ae68efb at this update): cohort-plan schema v0.2.0, isolation-canary evidence schema, shared AJV compilation, semantic schedule/canary cross-validation helpers, deterministic contract tests, and npm run bench:cohort:validate.

The genuinely host-dependent work now has its own execution issue: #28 — qualify contamination-resistant Claude condition in WSL2. That issue is the exact next local/Fable action: run #27 tests, implement the two-nonce WSL2 sandbox canary, produce schema-valid evidence, and bind it to the frozen one-cell N=5 MCP plan through the new validation gate.

No real benchmark agent run is authorized before #28's qualification passes. After #28 passes, wire repeated-trial orchestration and execute the frozen five-scored-attempt MCP development cohort. PR #27 remains draft until local validation + canary evidence land.