[bench] config: raise the per-stage Oracle timeout to 300s - #35
Conversation
At the library default of 120s the Oracle timed out on constitutionally heavy diffs. Because a timeout fails closed, it surfaced as a VETO carrying pipeline_error: the change was blocked with reasoning that looked like a ruling but was a pipeline failure. That happened three times while drafting the C-008 amendment and blocked it outright until the cause was found, so it is a correctness problem for every contributor on BENCH_PROVIDER=claude_code, not a machine preference. Raising the ceiling does not weaken enforcement. Fail-closed behavior is unchanged and confirmed in the chain: the entry recording the 120s timeout carries verdict VETO with pipeline_error true. A slow judge is simply no longer mistaken for a strict one. The machine-local BENCH_LEDGER_PATH pin is removed with it. It existed only to keep one chain while branches predating the privacy change still routed to ledger/, and the last such branch (bench/comment-trim, merged in PR #28) has been deleted. resolve_ledger_path() now computes the identical path on its own, verified before removal. The pin was also an absolute path tied to one machine, which would have silently split writers from readers if the repo ever moved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HM2xLE9X1hcE3mhwffvmmR
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b4dae77c3e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "env": { | ||
| "BENCH_PROVIDER": "claude_code" | ||
| "BENCH_PROVIDER": "claude_code", | ||
| "BENCH_CLAUDE_TIMEOUT": "300" |
There was a problem hiding this comment.
Extend the outer hook timeout beyond all stages
When a non-CLEAR diff makes Challenger, Defender, and Oracle each approach the new 300-second limit, the pipeline can require roughly 900 seconds before any parse retries, but the PreToolUse command still has no explicit timeout and therefore retains Claude Code's 600-second default. Claude Code can consequently cancel the entire hook before it emits a verdict or ledger receipt, so the heavy diffs this change is intended to unblock still fail at the enclosing timeout instead. The Claude Code command-hook reference documents timeout as a per-hook setting with a 600-second default; set it above the pipeline's cumulative worst-case duration when raising this per-call value.
Useful? React with 👍 / 👎.
Promotes the Oracle timeout from a machine-local override to committed config, and removes the ledger pin that is no longer needed.
Why this is a correctness fix, not a preference
At the library default of 120s the Oracle timed out on constitutionally heavy diffs. Because a timeout fails closed, it surfaced as a
VETOcarryingpipeline_error— the change was blocked with output that reads like a ruling but was a pipeline failure.This happened three times while drafting the C-008 sanitation amendment and blocked it outright until the cause was diagnosed. Anyone on
BENCH_PROVIDER=claude_codeproposing a substantial change hits the same wall, so it belongs in the shared config rather than in one developer'ssettings.local.json.Enforcement is not weakened
Bench asked this directly during review, twice. Fail-closed behavior is unchanged, and the chain has the receipt: the entry recording the 120s timeout carries
verdict: VETOwithpipeline_error: true. Raising the ceiling only stops a slow judge from being mistaken for a strict one.The ledger pin comes out with it
BENCH_LEDGER_PATHexisted only to keep a single chain while branches predating the privacy change still routed toledger/. The last such branch,bench/comment-trim(merged in PR #28), has been deleted, somainis the only remaining branch and all of it resolves to.bench/.Verified before removal:
resolve_ledger_path()computesC:/Users/mstar/Bench/.bench/bench-ledger.json, identical to what the pin specified, andpython -m cli verifystill reports VALID with genesis4e98fb41unchanged.Removing it also retires a real hazard Bench flagged when it was added: an absolute path tied to one machine would silently split writers from readers if the repo ever moved.
Testing
517 tests pass.
🤖 Generated with Claude Code
https://claude.ai/code/session_01HM2xLE9X1hcE3mhwffvmmR