Skip to content

[bench] config: raise the per-stage Oracle timeout to 300s - #35

Merged
dburks-svg merged 1 commit into
mainfrom
bench/oracle-timeout
Aug 22, 2026
Merged

[bench] config: raise the per-stage Oracle timeout to 300s#35
dburks-svg merged 1 commit into
mainfrom
bench/oracle-timeout

Conversation

@dburks-svg

Copy link
Copy Markdown
Contributor

Promotes the Oracle timeout from a machine-local override to committed config, and removes the ledger pin that is no longer needed.

Why this is a correctness fix, not a preference

At the library default of 120s the Oracle timed out on constitutionally heavy diffs. Because a timeout fails closed, it surfaced as a VETO carrying pipeline_error — the change was blocked with output that reads like a ruling but was a pipeline failure.

This happened three times while drafting the C-008 sanitation amendment and blocked it outright until the cause was diagnosed. Anyone on BENCH_PROVIDER=claude_code proposing a substantial change hits the same wall, so it belongs in the shared config rather than in one developer's settings.local.json.

Enforcement is not weakened

Bench asked this directly during review, twice. Fail-closed behavior is unchanged, and the chain has the receipt: the entry recording the 120s timeout carries verdict: VETO with pipeline_error: true. Raising the ceiling only stops a slow judge from being mistaken for a strict one.

The ledger pin comes out with it

BENCH_LEDGER_PATH existed only to keep a single chain while branches predating the privacy change still routed to ledger/. The last such branch, bench/comment-trim (merged in PR #28), has been deleted, so main is the only remaining branch and all of it resolves to .bench/.

Verified before removal: resolve_ledger_path() computes C:/Users/mstar/Bench/.bench/bench-ledger.json, identical to what the pin specified, and python -m cli verify still reports VALID with genesis 4e98fb41 unchanged.

Removing it also retires a real hazard Bench flagged when it was added: an absolute path tied to one machine would silently split writers from readers if the repo ever moved.

Testing

517 tests pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HM2xLE9X1hcE3mhwffvmmR

At the library default of 120s the Oracle timed out on constitutionally
heavy diffs. Because a timeout fails closed, it surfaced as a VETO carrying
pipeline_error: the change was blocked with reasoning that looked like a
ruling but was a pipeline failure. That happened three times while drafting
the C-008 amendment and blocked it outright until the cause was found, so it
is a correctness problem for every contributor on BENCH_PROVIDER=claude_code,
not a machine preference.

Raising the ceiling does not weaken enforcement. Fail-closed behavior is
unchanged and confirmed in the chain: the entry recording the 120s timeout
carries verdict VETO with pipeline_error true. A slow judge is simply no
longer mistaken for a strict one.

The machine-local BENCH_LEDGER_PATH pin is removed with it. It existed only
to keep one chain while branches predating the privacy change still routed
to ledger/, and the last such branch (bench/comment-trim, merged in PR #28)
has been deleted. resolve_ledger_path() now computes the identical path on
its own, verified before removal. The pin was also an absolute path tied to
one machine, which would have silently split writers from readers if the
repo ever moved.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HM2xLE9X1hcE3mhwffvmmR
@dburks-svg
dburks-svg merged commit d98f9a2 into main Aug 22, 2026
6 checks passed
@dburks-svg
dburks-svg deleted the bench/oracle-timeout branch August 22, 2026 23:49

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b4dae77c3e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .claude/settings.json
"env": {
"BENCH_PROVIDER": "claude_code"
"BENCH_PROVIDER": "claude_code",
"BENCH_CLAUDE_TIMEOUT": "300"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Extend the outer hook timeout beyond all stages

When a non-CLEAR diff makes Challenger, Defender, and Oracle each approach the new 300-second limit, the pipeline can require roughly 900 seconds before any parse retries, but the PreToolUse command still has no explicit timeout and therefore retains Claude Code's 600-second default. Claude Code can consequently cancel the entire hook before it emits a verdict or ledger receipt, so the heavy diffs this change is intended to unblock still fail at the enclosing timeout instead. The Claude Code command-hook reference documents timeout as a per-hook setting with a 600-second default; set it above the pipeline's cumulative worst-case duration when raising this per-call value.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant