Skip to content

[bench] config: raise the per-stage Oracle timeout to 300s #35

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dburks-svg merged 1 commit into from
Aug 22, 2026
Merged

[bench] config: raise the per-stage Oracle timeout to 300s #35

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .claude/settings.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{
"env": {
"BENCH_PROVIDER": "claude_code"
"BENCH_PROVIDER": "claude_code",
"BENCH_CLAUDE_TIMEOUT": "300"

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 22, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Extend the outer hook timeout beyond all stages

When a non-CLEAR diff makes Challenger, Defender, and Oracle each approach the new 300-second limit, the pipeline can require roughly 900 seconds before any parse retries, but the PreToolUse command still has no explicit timeout and therefore retains Claude Code's 600-second default. Claude Code can consequently cancel the entire hook before it emits a verdict or ledger receipt, so the heavy diffs this change is intended to unblock still fail at the enclosing timeout instead. The Claude Code command-hook reference documents timeout as a per-hook setting with a 600-second default; set it above the pipeline's cumulative worst-case duration when raising this per-call value.

Useful? React with 👍 / 👎.

},
"hooks": {
"PreToolUse": [
Expand Down
6 changes: 5 additions & 1 deletion CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -281,7 +281,11 @@ client = openai.OpenAI(
# The child is spawned with BENCH_SUBPROCESS=1 so Bench's own PreToolUse hook
# fails open instead of recursing. subprocess/shutil are stdlib, so this path
# adds no dependency. Per-stage timeout is BENCH_CLAUDE_TIMEOUT seconds
# (default 120).
# (library default 120). This repo sets 300 in .claude/settings.json: at 120
# the Oracle timed out on constitutionally heavy diffs, and because a timeout
# fails closed it surfaced as a VETO carrying pipeline_error, which reads like
# a ruling but is a pipeline failure. Raising the ceiling does not weaken
# enforcement; it stops a slow judge from being mistaken for a strict one.

# Model strings live in utils/api.py as CHALLENGER_MODEL, DEFENDER_MODEL,
# ORACLE_MODEL, and UTILITY_MODEL (the single source of truth). This section
Expand Down