From b4dae77c3e02c994aeb17430084c6d2f252dd52a Mon Sep 17 00:00:00 2001 From: Dana Burks Date: Sat, 22 Aug 2026 16:47:11 -0700 Subject: [PATCH] [bench] config: raise the per-stage Oracle timeout to 300s At the library default of 120s the Oracle timed out on constitutionally heavy diffs. Because a timeout fails closed, it surfaced as a VETO carrying pipeline_error: the change was blocked with reasoning that looked like a ruling but was a pipeline failure. That happened three times while drafting the C-008 amendment and blocked it outright until the cause was found, so it is a correctness problem for every contributor on BENCH_PROVIDER=claude_code, not a machine preference. Raising the ceiling does not weaken enforcement. Fail-closed behavior is unchanged and confirmed in the chain: the entry recording the 120s timeout carries verdict VETO with pipeline_error true. A slow judge is simply no longer mistaken for a strict one. The machine-local BENCH_LEDGER_PATH pin is removed with it. It existed only to keep one chain while branches predating the privacy change still routed to ledger/, and the last such branch (bench/comment-trim, merged in PR #28) has been deleted. resolve_ledger_path() now computes the identical path on its own, verified before removal. The pin was also an absolute path tied to one machine, which would have silently split writers from readers if the repo ever moved. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01HM2xLE9X1hcE3mhwffvmmR --- .claude/settings.json | 3 ++- CLAUDE.md | 6 +++++- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/.claude/settings.json b/.claude/settings.json index 84e9bf7..97a6779 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -1,6 +1,7 @@ { "env": { - "BENCH_PROVIDER": "claude_code" + "BENCH_PROVIDER": "claude_code", + "BENCH_CLAUDE_TIMEOUT": "300" }, "hooks": { "PreToolUse": [ diff --git a/CLAUDE.md b/CLAUDE.md index fb8dfb3..c322a7b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -281,7 +281,11 @@ client = openai.OpenAI( # The child is spawned with BENCH_SUBPROCESS=1 so Bench's own PreToolUse hook # fails open instead of recursing. subprocess/shutil are stdlib, so this path # adds no dependency. Per-stage timeout is BENCH_CLAUDE_TIMEOUT seconds -# (default 120). +# (library default 120). This repo sets 300 in .claude/settings.json: at 120 +# the Oracle timed out on constitutionally heavy diffs, and because a timeout +# fails closed it surfaced as a VETO carrying pipeline_error, which reads like +# a ruling but is a pipeline failure. Raising the ceiling does not weaken +# enforcement; it stops a slow judge from being mistaken for a strict one. # Model strings live in utils/api.py as CHALLENGER_MODEL, DEFENDER_MODEL, # ORACLE_MODEL, and UTILITY_MODEL (the single source of truth). This section