From 4c9065e7ecb11efb007c845b90bf10e01911a53b Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Tue, 25 Aug 2026 02:17:34 +0200 Subject: [PATCH 1/5] docs(babysit-pipeline): correct the backfill guidance against the finished run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #10628 was written while the backfill was still running. Finishing it falsified three of its claims. - Halt-on-cluster counted raw failed `Generate:` runs. Since #10627 an impossible pair spends three runs on its own retries, so two bad pairs trip a threshold of five while nothing is wrong — observed exactly that (6 failures, 2 pairs, healthy pipeline). Count distinct (spec, library) pairs instead. - "Static library against an interactive or 3D spec is the one gap that is usually real" is replaced by its opposite. Every category-level prediction the backfill made was wrong: 17 of 20 parked pairs generated fine, pygal and chartjs each succeeded on the plot types they had been written off for, and bar-3d-categorical succeeded in plotnine while scatter-3d did not. Three pairs failed under a full budget — plotnine on scatter-3d, contour-3d and line-3d-trajectory, all needing a spatial projection plotnine lacks. A gap is real when three attempts say so, not when the pairing sounds implausible. - The one-retry rule needs a precondition now: the workflow already spends three attempts per campaign, so check the run list before adding a manual one. Three failures minutes apart is a measured gap; one is a flake. Same check resolves `RESULT=TIMEOUT` with `recent generate failures: 0`, which only means they aged out of the driver's 25-minute window. New gotcha: intersect label-derived spec ids with the real `plots/` directories. 14 of 26 ids harvested from `impl:*:failed` issue titles had no spec on main, and such a dispatch dies at "Validate specification exists". Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01RbZuWNDFy7kjXh9kLfA4dP --- .claude/skills/babysit-pipeline/SKILL.md | 46 +++++++++++++++++++----- CHANGELOG.md | 13 +++++++ 2 files changed, 50 insertions(+), 9 deletions(-) diff --git a/.claude/skills/babysit-pipeline/SKILL.md b/.claude/skills/babysit-pipeline/SKILL.md index f26aa105ef..e70ca0cbde 100644 --- a/.claude/skills/babysit-pipeline/SKILL.md +++ b/.claude/skills/babysit-pipeline/SKILL.md @@ -102,9 +102,13 @@ with evidence — never let the user ask "still running?". - **Defer, don't halt**: one stuck library must not stop a multi-spec queue — log it, cap ~90 min/spec, move on, and sweep leftovers at the end with targeted regens. -- **Halt the whole queue** only on a failure CLUSTER (≥5 failed - `Generate:` runs in minutes = model daily quota exhausted; a - fallback model via `-f model=` can finish leftovers). +- **Halt the whole queue** only on a failure CLUSTER — ≥5 failed + `Generate:` runs in minutes across **≥3 distinct (spec, library) + pairs** = model daily quota exhausted; a fallback model via + `-f model=` can finish leftovers. Count pairs, not runs: one + genuinely impossible pair burns three runs on its own retries, so + two bad pairs alone trip a raw run count (observed 2026-08-24 — + 6 failures, 2 pairs, pipeline entirely healthy). - Closing/merging pipeline PRs or issues yourself is out of bounds (CLAUDE.md external-write rule + mandatory workflow). @@ -160,6 +164,17 @@ ggplot2/network-force-directed, all three of which had been read as capability gaps. Two failures on the same pair → `deferred.log` with the reason, move on, sweep at the end. +Check the run list before deferring, though: since #10627 the +workflow spends its own three attempts per campaign, so a pair that +comes back missing may already have failed three times, and the +manual retry adds nothing. `gh run list --workflow=impl-generate.yml` +filtered to the spec tells you which case you are in — three +`Generate: for ` failures minutes apart is a measured +gap, one failure is a flake worth the retry. The same check covers +`RESULT=TIMEOUT` with `recent generate failures: 0`, which only means +the failures fell outside the driver's 25-minute window, not that +nothing failed. + ## Gotchas - **`Marking as failed: N generation attempts` counts more than @@ -185,12 +200,25 @@ the reason, move on, sweep at the end. failed 17:55, retried and succeeded 18:02, no human involved), so a single occurrence is noise — only a repeat on the same pair means anything. -- **Static library + interactive/3D spec is the one gap that is - usually real.** plotnine, pygal, seaborn, matplotlib and ggplot2 - against `*-interactive`, `*-drilldown`, `*-realtime`, `*-streaming`, - `slider-*`, `*-3d` specs make up most genuine dead ends; 18 of the - 45 real gaps behind those labels were exactly this shape. Still - give them the one retry, then defer without further ceremony. +- **Do not predict capability gaps — measure them.** The 2026-08-24 + backfill guessed four times which pairs were genuinely impossible + (chartjs on treemap/sankey, plotnine on 3D, ggplot2 on wireframe, + "static library vs. interactive spec") and every category-level + guess was wrong: 17 of 20 parked pairs generated fine, pygal and + chartjs each succeeded on the very plot types they were written off + for, and `bar-3d-categorical` succeeded in plotnine while + `scatter-3d` did not. Only three pairs failed with a full budget — + plotnine on `scatter-3d`, `contour-3d`, `line-3d-trajectory`, all + needing a spatial projection plotnine does not have, while the + "3D" spec that is representable in 2D went through. A gap is real + when three attempts under a fresh campaign budget say so, never + because the pairing sounds implausible. +- **A spec id from an issue title may not exist.** `impl:*:failed` + labels outlive their specs: of 26 spec ids harvested that way, 14 + had no `plots//` on main at all, and a dispatch for one dies + at `Validate specification exists` seconds in. Intersect any + label-derived list with `git ls-tree -d origin/main plots/` before + queueing it. - **The scripts' library list is a copy** of `core/constants.py`'s registry (15 libs). When a library is added/removed, update `monitor_spec.sh`'s `ALL_LIBS` and `run_spec.sh`'s `lang_of` in the diff --git a/CHANGELOG.md b/CHANGELOG.md index 8b7f01943a..af98437edf 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -180,6 +180,19 @@ aggregate instead: an italic *Catalog* line at the end of the version section an ### Changed +- **`babysit-pipeline`'s gap-backfill guidance corrected against a completed run** — the + section shipped in #10628 was written mid-backfill and three of its claims did not + survive the rest of it. The halt-on-cluster threshold counted raw failed runs, but one + impossible pair now burns three runs on its own retries, so two bad pairs tripped it + while the pipeline was healthy; it counts distinct `(spec, library)` pairs now. The + "static library against an interactive or 3D spec is the gap that is usually real" + gotcha is replaced by its opposite: every category-level prediction made during the + backfill was wrong — 17 of 20 parked pairs generated fine, and `bar-3d-categorical` + succeeded in plotnine while `scatter-3d` did not — so a gap counts as real only after + three attempts under a fresh budget. Added: spec ids harvested from issue titles must be + intersected with the actual `plots/` directories (14 of 26 pointed at specs that no + longer exist), and a `TIMEOUT` with zero recent failures means the failures aged out of + the driver's window, not that nothing failed (#10633). - **`babysit-pipeline` learned gap backfill** — closing coverage gaps across the catalogue is a different job from watching one fresh spec, and the skill only described the latter. It now documents reading the missing set from `plots/*/metadata/` instead of from labels From c4e2425d37b073e83f5840a9602b2a85eae6c8df Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Tue, 25 Aug 2026 02:17:45 +0200 Subject: [PATCH 2/5] docs(changelog): reference #10660 in the corrections entry Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01RbZuWNDFy7kjXh9kLfA4dP --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index af98437edf..dc0f4ef05b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -192,7 +192,7 @@ aggregate instead: an italic *Catalog* line at the end of the version section an three attempts under a fresh budget. Added: spec ids harvested from issue titles must be intersected with the actual `plots/` directories (14 of 26 pointed at specs that no longer exist), and a `TIMEOUT` with zero recent failures means the failures aged out of - the driver's window, not that nothing failed (#10633). + the driver's window, not that nothing failed (#10660). - **`babysit-pipeline` learned gap backfill** — closing coverage gaps across the catalogue is a different job from watching one fresh spec, and the skill only described the latter. It now documents reading the missing set from `plots/*/metadata/` instead of from labels From e4711e002deda8c33fc4f5e5e4581b79e7ad859c Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Tue, 25 Aug 2026 02:51:18 +0200 Subject: [PATCH 3/5] fix(babysit-pipeline): retry run_spec.sh's fetch under concurrent drivers Two drivers polling the same checkout collide on .git/refs/remotes/origin/main: error: cannot lock ref 'refs/remotes/origin/main': is at 6b666c10 but expected c736a848 A lost fetch leaves origin/main stale, so meta_present understates what has landed and the poller reports PARTIAL for libraries that are already merged. Retry three times with backoff, and log a warning when all three lose instead of quietly polling a stale ref. Surfaced by the two-slot backfill this guidance introduced. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01RbZuWNDFy7kjXh9kLfA4dP --- .claude/skills/babysit-pipeline/run_spec.sh | 18 ++++++++++++++++-- CHANGELOG.md | 4 +++- 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/.claude/skills/babysit-pipeline/run_spec.sh b/.claude/skills/babysit-pipeline/run_spec.sh index 7a8c8cec8f..2a0cc115ae 100755 --- a/.claude/skills/babysit-pipeline/run_spec.sh +++ b/.claude/skills/babysit-pipeline/run_spec.sh @@ -77,7 +77,21 @@ recent_generate_failures() { || echo "?" } -git -C "$REPO" fetch origin main --quiet +# Two drivers polling the same checkout collide on `.git/refs/remotes/origin/main` +# ("cannot lock ref ... is at X but expected Y"). A lost fetch leaves origin/main +# stale, which makes meta_present understate progress — so retry, and say so in +# the log when all three attempts lose, rather than silently polling a stale ref. +fetch_main() { + local i + for i in 1 2 3; do + git -C "$REPO" fetch origin main --quiet 2>/dev/null && return 0 + sleep $(( i * 3 )) + done + echo "[warn] git fetch origin main failed 3x — origin/main may be stale this round" | tee -a "$LOG" + return 1 +} + +fetch_main TODO=() for lib in "${LIBS[@]}"; do if meta_present "$lib"; then @@ -106,7 +120,7 @@ iters=$(( MAXMIN * 60 / INTERVAL )) idle=0; last_done=-1 for ((i=1; i<=iters; i++)); do sleep "$INTERVAL" - git -C "$REPO" fetch origin main --quiet + fetch_main missing=(); done_n=0 for lib in "${TODO[@]}"; do if meta_present "$lib"; then done_n=$((done_n+1)); else missing+=("$lib"); fi diff --git a/CHANGELOG.md b/CHANGELOG.md index dc0f4ef05b..388af3e583 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -192,7 +192,9 @@ aggregate instead: an italic *Catalog* line at the end of the version section an three attempts under a fresh budget. Added: spec ids harvested from issue titles must be intersected with the actual `plots/` directories (14 of 26 pointed at specs that no longer exist), and a `TIMEOUT` with zero recent failures means the failures aged out of - the driver's window, not that nothing failed (#10660). + the driver's window, not that nothing failed. `run_spec.sh` also retries its + `git fetch`: two drivers polling one checkout collide on the `origin/main` ref lock, + and a lost fetch made the poller read stale metadata and understate progress (#10660). - **`babysit-pipeline` learned gap backfill** — closing coverage gaps across the catalogue is a different job from watching one fresh spec, and the skill only described the latter. It now documents reading the missing set from `plots/*/metadata/` instead of from labels From 415bae43bbafa1e4608db436da8c7043d31b8eb2 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Tue, 25 Aug 2026 03:00:32 +0200 Subject: [PATCH 4/5] docs(babysit-pipeline): address review on #10660 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - `gh run list` has no per-spec filter; show the jq title filter instead of a sentence that reads like a flag exists. - Keep git's stderr in fetch_main's warning: a ref-lock collision and an auth or network failure need different responses, and swallowing the message hides which one happened. - "spec ids" → "spec IDs", matching the repo's Google-style usage. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01RbZuWNDFy7kjXh9kLfA4dP --- .claude/skills/babysit-pipeline/SKILL.md | 17 ++++++++++++----- .claude/skills/babysit-pipeline/run_spec.sh | 7 +++++-- CHANGELOG.md | 2 +- 3 files changed, 18 insertions(+), 8 deletions(-) diff --git a/.claude/skills/babysit-pipeline/SKILL.md b/.claude/skills/babysit-pipeline/SKILL.md index e70ca0cbde..817fb0f594 100644 --- a/.claude/skills/babysit-pipeline/SKILL.md +++ b/.claude/skills/babysit-pipeline/SKILL.md @@ -167,10 +167,17 @@ the reason, move on, sweep at the end. Check the run list before deferring, though: since #10627 the workflow spends its own three attempts per campaign, so a pair that comes back missing may already have failed three times, and the -manual retry adds nothing. `gh run list --workflow=impl-generate.yml` -filtered to the spec tells you which case you are in — three -`Generate: for ` failures minutes apart is a measured -gap, one failure is a flake worth the retry. The same check covers +manual retry adds nothing. There is no per-spec filter on the API, so +filter the output by run title: + +```bash +gh run list --workflow=impl-generate.yml --limit 40 \ + --json conclusion,createdAt,displayTitle \ + --jq '.[] | select(.displayTitle | test("")) | "\(.createdAt) \(.conclusion) \(.displayTitle)"' +``` + +Three `Generate: for ` failures minutes apart is a measured +gap; one failure is a flake worth the retry. The same check covers `RESULT=TIMEOUT` with `recent generate failures: 0`, which only means the failures fell outside the driver's 25-minute window, not that nothing failed. @@ -214,7 +221,7 @@ nothing failed. when three attempts under a fresh campaign budget say so, never because the pairing sounds implausible. - **A spec id from an issue title may not exist.** `impl:*:failed` - labels outlive their specs: of 26 spec ids harvested that way, 14 + labels outlive their specs: of 26 spec IDs harvested that way, 14 had no `plots//` on main at all, and a dispatch for one dies at `Validate specification exists` seconds in. Intersect any label-derived list with `git ls-tree -d origin/main plots/` before diff --git a/.claude/skills/babysit-pipeline/run_spec.sh b/.claude/skills/babysit-pipeline/run_spec.sh index 2a0cc115ae..3591e310cc 100755 --- a/.claude/skills/babysit-pipeline/run_spec.sh +++ b/.claude/skills/babysit-pipeline/run_spec.sh @@ -82,12 +82,15 @@ recent_generate_failures() { # stale, which makes meta_present understate progress — so retry, and say so in # the log when all three attempts lose, rather than silently polling a stale ref. fetch_main() { - local i + local i err for i in 1 2 3; do - git -C "$REPO" fetch origin main --quiet 2>/dev/null && return 0 + # Keep stderr: a ref-lock collision and, say, an auth or network failure + # need different responses, and discarding the message hides which it was. + err=$(git -C "$REPO" fetch origin main --quiet 2>&1) && return 0 sleep $(( i * 3 )) done echo "[warn] git fetch origin main failed 3x — origin/main may be stale this round" | tee -a "$LOG" + echo "[warn] last error: ${err}" | tee -a "$LOG" return 1 } diff --git a/CHANGELOG.md b/CHANGELOG.md index 388af3e583..b6ec19a528 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -189,7 +189,7 @@ aggregate instead: an italic *Catalog* line at the end of the version section an gotcha is replaced by its opposite: every category-level prediction made during the backfill was wrong — 17 of 20 parked pairs generated fine, and `bar-3d-categorical` succeeded in plotnine while `scatter-3d` did not — so a gap counts as real only after - three attempts under a fresh budget. Added: spec ids harvested from issue titles must be + three attempts under a fresh budget. Added: spec IDs harvested from issue titles must be intersected with the actual `plots/` directories (14 of 26 pointed at specs that no longer exist), and a `TIMEOUT` with zero recent failures means the failures aged out of the driver's window, not that nothing failed. `run_spec.sh` also retries its From c23a8bd2b016aca96dcbb987d354a0cbda599060 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Wed, 26 Aug 2026 08:11:05 +0200 Subject: [PATCH 5/5] docs(babysit-pipeline): make the halt threshold purely pair-based MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The bullet kept a "≥5 failed runs" clause and then said "count pairs, not runs" — contradictory, and the kind of contradiction that gets resolved the wrong way during an actual outage. The condition is now just the pair count, with the run count named as the thing that does NOT decide it and the reason why. Reported by the Copilot reviewer on #10660. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01RbZuWNDFy7kjXh9kLfA4dP --- .claude/skills/babysit-pipeline/SKILL.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/.claude/skills/babysit-pipeline/SKILL.md b/.claude/skills/babysit-pipeline/SKILL.md index 817fb0f594..f3021e206d 100644 --- a/.claude/skills/babysit-pipeline/SKILL.md +++ b/.claude/skills/babysit-pipeline/SKILL.md @@ -102,13 +102,13 @@ with evidence — never let the user ask "still running?". - **Defer, don't halt**: one stuck library must not stop a multi-spec queue — log it, cap ~90 min/spec, move on, and sweep leftovers at the end with targeted regens. -- **Halt the whole queue** only on a failure CLUSTER — ≥5 failed - `Generate:` runs in minutes across **≥3 distinct (spec, library) - pairs** = model daily quota exhausted; a fallback model via - `-f model=` can finish leftovers. Count pairs, not runs: one - genuinely impossible pair burns three runs on its own retries, so - two bad pairs alone trip a raw run count (observed 2026-08-24 — - 6 failures, 2 pairs, pipeline entirely healthy). +- **Halt the whole queue** only on a failure CLUSTER: **≥3 distinct + (spec, library) pairs** failing within minutes of each other = model + daily quota exhausted; a fallback model via `-f model=` can finish + leftovers. The threshold is pairs, full stop — the raw run count + says nothing, because one impossible pair spends three runs on its + own auto-retries, so two bad pairs already produce six failures with + the pipeline entirely healthy (observed 2026-08-24). - Closing/merging pipeline PRs or issues yourself is out of bounds (CLAUDE.md external-write rule + mandatory workflow).