From c7b6174307aebf48d208cd26d3cfbc3d714fd71b Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Mon, 24 Aug 2026 22:41:13 +0200 Subject: [PATCH 1/2] docs(babysit-pipeline): add gap-backfill mode and the traps it exposed MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Backfilling coverage gaps across the catalogue is a different job from babysitting one fresh spec, and the skill only covered the latter. New section 5 documents it: compute the missing set from plots/{spec}/metadata/ rather than from impl:{lib}:done labels (most of those specs have closed issues, so the labels are absent or stale), drive one spec per invocation with run_spec.sh, run two specs in parallel, keep a done/deferred ledger under agentic/runs/, and give every missing library exactly one retry before deferring it. Four gotchas from the 2026-08-24 backfill, each of which cost or nearly cost real implementations: - `Marking as failed: N generation attempts` counts markers from more than the current run; check the `Previous failures` notice before reading it as a capability gap. - `impl::failed` is terminal (nothing re-dispatches it) and stale — 42 of 87 such labels sat on implementations that had since landed. - "Agent reports success, writes no file" is an intermittent ~9% failure that self-heals when retry budget remains; one occurrence means nothing. - Static library against an interactive or 3D spec is the gap shape that usually is genuine. run_spec.sh joins the bundled scripts, with its hardcoded repo path replaced by resolution from the script's own location (ANYPLOT_REPO overrides) so it works from any checkout. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01RbZuWNDFy7kjXh9kLfA4dP --- .claude/skills/babysit-pipeline/SKILL.md | 80 ++++++++++++++- .claude/skills/babysit-pipeline/run_spec.sh | 106 ++++++++++++++++++++ CHANGELOG.md | 10 ++ 3 files changed, 194 insertions(+), 2 deletions(-) create mode 100755 .claude/skills/babysit-pipeline/run_spec.sh diff --git a/.claude/skills/babysit-pipeline/SKILL.md b/.claude/skills/babysit-pipeline/SKILL.md index 30bc2635c5..3829f23bc1 100644 --- a/.claude/skills/babysit-pipeline/SKILL.md +++ b/.claude/skills/babysit-pipeline/SKILL.md @@ -1,6 +1,6 @@ --- name: babysit-pipeline -description: Run and monitor the bulk-generate / impl-* pipeline for one or more specs to full 15-library coverage — dispatch sequentially, poll with the bundled checked-in scripts (never hand-rolled loops), read completion from impl:{lib}:done labels + GCS spot-checks, apply the correct stall thresholds, and report progress proactively. Use when asked to babysit, run bulk-generate, regenerate specs, monitor the pipeline, or bring specs to full coverage. +description: Run and monitor the bulk-generate / impl-* pipeline for one or more specs to full 15-library coverage — dispatch sequentially, poll with the bundled checked-in scripts (never hand-rolled loops), read completion from impl:{lib}:done labels + GCS spot-checks, apply the correct stall thresholds, and report progress proactively. Also covers gap backfill across the whole catalogue (computing the missing set from repo metadata, the per-spec driver, the one-retry rule). Use when asked to babysit, run bulk-generate, regenerate specs, monitor the pipeline, close coverage gaps, or bring specs to full coverage. --- # Babysit the generation pipeline @@ -105,11 +105,87 @@ with evidence — never let the user ask "still running?". - Closing/merging pipeline PRs or issues yourself is out of bounds (CLAUDE.md external-write rule + mandatory workflow). +## 5 · Gap backfill (many specs, partial coverage) + +Different job from babysitting one fresh spec: dozens of older specs +each miss a few libraries, usually the ones added after they were +generated (JS libs, muix, makie). + +**Compute the missing set from repo metadata, never from labels.** +Most of those specs have closed issues, so `impl:{lib}:done` labels +are absent or stale. `plots/{spec}/metadata/{lang}/{lib}.yaml` on +`origin/main` is the durable signal — it is also what +`run_spec.sh` polls. + +```bash +git fetch origin main +git ls-tree --name-only -r origin/main -- plots/ \ + | awk -F/ '$3=="metadata" && NF==5 {f=$5; sub(/\.[^.]+$/,"",f); print $2" "f}' \ + | sort -u # → " " pairs that exist +``` + +Diff that against the 15-library registry per spec dir, write +` ` sorted fewest-missing-first, and work +the file top-down: the cheap specs finish early and the progress +number moves. + +**One spec per driver invocation** — +`.claude/skills/babysit-pipeline/run_spec.sh ...` +dispatches that spec's missing libraries staggered ~150 s +apart, then polls until each metadata file lands, and exits with +`RESULT=COMPLETE|PARTIAL|TIMEOUT`. Run it via Bash +`run_in_background: true`; it skips libraries already on main, so +re-running it after a partial result retries exactly the gaps. + +**Two specs in parallel is fine** (~4 specs/h vs ~2); ten concurrent +`impl-generate` runs showed no rate-limit effects. Keep a ledger — +`done.log` / `deferred.log` next to the queue file in `agentic/runs/` +— and append the result line before dispatching the next spec, so a +compaction or a crashed session can resume without recounting. + +**The one-retry rule.** A library that comes back missing gets +exactly one fresh targeted dispatch before it is deferred. This is +not optional politeness — on 2026-08-24 the single retry recovered +highcharts/treemap-basic, ggplot2/wireframe-3d-basic and +ggplot2/network-force-directed, all three of which had been read as +capability gaps. Two failures on the same pair → `deferred.log` with +the reason, move on, sweep at the end. + ## Gotchas +- **`Marking as failed: N generation attempts` counts more than + this run.** The number comes from `` + marker comments on the issue, and nothing deletes them; before the + campaign-window fix (#10627) it spanned the issue's whole lifetime, + so a pair that failed twice in some old campaign was capped at ONE + attempt forever. Read the `Previous failures for /: N` + notice in the log before concluding a library "can't do" a plot + type — a high N there means the cap fired, not that generation was + tried three times today. +- **`impl::failed` is terminal and it lies about coverage.** + Nothing re-dispatches it: watchdog case 3 fires once, then only + logs `already retried by watchdog — needs manual attention`. Audit + it against metadata before trusting it — of 87 such labels on + 2026-08-24, **42 sat on implementations that had since landed**. + Take the label as a hint to check, never as the coverage answer. +- **"Agent reports success, writes no file"** is a live intermittent + failure (8 of 85 generate runs on 2026-08-24, ~9%): the Claude step + ends `"subtype":"success","is_error":false` and the next step fails + with `Implementation file not found in repository`. With retry + budget left the workflow self-heals (`bubble-basic/highcharts`: + failed 17:55, retried and succeeded 18:02, no human involved), so a + single occurrence is noise — only a repeat on the same pair means + anything. +- **Static library + interactive/3D spec is the one gap that is + usually real.** plotnine, pygal, seaborn, matplotlib and ggplot2 + against `*-interactive`, `*-drilldown`, `*-realtime`, `*-streaming`, + `slider-*`, `*-3d` specs make up most genuine dead ends; 18 of the + 45 real gaps behind those labels were exactly this shape. Still + give them the one retry, then defer without further ceremony. - **The scripts' library list is a copy** of `core/constants.py`'s registry (15 libs). When a library is added/removed, update - `monitor_spec.sh`'s `ALL_LIBS`/`lang_of` in the same PR. + `monitor_spec.sh`'s `ALL_LIBS` and `run_spec.sh`'s `lang_of` in the + same PR. - **`gsutil` must be authenticated** for the GCS spot-check; a credentials failure reads as "incomplete" — check `gsutil ls gs://anyplot-images/ | head -1` before trusting a diff --git a/.claude/skills/babysit-pipeline/run_spec.sh b/.claude/skills/babysit-pipeline/run_spec.sh new file mode 100755 index 0000000000..091eaeace5 --- /dev/null +++ b/.claude/skills/babysit-pipeline/run_spec.sh @@ -0,0 +1,106 @@ +#!/usr/bin/env bash +# Per-SPEC driver: dispatch impl-generate for every missing library of one +# spec, staggered so the pipeline overlaps generate/review/merge across libs +# (the bulk-generate pacing pattern), then poll origin/main until each lib's +# metadata file exists. Repo metadata is the completion signal — works for +# specs whose issues closed long ago. Exits per spec so the agent can report +# and start the next queue entry. +# Usage: run_spec.sh [lib2 ...] +set -uo pipefail +SPEC="$1"; MODEL="$2"; shift 2; LIBS=("$@") +# Resolve the repo from this script's location (.claude/skills//), so the +# driver works from any checkout and any working directory. ANYPLOT_REPO wins +# when the script is copied elsewhere (e.g. a scratch queue under agentic/runs/). +HERE="$(cd "$(dirname "$0")" && pwd)" +REPO="${ANYPLOT_REPO:-$(git -C "$HERE" rev-parse --show-toplevel 2>/dev/null || echo "$PWD")}" +STAGGER=150 # seconds between dispatches +INTERVAL=180 # poll interval +LOGDIR="${POLL_LOG_DIR:-$HOME/.cache/anyplot-babysit}" +mkdir -p "$LOGDIR" +LOG="$LOGDIR/spec_${SPEC}.log" + +lang_of() { + case "$1" in + ggplot2) echo r ;; + makie) echo julia ;; + chartjs|d3|echarts|highcharts|muix) echo javascript ;; + *) echo python ;; + esac +} + +meta_present() { # assumes a fresh `git fetch` already ran this iteration + local lang; lang=$(lang_of "$1") + [ -n "$(git -C "$REPO" ls-tree --name-only origin/main -- "plots/${SPEC}/metadata/${lang}/$1.yaml" 2>/dev/null)" ] +} + +pipeline_active() { + for wf in impl-generate.yml impl-review.yml impl-repair.yml impl-merge.yml impl-review-retry.yml; do + local out + if ! out=$(gh run list --workflow="$wf" --limit 12 --json status \ + --jq '.[] | select(.status=="in_progress" or .status=="queued") | .status' 2>&1); then + printf 'WARN: gh run list %s failed; assuming active: %s\n' "$wf" "$out" >> "$LOG" + return 0 + fi + grep -q . <<<"$out" && return 0 + done + return 1 +} + +recent_generate_failures() { + gh run list --workflow=impl-generate.yml --limit 15 \ + --json conclusion,updatedAt \ + --jq "[.[] | select(.conclusion==\"failure\" and (.updatedAt > (now - 1500 | todate)))] | length" 2>/dev/null || echo 0 +} + +git -C "$REPO" fetch origin main --quiet +TODO=() +for lib in "${LIBS[@]}"; do + if meta_present "$lib"; then + echo "[skip] $SPEC/$lib already on main" | tee -a "$LOG" + else + TODO+=("$lib") + fi +done +if [ ${#TODO[@]} -eq 0 ]; then + echo "RESULT=COMPLETE spec=$SPEC (all libs already present)" + exit 0 +fi + +for i in "${!TODO[@]}"; do + lib="${TODO[$i]}" + echo "[$(TZ=UTC date -u +%H:%M:%S)] dispatch $SPEC/$lib model=$MODEL" | tee -a "$LOG" + if ! gh workflow run impl-generate.yml \ + -f "specification_id=$SPEC" -f "library=$lib" -f "model=$MODEL" >> "$LOG" 2>&1; then + echo "WARN: dispatch failed for $SPEC/$lib" | tee -a "$LOG" + fi + [ "$i" -lt $(( ${#TODO[@]} - 1 )) ] && sleep "$STAGGER" +done + +MAXMIN=$(( 30 + 15 * ${#TODO[@]} )); [ "$MAXMIN" -gt 150 ] && MAXMIN=150 +iters=$(( MAXMIN * 60 / INTERVAL )) +idle=0; last_done=-1 +for ((i=1; i<=iters; i++)); do + sleep "$INTERVAL" + git -C "$REPO" fetch origin main --quiet + missing=(); done_n=0 + for lib in "${TODO[@]}"; do + if meta_present "$lib"; then done_n=$((done_n+1)); else missing+=("$lib"); fi + done + if [ ${#missing[@]} -eq 0 ]; then + echo "RESULT=COMPLETE spec=$SPEC libs=${TODO[*]} after ~$(( (i*INTERVAL)/60 + (${#TODO[@]}-1)*STAGGER/60 )) min" + exit 0 + fi + if [ "$done_n" -gt "$last_done" ]; then idle=0; last_done=$done_n + elif pipeline_active; then idle=0 + else idle=$((idle+1)); fi + printf '[%s iter %d/%d] %d/%d done, missing: %s (idle=%d)\n' \ + "$(TZ=UTC date -u +%H:%M:%S)" "$i" "$iters" "$done_n" "${#TODO[@]}" "${missing[*]}" "$idle" >> "$LOG" + if [ "$idle" -ge 3 ]; then + echo "RESULT=PARTIAL spec=$SPEC done=$done_n/${#TODO[@]} still-missing: ${missing[*]} — pipeline idle ~$((3*INTERVAL/60)) min" + echo "recent generate failures (25 min): $(recent_generate_failures)" + exit 0 + fi +done +echo "RESULT=TIMEOUT spec=$SPEC done=$last_done/${#TODO[@]} still-missing: ${missing[*]} after ${MAXMIN} min" +echo "recent generate failures (25 min): $(recent_generate_failures)" +exit 0 diff --git a/CHANGELOG.md b/CHANGELOG.md index 81b7287026..380c68f3c6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -158,6 +158,16 @@ aggregate instead: an italic *Catalog* line at the end of the version section an ### Changed +- **`babysit-pipeline` learned gap backfill** — closing coverage gaps across the catalogue is + a different job from watching one fresh spec, and the skill only described the latter. It + now documents reading the missing set from `plots/*/metadata/` instead of from labels + (most affected specs have closed issues, so their labels are absent or stale), the + per-spec `run_spec.sh` driver, two-specs-in-parallel pacing, and the one-retry rule. Four + new gotchas record traps that cost real implementations during the 2026-08-24 backfill: + the failure-marker count spanning more than the current run, `impl::failed` being + both terminal and unreliable (42 of 87 sat on implementations that had since landed), the + intermittent "reports success, writes no file" generation failure, and the one gap shape + that usually is real — a static library against an interactive or 3D spec (#10628). - **llms.txt now tells agents how to actually fetch things** — the file linked nine human-facing HTML pages and named no machine endpoint. New sections document the REST API (base URL, the retrieval endpoints, OpenAPI), the GCS render URL pattern (themes, responsive widths, WebP), From 6ccae869570af1dfbc7ccc2cf467b68b21052d26 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Mon, 24 Aug 2026 22:47:29 +0200 Subject: [PATCH 2/2] fix(babysit-pipeline): guard run_spec.sh inputs, scope the parallelism claim MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three points from the Copilot review on #10628: - run_spec.sh took its arguments under `set -u` with no arity check, so a missing library surfaced as an unbound-variable error from inside the polling loop. Explicit usage guard instead. - Repo resolution fell back to $PWD. An unvalidated root makes every meta_present check return false, which reads as "nothing ever landed" and burns the full timeout; now it fails fast and names ANYPLOT_REPO. - "Two specs in parallel is fine" contradicted §0 and §3. It holds only for run_spec.sh, which reads completion per library from origin/main metadata; poll_spec.sh and monitor_spec.sh infer progress from active runs and would call a stalled spec healthy with a second spec in flight. Both sections now say so. Also raise the generate-failure counter's run limit from 15 to 60 and report `?` instead of 0 on API error: 15 no longer covers the 25-minute window once two specs are in flight, and both defects fail in the same direction — silently reporting a healthy pipeline. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01RbZuWNDFy7kjXh9kLfA4dP --- .claude/skills/babysit-pipeline/SKILL.md | 15 ++++++++-- .claude/skills/babysit-pipeline/run_spec.sh | 31 +++++++++++++++++++-- 2 files changed, 40 insertions(+), 6 deletions(-) diff --git a/.claude/skills/babysit-pipeline/SKILL.md b/.claude/skills/babysit-pipeline/SKILL.md index 3829f23bc1..f26aa105ef 100644 --- a/.claude/skills/babysit-pipeline/SKILL.md +++ b/.claude/skills/babysit-pipeline/SKILL.md @@ -20,7 +20,10 @@ Never manually merge pipeline PRs, never hand-create metadata stale main has produced wrong counts. - One spec at a time: bulk-generate serializes dispatch via a concurrency group, but overlapping impl pipelines across specs make - completion unreadable. Process strictly sequentially. + completion unreadable. Process strictly sequentially. (Backfill with + `run_spec.sh` is the one exception — it reads completion per library + from repo metadata rather than from "is anything running?", so it + tolerates two specs in flight. See §5.) ## 1 · Dispatch @@ -137,8 +140,14 @@ apart, then polls until each metadata file lands, and exits with `run_in_background: true`; it skips libraries already on main, so re-running it after a partial result retries exactly the gaps. -**Two specs in parallel is fine** (~4 specs/h vs ~2); ten concurrent -`impl-generate` runs showed no rate-limit effects. Keep a ledger — +**Two specs in parallel is fine — but only in this mode** (~4 specs/h +vs ~2); ten concurrent `impl-generate` runs showed no rate-limit +effects. It works because `run_spec.sh` decides per library from +`origin/main` metadata. `poll_spec.sh` and `monitor_spec.sh` must +still run one spec at a time: their stall logic reads *any* active +`impl-*` run as belonging to the spec they are watching (§3), so a +second spec in flight makes them call a stalled spec healthy. Never +mix the two modes on the same queue. Keep a ledger — `done.log` / `deferred.log` next to the queue file in `agentic/runs/` — and append the result line before dispatching the next spec, so a compaction or a crashed session can resume without recounting. diff --git a/.claude/skills/babysit-pipeline/run_spec.sh b/.claude/skills/babysit-pipeline/run_spec.sh index 091eaeace5..7a8c8cec8f 100755 --- a/.claude/skills/babysit-pipeline/run_spec.sh +++ b/.claude/skills/babysit-pipeline/run_spec.sh @@ -7,12 +7,30 @@ # and start the next queue entry. # Usage: run_spec.sh [lib2 ...] set -uo pipefail + +usage() { + echo "usage: $(basename "$0") [lib2 ...]" >&2 + echo " e.g. $(basename "$0") line-basic sonnet highcharts muix" >&2 + exit 2 +} +# Explicit guard: with `set -u` a missing argument would otherwise surface as an +# unbound-variable error from somewhere deep in the polling loop. +[ "$#" -ge 3 ] || usage SPEC="$1"; MODEL="$2"; shift 2; LIBS=("$@") + # Resolve the repo from this script's location (.claude/skills//), so the # driver works from any checkout and any working directory. ANYPLOT_REPO wins # when the script is copied elsewhere (e.g. a scratch queue under agentic/runs/). +# Fail fast rather than falling back to $PWD: an unvalidated repo makes every +# `meta_present` check return false, which reads as "nothing ever landed" and +# burns the full polling timeout before anyone notices. HERE="$(cd "$(dirname "$0")" && pwd)" -REPO="${ANYPLOT_REPO:-$(git -C "$HERE" rev-parse --show-toplevel 2>/dev/null || echo "$PWD")}" +REPO="${ANYPLOT_REPO:-$(git -C "$HERE" rev-parse --show-toplevel 2>/dev/null || true)}" +if [ -z "$REPO" ] || [ ! -d "$REPO/plots" ]; then + echo "error: could not resolve the anyplot repo root (tried \$ANYPLOT_REPO, then git from $HERE)." >&2 + echo " set ANYPLOT_REPO=/path/to/anyplot and re-run." >&2 + exit 2 +fi STAGGER=150 # seconds between dispatches INTERVAL=180 # poll interval LOGDIR="${POLL_LOG_DIR:-$HOME/.cache/anyplot-babysit}" @@ -46,10 +64,17 @@ pipeline_active() { return 1 } +# Rough health signal for the report line: how many generate runs failed in the +# last 25 min. The limit must cover the whole window — two specs in flight can +# put 10+ runs in it, and a short limit silently reports "0 failures" because +# the failures fell off the end of the list rather than because there were none. +# `?` on error, never 0: a masked API failure reading as "all healthy" is how a +# quota outage gets mistaken for a slow queue. recent_generate_failures() { - gh run list --workflow=impl-generate.yml --limit 15 \ + gh run list --workflow=impl-generate.yml --limit 60 \ --json conclusion,updatedAt \ - --jq "[.[] | select(.conclusion==\"failure\" and (.updatedAt > (now - 1500 | todate)))] | length" 2>/dev/null || echo 0 + --jq "[.[] | select(.conclusion==\"failure\" and (.updatedAt > (now - 1500 | todate)))] | length" 2>/dev/null \ + || echo "?" } git -C "$REPO" fetch origin main --quiet