diff --git a/.claude/skills/babysit-pipeline/SKILL.md b/.claude/skills/babysit-pipeline/SKILL.md index 30bc2635c5..f26aa105ef 100644 --- a/.claude/skills/babysit-pipeline/SKILL.md +++ b/.claude/skills/babysit-pipeline/SKILL.md @@ -1,6 +1,6 @@ --- name: babysit-pipeline -description: Run and monitor the bulk-generate / impl-* pipeline for one or more specs to full 15-library coverage — dispatch sequentially, poll with the bundled checked-in scripts (never hand-rolled loops), read completion from impl:{lib}:done labels + GCS spot-checks, apply the correct stall thresholds, and report progress proactively. Use when asked to babysit, run bulk-generate, regenerate specs, monitor the pipeline, or bring specs to full coverage. +description: Run and monitor the bulk-generate / impl-* pipeline for one or more specs to full 15-library coverage — dispatch sequentially, poll with the bundled checked-in scripts (never hand-rolled loops), read completion from impl:{lib}:done labels + GCS spot-checks, apply the correct stall thresholds, and report progress proactively. Also covers gap backfill across the whole catalogue (computing the missing set from repo metadata, the per-spec driver, the one-retry rule). Use when asked to babysit, run bulk-generate, regenerate specs, monitor the pipeline, close coverage gaps, or bring specs to full coverage. --- # Babysit the generation pipeline @@ -20,7 +20,10 @@ Never manually merge pipeline PRs, never hand-create metadata stale main has produced wrong counts. - One spec at a time: bulk-generate serializes dispatch via a concurrency group, but overlapping impl pipelines across specs make - completion unreadable. Process strictly sequentially. + completion unreadable. Process strictly sequentially. (Backfill with + `run_spec.sh` is the one exception — it reads completion per library + from repo metadata rather than from "is anything running?", so it + tolerates two specs in flight. See §5.) ## 1 · Dispatch @@ -105,11 +108,93 @@ with evidence — never let the user ask "still running?". - Closing/merging pipeline PRs or issues yourself is out of bounds (CLAUDE.md external-write rule + mandatory workflow). +## 5 · Gap backfill (many specs, partial coverage) + +Different job from babysitting one fresh spec: dozens of older specs +each miss a few libraries, usually the ones added after they were +generated (JS libs, muix, makie). + +**Compute the missing set from repo metadata, never from labels.** +Most of those specs have closed issues, so `impl:{lib}:done` labels +are absent or stale. `plots/{spec}/metadata/{lang}/{lib}.yaml` on +`origin/main` is the durable signal — it is also what +`run_spec.sh` polls. + +```bash +git fetch origin main +git ls-tree --name-only -r origin/main -- plots/ \ + | awk -F/ '$3=="metadata" && NF==5 {f=$5; sub(/\.[^.]+$/,"",f); print $2" "f}' \ + | sort -u # → " " pairs that exist +``` + +Diff that against the 15-library registry per spec dir, write +` ` sorted fewest-missing-first, and work +the file top-down: the cheap specs finish early and the progress +number moves. + +**One spec per driver invocation** — +`.claude/skills/babysit-pipeline/run_spec.sh ...` +dispatches that spec's missing libraries staggered ~150 s +apart, then polls until each metadata file lands, and exits with +`RESULT=COMPLETE|PARTIAL|TIMEOUT`. Run it via Bash +`run_in_background: true`; it skips libraries already on main, so +re-running it after a partial result retries exactly the gaps. + +**Two specs in parallel is fine — but only in this mode** (~4 specs/h +vs ~2); ten concurrent `impl-generate` runs showed no rate-limit +effects. It works because `run_spec.sh` decides per library from +`origin/main` metadata. `poll_spec.sh` and `monitor_spec.sh` must +still run one spec at a time: their stall logic reads *any* active +`impl-*` run as belonging to the spec they are watching (§3), so a +second spec in flight makes them call a stalled spec healthy. Never +mix the two modes on the same queue. Keep a ledger — +`done.log` / `deferred.log` next to the queue file in `agentic/runs/` +— and append the result line before dispatching the next spec, so a +compaction or a crashed session can resume without recounting. + +**The one-retry rule.** A library that comes back missing gets +exactly one fresh targeted dispatch before it is deferred. This is +not optional politeness — on 2026-08-24 the single retry recovered +highcharts/treemap-basic, ggplot2/wireframe-3d-basic and +ggplot2/network-force-directed, all three of which had been read as +capability gaps. Two failures on the same pair → `deferred.log` with +the reason, move on, sweep at the end. + ## Gotchas +- **`Marking as failed: N generation attempts` counts more than + this run.** The number comes from `` + marker comments on the issue, and nothing deletes them; before the + campaign-window fix (#10627) it spanned the issue's whole lifetime, + so a pair that failed twice in some old campaign was capped at ONE + attempt forever. Read the `Previous failures for /: N` + notice in the log before concluding a library "can't do" a plot + type — a high N there means the cap fired, not that generation was + tried three times today. +- **`impl::failed` is terminal and it lies about coverage.** + Nothing re-dispatches it: watchdog case 3 fires once, then only + logs `already retried by watchdog — needs manual attention`. Audit + it against metadata before trusting it — of 87 such labels on + 2026-08-24, **42 sat on implementations that had since landed**. + Take the label as a hint to check, never as the coverage answer. +- **"Agent reports success, writes no file"** is a live intermittent + failure (8 of 85 generate runs on 2026-08-24, ~9%): the Claude step + ends `"subtype":"success","is_error":false` and the next step fails + with `Implementation file not found in repository`. With retry + budget left the workflow self-heals (`bubble-basic/highcharts`: + failed 17:55, retried and succeeded 18:02, no human involved), so a + single occurrence is noise — only a repeat on the same pair means + anything. +- **Static library + interactive/3D spec is the one gap that is + usually real.** plotnine, pygal, seaborn, matplotlib and ggplot2 + against `*-interactive`, `*-drilldown`, `*-realtime`, `*-streaming`, + `slider-*`, `*-3d` specs make up most genuine dead ends; 18 of the + 45 real gaps behind those labels were exactly this shape. Still + give them the one retry, then defer without further ceremony. - **The scripts' library list is a copy** of `core/constants.py`'s registry (15 libs). When a library is added/removed, update - `monitor_spec.sh`'s `ALL_LIBS`/`lang_of` in the same PR. + `monitor_spec.sh`'s `ALL_LIBS` and `run_spec.sh`'s `lang_of` in the + same PR. - **`gsutil` must be authenticated** for the GCS spot-check; a credentials failure reads as "incomplete" — check `gsutil ls gs://anyplot-images/ | head -1` before trusting a diff --git a/.claude/skills/babysit-pipeline/run_spec.sh b/.claude/skills/babysit-pipeline/run_spec.sh new file mode 100755 index 0000000000..7a8c8cec8f --- /dev/null +++ b/.claude/skills/babysit-pipeline/run_spec.sh @@ -0,0 +1,131 @@ +#!/usr/bin/env bash +# Per-SPEC driver: dispatch impl-generate for every missing library of one +# spec, staggered so the pipeline overlaps generate/review/merge across libs +# (the bulk-generate pacing pattern), then poll origin/main until each lib's +# metadata file exists. Repo metadata is the completion signal — works for +# specs whose issues closed long ago. Exits per spec so the agent can report +# and start the next queue entry. +# Usage: run_spec.sh [lib2 ...] +set -uo pipefail + +usage() { + echo "usage: $(basename "$0") [lib2 ...]" >&2 + echo " e.g. $(basename "$0") line-basic sonnet highcharts muix" >&2 + exit 2 +} +# Explicit guard: with `set -u` a missing argument would otherwise surface as an +# unbound-variable error from somewhere deep in the polling loop. +[ "$#" -ge 3 ] || usage +SPEC="$1"; MODEL="$2"; shift 2; LIBS=("$@") + +# Resolve the repo from this script's location (.claude/skills//), so the +# driver works from any checkout and any working directory. ANYPLOT_REPO wins +# when the script is copied elsewhere (e.g. a scratch queue under agentic/runs/). +# Fail fast rather than falling back to $PWD: an unvalidated repo makes every +# `meta_present` check return false, which reads as "nothing ever landed" and +# burns the full polling timeout before anyone notices. +HERE="$(cd "$(dirname "$0")" && pwd)" +REPO="${ANYPLOT_REPO:-$(git -C "$HERE" rev-parse --show-toplevel 2>/dev/null || true)}" +if [ -z "$REPO" ] || [ ! -d "$REPO/plots" ]; then + echo "error: could not resolve the anyplot repo root (tried \$ANYPLOT_REPO, then git from $HERE)." >&2 + echo " set ANYPLOT_REPO=/path/to/anyplot and re-run." >&2 + exit 2 +fi +STAGGER=150 # seconds between dispatches +INTERVAL=180 # poll interval +LOGDIR="${POLL_LOG_DIR:-$HOME/.cache/anyplot-babysit}" +mkdir -p "$LOGDIR" +LOG="$LOGDIR/spec_${SPEC}.log" + +lang_of() { + case "$1" in + ggplot2) echo r ;; + makie) echo julia ;; + chartjs|d3|echarts|highcharts|muix) echo javascript ;; + *) echo python ;; + esac +} + +meta_present() { # assumes a fresh `git fetch` already ran this iteration + local lang; lang=$(lang_of "$1") + [ -n "$(git -C "$REPO" ls-tree --name-only origin/main -- "plots/${SPEC}/metadata/${lang}/$1.yaml" 2>/dev/null)" ] +} + +pipeline_active() { + for wf in impl-generate.yml impl-review.yml impl-repair.yml impl-merge.yml impl-review-retry.yml; do + local out + if ! out=$(gh run list --workflow="$wf" --limit 12 --json status \ + --jq '.[] | select(.status=="in_progress" or .status=="queued") | .status' 2>&1); then + printf 'WARN: gh run list %s failed; assuming active: %s\n' "$wf" "$out" >> "$LOG" + return 0 + fi + grep -q . <<<"$out" && return 0 + done + return 1 +} + +# Rough health signal for the report line: how many generate runs failed in the +# last 25 min. The limit must cover the whole window — two specs in flight can +# put 10+ runs in it, and a short limit silently reports "0 failures" because +# the failures fell off the end of the list rather than because there were none. +# `?` on error, never 0: a masked API failure reading as "all healthy" is how a +# quota outage gets mistaken for a slow queue. +recent_generate_failures() { + gh run list --workflow=impl-generate.yml --limit 60 \ + --json conclusion,updatedAt \ + --jq "[.[] | select(.conclusion==\"failure\" and (.updatedAt > (now - 1500 | todate)))] | length" 2>/dev/null \ + || echo "?" +} + +git -C "$REPO" fetch origin main --quiet +TODO=() +for lib in "${LIBS[@]}"; do + if meta_present "$lib"; then + echo "[skip] $SPEC/$lib already on main" | tee -a "$LOG" + else + TODO+=("$lib") + fi +done +if [ ${#TODO[@]} -eq 0 ]; then + echo "RESULT=COMPLETE spec=$SPEC (all libs already present)" + exit 0 +fi + +for i in "${!TODO[@]}"; do + lib="${TODO[$i]}" + echo "[$(TZ=UTC date -u +%H:%M:%S)] dispatch $SPEC/$lib model=$MODEL" | tee -a "$LOG" + if ! gh workflow run impl-generate.yml \ + -f "specification_id=$SPEC" -f "library=$lib" -f "model=$MODEL" >> "$LOG" 2>&1; then + echo "WARN: dispatch failed for $SPEC/$lib" | tee -a "$LOG" + fi + [ "$i" -lt $(( ${#TODO[@]} - 1 )) ] && sleep "$STAGGER" +done + +MAXMIN=$(( 30 + 15 * ${#TODO[@]} )); [ "$MAXMIN" -gt 150 ] && MAXMIN=150 +iters=$(( MAXMIN * 60 / INTERVAL )) +idle=0; last_done=-1 +for ((i=1; i<=iters; i++)); do + sleep "$INTERVAL" + git -C "$REPO" fetch origin main --quiet + missing=(); done_n=0 + for lib in "${TODO[@]}"; do + if meta_present "$lib"; then done_n=$((done_n+1)); else missing+=("$lib"); fi + done + if [ ${#missing[@]} -eq 0 ]; then + echo "RESULT=COMPLETE spec=$SPEC libs=${TODO[*]} after ~$(( (i*INTERVAL)/60 + (${#TODO[@]}-1)*STAGGER/60 )) min" + exit 0 + fi + if [ "$done_n" -gt "$last_done" ]; then idle=0; last_done=$done_n + elif pipeline_active; then idle=0 + else idle=$((idle+1)); fi + printf '[%s iter %d/%d] %d/%d done, missing: %s (idle=%d)\n' \ + "$(TZ=UTC date -u +%H:%M:%S)" "$i" "$iters" "$done_n" "${#TODO[@]}" "${missing[*]}" "$idle" >> "$LOG" + if [ "$idle" -ge 3 ]; then + echo "RESULT=PARTIAL spec=$SPEC done=$done_n/${#TODO[@]} still-missing: ${missing[*]} — pipeline idle ~$((3*INTERVAL/60)) min" + echo "recent generate failures (25 min): $(recent_generate_failures)" + exit 0 + fi +done +echo "RESULT=TIMEOUT spec=$SPEC done=$last_done/${#TODO[@]} still-missing: ${missing[*]} after ${MAXMIN} min" +echo "recent generate failures (25 min): $(recent_generate_failures)" +exit 0 diff --git a/CHANGELOG.md b/CHANGELOG.md index 81b7287026..380c68f3c6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -158,6 +158,16 @@ aggregate instead: an italic *Catalog* line at the end of the version section an ### Changed +- **`babysit-pipeline` learned gap backfill** — closing coverage gaps across the catalogue is + a different job from watching one fresh spec, and the skill only described the latter. It + now documents reading the missing set from `plots/*/metadata/` instead of from labels + (most affected specs have closed issues, so their labels are absent or stale), the + per-spec `run_spec.sh` driver, two-specs-in-parallel pacing, and the one-retry rule. Four + new gotchas record traps that cost real implementations during the 2026-08-24 backfill: + the failure-marker count spanning more than the current run, `impl::failed` being + both terminal and unreliable (42 of 87 sat on implementations that had since landed), the + intermittent "reports success, writes no file" generation failure, and the one gap shape + that usually is real — a static library against an interactive or 3D spec (#10628). - **llms.txt now tells agents how to actually fetch things** — the file linked nine human-facing HTML pages and named no machine endpoint. New sections document the REST API (base URL, the retrieval endpoints, OpenAPI), the GCS render URL pattern (themes, responsive widths, WebP),