docs(babysit-pipeline): add gap-backfill mode and the traps it exposed

.claude/skills/babysit-pipeline/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: babysit-pipeline
-description: Run and monitor the bulk-generate / impl-* pipeline for one or more specs to full 15-library coverage — dispatch sequentially, poll with the bundled checked-in scripts (never hand-rolled loops), read completion from impl:{lib}:done labels + GCS spot-checks, apply the correct stall thresholds, and report progress proactively. Use when asked to babysit, run bulk-generate, regenerate specs, monitor the pipeline, or bring specs to full coverage.
+description: Run and monitor the bulk-generate / impl-* pipeline for one or more specs to full 15-library coverage — dispatch sequentially, poll with the bundled checked-in scripts (never hand-rolled loops), read completion from impl:{lib}:done labels + GCS spot-checks, apply the correct stall thresholds, and report progress proactively. Also covers gap backfill across the whole catalogue (computing the missing set from repo metadata, the per-spec driver, the one-retry rule). Use when asked to babysit, run bulk-generate, regenerate specs, monitor the pipeline, close coverage gaps, or bring specs to full coverage.
 ---
 
 # Babysit the generation pipeline
@@ -20,7 +20,10 @@ Never manually merge pipeline PRs, never hand-create metadata
   stale main has produced wrong counts.
 - One spec at a time: bulk-generate serializes dispatch via a
   concurrency group, but overlapping impl pipelines across specs make
-  completion unreadable. Process strictly sequentially.
+  completion unreadable. Process strictly sequentially. (Backfill with
+  `run_spec.sh` is the one exception — it reads completion per library
+  from repo metadata rather than from "is anything running?", so it
+  tolerates two specs in flight. See §5.)
 
 ## 1 · Dispatch
 
@@ -105,11 +108,93 @@ with evidence — never let the user ask "still running?".
 - Closing/merging pipeline PRs or issues yourself is out of bounds
   (CLAUDE.md external-write rule + mandatory workflow).
 
+## 5 · Gap backfill (many specs, partial coverage)
+
+Different job from babysitting one fresh spec: dozens of older specs
+each miss a few libraries, usually the ones added after they were
+generated (JS libs, muix, makie).
+
+**Compute the missing set from repo metadata, never from labels.**
+Most of those specs have closed issues, so `impl:{lib}:done` labels
+are absent or stale. `plots/{spec}/metadata/{lang}/{lib}.yaml` on
+`origin/main` is the durable signal — it is also what
+`run_spec.sh` polls.
+
+```bash
+git fetch origin main
+git ls-tree --name-only -r origin/main -- plots/ \
+  | awk -F/ '$3=="metadata" && NF==5 {f=$5; sub(/\.[^.]+$/,"",f); print $2" "f}' \
+  | sort -u          # → "<spec> <lib>" pairs that exist
+```
+
+Diff that against the 15-library registry per spec dir, write
+`<n-missing> <spec> <lib...>` sorted fewest-missing-first, and work
+the file top-down: the cheap specs finish early and the progress
+number moves.
+
+**One spec per driver invocation** —
+`.claude/skills/babysit-pipeline/run_spec.sh <spec> <model> <lib>...`
+dispatches that spec's missing libraries staggered ~150 s
+apart, then polls until each metadata file lands, and exits with
+`RESULT=COMPLETE|PARTIAL|TIMEOUT`. Run it via Bash
+`run_in_background: true`; it skips libraries already on main, so
+re-running it after a partial result retries exactly the gaps.
+
+**Two specs in parallel is fine — but only in this mode** (~4 specs/h
+vs ~2); ten concurrent `impl-generate` runs showed no rate-limit
+effects. It works because `run_spec.sh` decides per library from
+`origin/main` metadata. `poll_spec.sh` and `monitor_spec.sh` must
+still run one spec at a time: their stall logic reads *any* active
+`impl-*` run as belonging to the spec they are watching (§3), so a
+second spec in flight makes them call a stalled spec healthy. Never
+mix the two modes on the same queue. Keep a ledger —
+`done.log` / `deferred.log` next to the queue file in `agentic/runs/`
+— and append the result line before dispatching the next spec, so a
+compaction or a crashed session can resume without recounting.
+
+**The one-retry rule.** A library that comes back missing gets
+exactly one fresh targeted dispatch before it is deferred. This is
+not optional politeness — on 2026-08-24 the single retry recovered
+highcharts/treemap-basic, ggplot2/wireframe-3d-basic and
+ggplot2/network-force-directed, all three of which had been read as
+capability gaps. Two failures on the same pair → `deferred.log` with
+the reason, move on, sweep at the end.
+
 ## Gotchas
 
+- **`Marking <lib> as failed: N generation attempts` counts more than
+  this run.** The number comes from `<!-- impl-fail:spec:lib -->`
+  marker comments on the issue, and nothing deletes them; before the
+  campaign-window fix (#10627) it spanned the issue's whole lifetime,
+  so a pair that failed twice in some old campaign was capped at ONE
+  attempt forever. Read the `Previous failures for <lib>/<spec>: N`
+  notice in the log before concluding a library "can't do" a plot
+  type — a high N there means the cap fired, not that generation was
+  tried three times today.
+- **`impl:<lib>:failed` is terminal and it lies about coverage.**
+  Nothing re-dispatches it: watchdog case 3 fires once, then only
+  logs `already retried by watchdog — needs manual attention`. Audit
+  it against metadata before trusting it — of 87 such labels on
+  2026-08-24, **42 sat on implementations that had since landed**.
+  Take the label as a hint to check, never as the coverage answer.
+- **"Agent reports success, writes no file"** is a live intermittent
+  failure (8 of 85 generate runs on 2026-08-24, ~9%): the Claude step
+  ends `"subtype":"success","is_error":false` and the next step fails
+  with `Implementation file not found in repository`. With retry
+  budget left the workflow self-heals (`bubble-basic/highcharts`:
+  failed 17:55, retried and succeeded 18:02, no human involved), so a
+  single occurrence is noise — only a repeat on the same pair means
+  anything.
+- **Static library + interactive/3D spec is the one gap that is
+  usually real.** plotnine, pygal, seaborn, matplotlib and ggplot2
+  against `*-interactive`, `*-drilldown`, `*-realtime`, `*-streaming`,
+  `slider-*`, `*-3d` specs make up most genuine dead ends; 18 of the
+  45 real gaps behind those labels were exactly this shape. Still
+  give them the one retry, then defer without further ceremony.
 - **The scripts' library list is a copy** of `core/constants.py`'s
   registry (15 libs). When a library is added/removed, update
-  `monitor_spec.sh`'s `ALL_LIBS`/`lang_of` in the same PR.
+  `monitor_spec.sh`'s `ALL_LIBS` and `run_spec.sh`'s `lang_of` in the
+  same PR.
 - **`gsutil` must be authenticated** for the GCS spot-check; a
   credentials failure reads as "incomplete" — check
   `gsutil ls gs://anyplot-images/ | head -1` before trusting a

.claude/skills/babysit-pipeline/run_spec.sh

@@ -0,0 +1,131 @@
+#!/usr/bin/env bash
+# Per-SPEC driver: dispatch impl-generate for every missing library of one
+# spec, staggered so the pipeline overlaps generate/review/merge across libs
+# (the bulk-generate pacing pattern), then poll origin/main until each lib's
+# metadata file exists. Repo metadata is the completion signal — works for
+# specs whose issues closed long ago. Exits per spec so the agent can report
+# and start the next queue entry.
+# Usage: run_spec.sh <spec-id> <model> <lib1> [lib2 ...]
+set -uo pipefail
+
+usage() {
+  echo "usage: $(basename "$0") <spec-id> <model> <lib1> [lib2 ...]" >&2
+  echo "  e.g. $(basename "$0") line-basic sonnet highcharts muix" >&2
+  exit 2
+}
+# Explicit guard: with `set -u` a missing argument would otherwise surface as an
+# unbound-variable error from somewhere deep in the polling loop.
+[ "$#" -ge 3 ] || usage
+SPEC="$1"; MODEL="$2"; shift 2; LIBS=("$@")
+
+# Resolve the repo from this script's location (.claude/skills/<name>/), so the
+# driver works from any checkout and any working directory. ANYPLOT_REPO wins
+# when the script is copied elsewhere (e.g. a scratch queue under agentic/runs/).
+# Fail fast rather than falling back to $PWD: an unvalidated repo makes every
+# `meta_present` check return false, which reads as "nothing ever landed" and
+# burns the full polling timeout before anyone notices.
+HERE="$(cd "$(dirname "$0")" && pwd)"
+REPO="${ANYPLOT_REPO:-$(git -C "$HERE" rev-parse --show-toplevel 2>/dev/null || true)}"
+if [ -z "$REPO" ] || [ ! -d "$REPO/plots" ]; then
+  echo "error: could not resolve the anyplot repo root (tried \$ANYPLOT_REPO, then git from $HERE)." >&2
+  echo "       set ANYPLOT_REPO=/path/to/anyplot and re-run." >&2
+  exit 2
+fi
+STAGGER=150      # seconds between dispatches
+INTERVAL=180     # poll interval
+LOGDIR="${POLL_LOG_DIR:-$HOME/.cache/anyplot-babysit}"
+mkdir -p "$LOGDIR"
+LOG="$LOGDIR/spec_${SPEC}.log"
+
+lang_of() {
+  case "$1" in
+    ggplot2) echo r ;;
+    makie) echo julia ;;
+    chartjs|d3|echarts|highcharts|muix) echo javascript ;;
+    *) echo python ;;
+  esac
+}
+
+meta_present() {  # assumes a fresh `git fetch` already ran this iteration
+  local lang; lang=$(lang_of "$1")
+  [ -n "$(git -C "$REPO" ls-tree --name-only origin/main -- "plots/${SPEC}/metadata/${lang}/$1.yaml" 2>/dev/null)" ]
+}
+
+pipeline_active() {
+  for wf in impl-generate.yml impl-review.yml impl-repair.yml impl-merge.yml impl-review-retry.yml; do
+    local out
+    if ! out=$(gh run list --workflow="$wf" --limit 12 --json status \
+        --jq '.[] | select(.status=="in_progress" or .status=="queued") | .status' 2>&1); then
+      printf 'WARN: gh run list %s failed; assuming active: %s\n' "$wf" "$out" >> "$LOG"
+      return 0
+    fi
+    grep -q . <<<"$out" && return 0
+  done
+  return 1
+}
+
+# Rough health signal for the report line: how many generate runs failed in the
+# last 25 min. The limit must cover the whole window — two specs in flight can
+# put 10+ runs in it, and a short limit silently reports "0 failures" because
+# the failures fell off the end of the list rather than because there were none.
+# `?` on error, never 0: a masked API failure reading as "all healthy" is how a
+# quota outage gets mistaken for a slow queue.
+recent_generate_failures() {
+  gh run list --workflow=impl-generate.yml --limit 60 \
+    --json conclusion,updatedAt \
+    --jq "[.[] | select(.conclusion==\"failure\" and (.updatedAt > (now - 1500 | todate)))] | length" 2>/dev/null \
+    || echo "?"
+}
+
+git -C "$REPO" fetch origin main --quiet
+TODO=()
+for lib in "${LIBS[@]}"; do
+  if meta_present "$lib"; then
+    echo "[skip] $SPEC/$lib already on main" | tee -a "$LOG"
+  else
+    TODO+=("$lib")
+  fi
+done
+if [ ${#TODO[@]} -eq 0 ]; then
+  echo "RESULT=COMPLETE spec=$SPEC (all libs already present)"
+  exit 0
+fi
+
+for i in "${!TODO[@]}"; do
+  lib="${TODO[$i]}"
+  echo "[$(TZ=UTC date -u +%H:%M:%S)] dispatch $SPEC/$lib model=$MODEL" | tee -a "$LOG"
+  if ! gh workflow run impl-generate.yml \
+      -f "specification_id=$SPEC" -f "library=$lib" -f "model=$MODEL" >> "$LOG" 2>&1; then
+    echo "WARN: dispatch failed for $SPEC/$lib" | tee -a "$LOG"
+  fi
+  [ "$i" -lt $(( ${#TODO[@]} - 1 )) ] && sleep "$STAGGER"
+done
+
+MAXMIN=$(( 30 + 15 * ${#TODO[@]} )); [ "$MAXMIN" -gt 150 ] && MAXMIN=150
+iters=$(( MAXMIN * 60 / INTERVAL ))
+idle=0; last_done=-1
+for ((i=1; i<=iters; i++)); do
+  sleep "$INTERVAL"
+  git -C "$REPO" fetch origin main --quiet
+  missing=(); done_n=0
+  for lib in "${TODO[@]}"; do
+    if meta_present "$lib"; then done_n=$((done_n+1)); else missing+=("$lib"); fi
+  done
+  if [ ${#missing[@]} -eq 0 ]; then
+    echo "RESULT=COMPLETE spec=$SPEC libs=${TODO[*]} after ~$(( (i*INTERVAL)/60 + (${#TODO[@]}-1)*STAGGER/60 )) min"
+    exit 0
+  fi
+  if [ "$done_n" -gt "$last_done" ]; then idle=0; last_done=$done_n
+  elif pipeline_active; then idle=0
+  else idle=$((idle+1)); fi
+  printf '[%s iter %d/%d] %d/%d done, missing: %s (idle=%d)\n' \
+    "$(TZ=UTC date -u +%H:%M:%S)" "$i" "$iters" "$done_n" "${#TODO[@]}" "${missing[*]}" "$idle" >> "$LOG"
+  if [ "$idle" -ge 3 ]; then
+    echo "RESULT=PARTIAL spec=$SPEC done=$done_n/${#TODO[@]} still-missing: ${missing[*]} — pipeline idle ~$((3*INTERVAL/60)) min"
+    echo "recent generate failures (25 min): $(recent_generate_failures)"
+    exit 0
+  fi
+done
+echo "RESULT=TIMEOUT spec=$SPEC done=$last_done/${#TODO[@]} still-missing: ${missing[*]} after ${MAXMIN} min"
+echo "recent generate failures (25 min): $(recent_generate_failures)"
+exit 0

CHANGELOG.md

@@ -158,6 +158,16 @@ aggregate instead: an italic *Catalog* line at the end of the version section an
 
 ### Changed
 
+- **`babysit-pipeline` learned gap backfill** — closing coverage gaps across the catalogue is
+  a different job from watching one fresh spec, and the skill only described the latter. It
+  now documents reading the missing set from `plots/*/metadata/` instead of from labels
+  (most affected specs have closed issues, so their labels are absent or stale), the
+  per-spec `run_spec.sh` driver, two-specs-in-parallel pacing, and the one-retry rule. Four
+  new gotchas record traps that cost real implementations during the 2026-08-24 backfill:
+  the failure-marker count spanning more than the current run, `impl:<lib>:failed` being
+  both terminal and unreliable (42 of 87 sat on implementations that had since landed), the
+  intermittent "reports success, writes no file" generation failure, and the one gap shape
+  that usually is real — a static library against an interactive or 3D spec (#10628).
 - **llms.txt now tells agents how to actually fetch things** — the file linked nine human-facing
   HTML pages and named no machine endpoint. New sections document the REST API (base URL, the
   retrieval endpoints, OpenAPI), the GCS render URL pattern (themes, responsive widths, WebP),