docs(babysit-pipeline): add gap-backfill mode and the traps it exposed

[MarkusNeusinger]

Why

The babysit-pipeline skill covers watching one fresh spec to 15/15. Backfilling coverage gaps across the catalogue — dozens of older specs each missing the libraries added after they were generated — is a different job, and the skill said nothing about it. Running it by hand on 2026-08-24 surfaced four traps that cost, or nearly cost, real implementations.

What's new

Section 5 · Gap backfill

• Compute the missing set from plots/{spec}/metadata/{lang}/{lib}.yaml on origin/main, not from impl:{lib}:done labels — most affected specs have closed issues, so their labels are absent or stale. Includes the git ls-tree | awk one-liner.
• run_spec.sh <spec> <model> <lib>... as the driver: staggered dispatch, poll to metadata, RESULT=COMPLETE|PARTIAL|TIMEOUT, skips libraries already on main so a re-run retries exactly the gaps.
• Two specs in parallel (~4 specs/h vs ~2; ten concurrent impl-generate runs showed no rate-limit effects).
• A done.log / deferred.log ledger written before dispatching the next spec, so a compaction or crashed session resumes without recounting.
• The one-retry rule, with the evidence for why it is not optional: the single retry recovered highcharts/treemap-basic, ggplot2/wireframe-3d-basic and ggplot2/network-force-directed, all three of which had already been read as capability gaps.

Four new gotchas

Trap	Why it matters
Marking <lib> as failed: N generation attempts counts more than this run	The markers span past campaigns (#10627 scopes this to 12 h). Check the Previous failures for <lib>/<spec>: N notice before concluding a library can't do a plot type.
impl:<lib>:failed is terminal and unreliable	Nothing re-dispatches it — watchdog case 3 fires once, then only logs needs manual attention. And of 87 such labels, 42 sat on implementations that had since landed.
"Agent reports success, writes no file"	8 of 85 generate runs (~9%). Self-heals when retry budget remains (bubble-basic/highcharts: failed 17:55, succeeded on auto-retry 18:02). One occurrence is noise.
Static library + interactive/3D spec	The one gap shape that usually is genuine — 18 of the 45 real gaps. Still give the one retry, then defer.

run_spec.sh joins the bundled scripts. Its hardcoded REPO=/home/tirao/anyplot is replaced by resolution from the script's own location (ANYPLOT_REPO overrides), matching how poll_spec.sh derives HERE.

Verification

• bash -n on run_spec.sh; repo auto-resolution checked from the skill directory (git -C "$HERE" rev-parse --show-toplevel → the repo root).
• The skill's existing ALL_LIBS gotcha now names run_spec.sh's lang_of too, so a library addition updates both scripts.
• Content is drawn from a real backfill run, not invented: every number above has a corresponding workflow run or label query behind it.

Related

• fix(impl-generate): scope the retry cap to the current campaign #10627 fixes the retry-cap bug the first gotcha describes.
• fix(watchdog): skip cron-liveness rescue when daily-regen is disabled #10540 fixes the watchdog crash found in the same session.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RbZuWNDFy7kjXh9kLfA4dP

[Copilot]

🟡 Changes recommended

run_spec.sh currently isn’t reliably CWD-independent as documented and has a fragile timestamp comparison in its failure counter, which can lead to incorrect idle/health reporting during backfill runs.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR extends the .claude/skills/babysit-pipeline operational documentation to cover “gap backfill” (many specs with partial coverage) and adds a new per-spec driver script (run_spec.sh) to dispatch missing libraries and poll origin/main metadata until coverage lands.

Changes:

• Document a new “Gap backfill” mode, including how to compute missing (spec, lib) pairs from plots/*/metadata/** on origin/main and how to pace parallel dispatch.
• Add .claude/skills/babysit-pipeline/run_spec.sh to dispatch impl-generate per missing library and poll for metadata completion.
• Add a changelog entry under [Unreleased] describing the new skill capabilities and gotchas.

File summaries

File	Description
CHANGELOG.md	Adds an [Unreleased] entry describing the new babysit-pipeline gap-backfill workflow and gotchas.
.claude/skills/babysit-pipeline/SKILL.md	Documents the new “Gap backfill” workflow and updates gotchas to reference the new driver behavior and maintenance requirements.
.claude/skills/babysit-pipeline/run_spec.sh	Introduces a per-spec backfill driver that dispatches missing libraries and polls origin/main metadata as the durable completion signal.

Review details

Suppressed comments (3)

.claude/skills/babysit-pipeline/run_spec.sh:40

• pipeline_active() calls gh run list from the caller’s current working directory, even though the script claims it works from any working directory. If the user runs this script outside the repo (or without GH_REPO configured), gh will fail and the function will always assume "active", preventing idle detection from working as intended. Run gh from $REPO so the repository context is correct.

pipeline_active() {
  for wf in impl-generate.yml impl-review.yml impl-repair.yml impl-merge.yml impl-review-retry.yml; do
    local out
    if ! out=$(gh run list --workflow="$wf" --limit 12 --json status \
        --jq '.[] | select(.status=="in_progress" or .status=="queued") | .status' 2>&1); then

.claude/skills/babysit-pipeline/run_spec.sh:53

• recent_generate_failures() compares ISO timestamps using a string comparison against todate, which can be format-sensitive and produce incorrect counts. It also runs gh from the caller’s CWD, which breaks if the script is invoked outside the repo. Compare timestamps numerically via fromdateiso8601 and run gh from $REPO.

recent_generate_failures() {
  gh run list --workflow=impl-generate.yml --limit 15 \
    --json conclusion,updatedAt \
    --jq "[.[] | select(.conclusion==\"failure\" and (.updatedAt > (now - 1500 | todate)))] | length" 2>/dev/null || echo 0
}

.claude/skills/babysit-pipeline/run_spec.sh:74

• The dispatch path (gh workflow run ...) is also executed from the caller’s current working directory. If the script is launched from outside the repo, dispatch will fail even though $REPO was resolved successfully. Run the gh invocation from $REPO to make the driver actually CWD-independent as documented.

  if ! gh workflow run impl-generate.yml \
      -f "specification_id=$SPEC" -f "library=$lib" -f "model=$MODEL" >> "$LOG" 2>&1; then
    echo "WARN: dispatch failed for $SPEC/$lib" | tee -a "$LOG"

• Files reviewed: 3/3 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[MarkusNeusinger]

Both applied. run_spec.sh now has an explicit arity guard ([ "$#" -ge 3 ] || usage) rather than letting set -u surface a missing argument from inside the polling loop, and repo resolution fails fast instead of falling back to $PWD — an unvalidated root makes every meta_present check return false, which reads as "nothing ever landed" and burns the full timeout before anyone notices. The error names ANYPLOT_REPO as the override. Verified: no-args exits 2 with usage, ANYPLOT_REPO=/nonexistent exits 2 with the resolution error.

Related, same failure direction: the generate-failure counter used --limit 15, which no longer covers its own 25-minute window once two specs are in flight, and masked API errors as 0. Both silently report a healthy pipeline; raised to 60 and it now reports ? on error.

[MarkusNeusinger]

You are right, and the contradiction was load-bearing rather than cosmetic. Parallelism is safe only for run_spec.sh, which decides per library from origin/main metadata; poll_spec.sh and monitor_spec.sh infer progress from active impl-* runs, so a second spec in flight makes them call a stalled spec healthy — exactly backwards.

Both places now say so: §0 marks backfill as the single exception with the reason, and §5 states that the two modes must not be mixed on one queue.

[Copilot]

🟡 Changes recommended

run_spec.sh has correctness issues around gh repo targeting and unchecked git fetch failures that can lead to dispatching/polling against the wrong repo or stale origin/main state.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

.claude/skills/babysit-pipeline/run_spec.sh:109

• Inside the polling loop, git -C "$REPO" fetch origin main --quiet is also unchecked. A transient fetch failure would silently keep checking stale origin/main metadata and can lead to a misleading RESULT=TIMEOUT/RESULT=PARTIAL even if metadata landed.

Handle fetch errors explicitly (same as the initial fetch) so polling results reflect reality.

  git -C "$REPO" fetch origin main --quiet

• Files reviewed: 3/3 changed files
• Comments generated: 2
• Review effort level: Lite