From c3c9f168099149fb2710ec192c3bf9d08c16ff2e Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Mon, 24 Aug 2026 22:38:05 +0200 Subject: [PATCH 1/3] fix(impl-generate): scope the retry cap to the current campaign MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 3-attempt budget is counted from hidden `` marker comments on the spec issue. Those markers are the permanent audit trail — no workflow deletes them — so the count covered the issue's entire lifetime rather than the current generation run. Any (spec, library) pair that had ever failed twice was capped forever: each later dispatch got one attempt, and a single hit of the intermittent "Claude reports success but writes no implementation file" failure (8 of 85 generate runs on 2026-08-24) marked it `impl::failed` — a terminal state, since the watchdog only reports those as needing manual attention. 87 (spec, library) pairs across 54 issues sit in exactly that state. Observed counts on the pairs that failed this way: 4 for chartjs/treemap-basic, 4 for plotnine/wireframe-3d-basic, 3 for ggplot2/network-force-directed — the last of which then succeeded on the very next manual dispatch. Count only markers newer than CAMPAIGN_WINDOW_H (12 h). A campaign runs well under an hour, so the window is generous while stale history ages out by itself, which also unblocks the existing 87 without touching the issue comments. The fail-closed path for an unreadable counter (#1010: ~1,200 runs in 38 h) is unchanged. Also state the real number in the cap message: "3 generation attempts" was printed after a single attempt and reads as a proven capability gap. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01RbZuWNDFy7kjXh9kLfA4dP --- .github/workflows/impl-generate.yml | 29 ++++++++++++++++++++++++++--- CHANGELOG.md | 16 ++++++++++++++++ 2 files changed, 42 insertions(+), 3 deletions(-) diff --git a/.github/workflows/impl-generate.yml b/.github/workflows/impl-generate.yml index 6da4dd4107..eedc8ff387 100644 --- a/.github/workflows/impl-generate.yml +++ b/.github/workflows/impl-generate.yml @@ -1016,6 +1016,11 @@ jobs: # Pass via env (not template-interpolated into the script) so hints # containing quotes / $ / backticks can't break shell parsing. CHANGE_REQUEST: ${{ inputs.change_request }} + # How far back a failure marker still counts toward the 3-attempt cap. + # Covers a whole campaign (a generate→review→repair→merge cycle runs + # well under an hour) with room for a stalled tail, without letting + # last month's failures veto today's retries. + CAMPAIGN_WINDOW_H: '12' run: | echo "::notice::Handling generation failure for $LIBRARY/$SPEC_ID" @@ -1035,11 +1040,25 @@ jobs: # (the old code broke on issues with >100 comments for this reason). # pipefail (Actions bash default) makes a gh failure fail the whole # pipeline, so the fail-closed else-branch still triggers. + # + # CAMPAIGN-SCOPED: only markers younger than CAMPAIGN_WINDOW_H count. + # The markers are the permanent audit trail — nothing deletes them — + # so counting all of them made the "3 attempts" cap per-issue-lifetime + # instead of per-generation-run. Any (spec, library) pair that ever + # failed twice then got exactly ONE attempt on every future dispatch, + # and a single hit of the ~9%/run "agent reports success but writes no + # file" flake parked it under `impl::failed`, which nothing + # retries (the watchdog reports it as "needs manual attention"). + # Measured 2026-08-24: counts of 3 and 4 on pairs that then succeeded + # on the very next manual dispatch, and 87 pairs parked repo-wide. + # A generation campaign is minutes long, so a window of hours is + # generous while still letting stale history age out on its own. MARKER="" + CAMPAIGN_CUTOFF=$(date -u -d "${CAMPAIGN_WINDOW_H} hours ago" +%Y-%m-%dT%H:%M:%SZ) if FAILURE_COUNT=$(gh api --paginate "repos/${{ github.repository }}/issues/${ISSUE}/comments?per_page=100" \ - --jq "[.[] | select(.body != null and (.body | contains(\"$MARKER\")))] | length" \ + --jq "[.[] | select(.body != null and (.body | contains(\"$MARKER\")) and .created_at > \"$CAMPAIGN_CUTOFF\")] | length" \ | awk '{ sum += $1 } END { print sum + 0 }'); then - echo "::notice::Previous failures for ${LIBRARY}/${SPEC_ID}: $FAILURE_COUNT" + echo "::notice::Previous failures for ${LIBRARY}/${SPEC_ID} since ${CAMPAIGN_CUTOFF}: $FAILURE_COUNT" else echo "::warning::Failure-count API call failed — failing closed (treating retry cap as reached, no auto-retry)" FAILURE_COUNT=999 @@ -1056,7 +1075,11 @@ jobs: if [ "$FAILURE_COUNT" -eq 999 ]; then CAP_REASON="the failure counter could not be read (failed closed — auto-retry disabled)" else - CAP_REASON="3 generation attempts" + # State the counted failures, not a flat "3 attempts": this run + # made ONE attempt, and reading the old wording as "tried three + # times, must be a real capability gap" is exactly how genuinely + # recoverable pairs got written off (2026-08-24). + CAP_REASON="${FAILURE_COUNT} failed attempt(s) in the last ${CAMPAIGN_WINDOW_H}h (cap: 3 per campaign)" fi echo "::warning::Marking $LIBRARY as failed: $CAP_REASON" diff --git a/CHANGELOG.md b/CHANGELOG.md index 81b7287026..6559a1fc91 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -32,6 +32,22 @@ aggregate instead: an italic *Catalog* line at the end of the version section an ### Fixed +- **The generation retry cap counted a library's entire failure history, not the current + run** — `impl-generate.yml` derives its "3 attempts" budget from hidden marker comments + on the spec issue, and nothing ever deletes those markers. A `(spec, library)` pair that + failed twice at any point in the past was therefore capped forever: every later dispatch + got a single attempt, and one hit of the intermittent "agent reports success but writes + no implementation file" failure (8 of 85 generate runs on 2026-08-24) parked it under + `impl::failed`, which nothing retries — the watchdog only reports it as needing + manual attention. Repo-wide that state had accumulated on **87 `(spec, library)` pairs**. + Marker counting is now scoped to a 12-hour campaign window, so stale history ages out and + a fresh dispatch gets its full three attempts. The fail-closed behaviour on an unreadable + counter (issue #1010: ~1,200 runs in 38 h) is untouched (#10554). +- **A failed library no longer claims three attempts it never made** — the cap message read + `Marking as failed: 3 generation attempts` even when the run made exactly one, which + reads as "tried repeatedly, must be a genuine capability gap". It now names the counted + failures and the window. Three pairs written off under the old wording on 2026-08-24 + succeeded on the very next dispatch (#10554). - **Image structured data carries the licensing fields Google recommends** — Search Console flagged every implementation page's `ImageObject` for missing `creator`, `copyrightNotice`, `creditText`, and `acquireLicensePage`. The bot-page JSON-LD now From 28ccfe1b2b4893215345f6f1eda84bcbb45af27e Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Mon, 24 Aug 2026 22:38:41 +0200 Subject: [PATCH 2/3] docs(changelog): reference #10627 in the retry-cap entries Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01RbZuWNDFy7kjXh9kLfA4dP --- CHANGELOG.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6559a1fc91..f6d315e1cd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,12 +42,12 @@ aggregate instead: an italic *Catalog* line at the end of the version section an manual attention. Repo-wide that state had accumulated on **87 `(spec, library)` pairs**. Marker counting is now scoped to a 12-hour campaign window, so stale history ages out and a fresh dispatch gets its full three attempts. The fail-closed behaviour on an unreadable - counter (issue #1010: ~1,200 runs in 38 h) is untouched (#10554). + counter (issue #1010: ~1,200 runs in 38 h) is untouched (#10627). - **A failed library no longer claims three attempts it never made** — the cap message read `Marking as failed: 3 generation attempts` even when the run made exactly one, which reads as "tried repeatedly, must be a genuine capability gap". It now names the counted failures and the window. Three pairs written off under the old wording on 2026-08-24 - succeeded on the very next dispatch (#10554). + succeeded on the very next dispatch (#10627). - **Image structured data carries the licensing fields Google recommends** — Search Console flagged every implementation page's `ImageObject` for missing `creator`, `copyrightNotice`, `creditText`, and `acquireLicensePage`. The bot-page JSON-LD now From 06a595d155912fb5964e32a3ba6ca5cdc4124658 Mon Sep 17 00:00:00 2001 From: Markus Neusinger <2921697+MarkusNeusinger@users.noreply.github.com> Date: Mon, 24 Aug 2026 22:45:55 +0200 Subject: [PATCH 3/3] fix(impl-generate): count the failure being handled in the cap message FAILURE_COUNT excludes the run whose failure triggered this step, so at the cap the message said "2 failed attempt(s)" while marking the library failed after the third. ATTEMPT already carries the corrected total. Reported by the Copilot reviewer on #10627. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01RbZuWNDFy7kjXh9kLfA4dP --- .github/workflows/impl-generate.yml | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/.github/workflows/impl-generate.yml b/.github/workflows/impl-generate.yml index eedc8ff387..180cd56ce2 100644 --- a/.github/workflows/impl-generate.yml +++ b/.github/workflows/impl-generate.yml @@ -1075,11 +1075,13 @@ jobs: if [ "$FAILURE_COUNT" -eq 999 ]; then CAP_REASON="the failure counter could not be read (failed closed — auto-retry disabled)" else - # State the counted failures, not a flat "3 attempts": this run - # made ONE attempt, and reading the old wording as "tried three - # times, must be a real capability gap" is exactly how genuinely + # State the real total, not a flat "3 attempts": a pair capped by + # stale history had ONE attempt today, and reading the old wording + # as "tried three times, must be a capability gap" is exactly how # recoverable pairs got written off (2026-08-24). - CAP_REASON="${FAILURE_COUNT} failed attempt(s) in the last ${CAMPAIGN_WINDOW_H}h (cap: 3 per campaign)" + # ATTEMPT, not FAILURE_COUNT: the latter excludes the failure being + # handled right now, so at the cap it would under-report by one. + CAP_REASON="${ATTEMPT} failed attempt(s) in the last ${CAMPAIGN_WINDOW_H}h (cap: 3 per campaign)" fi echo "::warning::Marking $LIBRARY as failed: $CAP_REASON"