Skip to content

feat(pm): archive the open board before the closed history, and move the snapshot cron off the patrol's hour - #17442

Queued
os-litant wants to merge 1 commit into
mainfrom
claude/issue-17434-board-snapshot-open-first
Queued

feat(pm): archive the open board before the closed history, and move the snapshot cron off the patrol's hour#17442
os-litant wants to merge 1 commit into
mainfrom
claude/issue-17434-board-snapshot-open-first

Conversation

@os-litant

Copy link
Copy Markdown
Collaborator

Fixes #17434

The archiver's first scheduled run measured the defect this fixes: walking state=all oldest-first, 800 requests bought 137 closed issues and 268 closed pull requests from February and zero open cards. The records an account suspension destroys are the open board, so that half was being archived last — days away, at ~405 numbers per run.

1. Open first — the walk order

A first walk now runs two phases, in one order, never interleaved:

phase listing ends when
open state=open&sort=updated&direction=asc — the /issues listing carries pull requests too, so one walk is the whole open board a short page
history state=all from the stored history cursor — the closed record, which has no other reader anywhere a short page
incremental one since walk, unchanged by any of the above a short page

The two cursors are kept apart on purpose: a run that runs out of budget inside the open set resumes inside it, and the history cursor waits untouched. The page-walk rule is unchanged — a short page ends a walk, and the walk re-anchors on the last row's value rather than incrementing an offset.

Before / after, measured on this board with --limit=20 into a scratch directory (the same flags, two builds):

origin/main   read 20 number(s) in 49 request(s)
              archive 20 record(s): 0 open / 1 closed issue(s), 0 open / 19 closed pull request(s)
              resume cursor 2026-01-20T03:06:53Z          # January, all of it closed

this branch   phase        open — the open board first: the records a suspension destroys
              open set     INCOMPLETE — the open board is still being walked, resuming at 2026-08-18T04:39:33Z
              read 20 number(s) in 19 request(s)
              archive 20 record(s): 20 open / 0 closed issue(s), 0 open / 0 closed pull request(s)

The manifest says which phase, so a reader can tell the two situations apart (from the same drill):

"walk_phase": "open",
"walk": {
  "open_set": { "complete": false, "completed_at": null, "cursor": "2026-08-18T04:39:33Z" },
  "history":  { "complete": false, "cursor": null }
},
"resume": { "since": "2026-08-18T04:39:33Z", "phase": "open", "stopped_by": "limit", "reason": "--limit=20 reached" },
"run": { "numbers_read": 20, "files_written": 38, "walk_complete": false, "open_set_completed_here": false }

walk_phase: history with open_set.complete: true is "open set complete, history resuming"; walk_phase: open is "still inside the open set". A resumed run continues the phase it stopped in — drilled live: the second run planned resuming the open set at 2026-08-18T04:39:33Z — the closed history does not start until it is complete, read 60 more numbers, and stayed in the open phase.

Migration, which the next scheduled run will take: the archive on board-archive was written before this order existed, so it carries no walk block. Its cursor is a history cursor by construction (the only walk that version had was state=all), so it is kept and the open board is walked first from the beginning. An archive whose old walk had completed is already past both phases and stays incremental.

2. The request arithmetic for the open set

Measured, not estimated — cumulative over the two drill runs above: 79 unique open issues archived in 74 requests = 0.94 requests per number (one listing page per re-anchor, plus one comments read for a number that has comments).

The board's own arithmetic, read by the drill at 14:20Z: open_issues_count 596 minus 14 open pull requests = 582 open issues.

582 open issues   x 0.94  ~ 547
 14 open PRs      x 3     ~  42   (reviews + review comments + the comment thread)
  7 listing pages          =   7
  the board's own arithmetic =  2   (repo meta + one open-PR page)
                            -------
                            ~ 598 requests

So the open set fits one 800-request run, with room to start the history walk in the same run. It is not assumed to: the resume-inside-the-open-phase path is drilled above and self-tested, because a busier board or a slower page would land it in a second run.

3. The cron, and the budget it is capped against

- cron: '37 1,7,13,19 * * *'      before — the half-state patrol's cron to the minute
- cron: '7 2,8,14,20 * * *'       after

Both are scheduled board readers authenticating as this repository's GITHUB_TOKEN, and that budget is 1,000 requests per hour per repository — one pool, not a per-workflow allowance — so a snapshot run spending its full 800 leaves the patrol the remainder of the same window. The collision reading: the 13:37Z patrol run (34484272265) and the 13:41Z snapshot run (34484192186) were both green on the day this was filed, so nothing had been starved yet. The offset is prevention, taken while the archive was still small enough that no run had spent its whole cap against the patrol's window — and the header now says exactly that.

Deviation from the card's suggested 7 4,10,16,22, on a measurement. Of the five hour sets that share no hour with the patrol's 1,7,13,19, 2,8,14,20 is the least contended on this repository's cron inventory (22 scheduled workflows read out of .github/workflows/):

set contention
2,8,14,20 (chosen) 08, 14, 20 carry no other scheduled workflow at all; 02 carries the platform-checklist watchdog at :51 (44 min after this run starts) and CodeQL on Mondays
4,10,16,22 (suggested) drops an 800-request run into the busiest scheduled hour on this board — 04:00 rerun-safety, 04:19 release-coverage patrol, 04:23 required-set patrol, 04:37 branch reaper (Mondays), 04:41 create-smoke — and shares hour 16 with the required-set patrol
0,6,12,18 release.yml runs 0 */6 — every one of those four hours
3,9,15,21 / 5,11,17,23 one build-heavy hour each (03: validate-deps, scaffold-e2e; 05: coverage-nightly, nightly tiers)

Four runs a day and workflow_dispatch are unchanged. The workflow header now names the shared pool, every standing spender on it, and why the per-run cap is 800: it is the largest cap that still leaves a fifth of the hour to whatever else lands in the same window. No retry loop, no cap raise, no second token.

4. The count check's pending predicate — Zone 2C, half falsified

The card's assumption was that the check turns from pending to a verdict once the open set is complete, because its arithmetic needs only the open set. That holds for the run that enumerates the open set, and not for the runs after it. The history walk takes days; every card closed or opened during it is still archived in its old state, so a verdict in a history-phase run would report the clock as a board defect (a permanently red alarm is silence — this tool's own header says so about gone.json).

So the predicate moved from "this run stopped" to "this run did not enumerate the open board", which is strictly more precise, and every pending now carries the reason:

  • the run that finishes the open phase reads the board's own arithmetic right there — the tightest instant the two sides of the comparison can name — and gets a verdict even though the history is still resuming;
  • a later history-phase run reports pending with the open set is complete but was enumerated in an earlier run, and the closed history is still walking — cards closed or opened since then are archived in their old state;
  • a run still inside the open set, or one that never read the board's count, says which.

The check's own contract is kept: pending is a third answer and never a pass (ok is null).

5. Bounded in-place fix, in the same file: the nested run stamp

The card's negative control — "an identical re-run still writes nothing" — did not hold in the steady state on main, and writing item 4's self-test case for it is what surfaced it. materialManifest strips the three top-level fields that move every run, but board.read_at is nested and escaped it, so a run whose whole job is to write nothing would move that stamp and commit a manifest-only diff on every scheduled run — burying the real diffs exactly the way the file's header promises this tool never does. It is now stripped with the others.

Reverse verification (fix committed first, mutation proved on disk, restore proved against the HEAD blob):

before mutation   marker count 2   blob 8a20b0597a4e09265e0cb29fdd7d2d3bfb655829 == HEAD
after mutation    marker count 0   blob 2470002619d8fe67e2273c1844273e57f668af51
  MUTATED   run1 -> written 0 file(s) + manifest      run2 -> written 0 file(s) + manifest
after restore     marker count 2   blob 8a20b0597a4e09265e0cb29fdd7d2d3bfb655829 == HEAD; git diff HEAD empty
  RESTORED  run1 -> written 0 file(s) + manifest      run2 -> written 0 file(s) + idempotent

Direction: the negative control turns red under the ablation and green on restore. Both legs ran from the committed state, against the same live board, minutes apart.

Verification

  • pnpm check:pm-board-snapshot89 cases across 8 batteries (70 before). New: the open-first order, the two phase cursors, the resume-inside-the-open-phase case, the migration of a pre-phase archive, the state=open / state=all spellings, the pending predicate's five answers, and the nested run stamp. The 70 existing cases are all still present; the 7 that named selectSince now name selectWalkPlan (a second cursor decision function would have been a fork) with their assertions and case names unchanged.
  • Gate families derived with node scripts/pm/dispatch-gates.mjs --commands on the real diff and reconciled with --ran; exit codes captured before any pipe.
  • npx eslint --no-inline-config scripts/pm/board-snapshot.mjs exit 0, no output.
  • scripts/pm/check-governed-merges.mjs --test on the final file list: NOT governed.
  • Live drills on the user token: 4 narrow-window runs plus the two ablation pairs, ~200 requests total. No full-board pull.

Acceptance notes

  • ARCHIVE_SCHEMA stays 1: the file's own rule is to bump it when a record's shape changes, and no record shape moved — the manifest gained fields, and the manifest's only reader is this tool.
  • The count check's pending verdict gained a reason key, and every non-pending verdict carries reason: null so the key set is identical in both branches and the manifest diff stays stable.
  • The PR-time dry run is untouched — --dry-run --limit=5 --max-requests=25 into the runner's temp dir, no archive checkout, no commit. It now reads 5 open numbers instead of 5 closed ones, at the same cap.
  • noted, not filed: the drill archive's count check exits 2 with a SHORTFALL, which is correct for a 79-record scratch directory but means a --limit drill can never end green; a drill flag and a scheduled run share one exit register. Successor: whoever next adds a flag to this tool.
  • noted, not filed: a card closed on the board while the open phase is walking stays archived as open until the history walk reaches it, so the first open-phase verdict can carry a surplus of the walk-window churn. Inherent to comparing a windowed enumeration with a point-in-time count; the board's arithmetic is now read at the tightest available instant (immediately after the open phase) to keep the window as small as it can be. Successor: whoever acts on the first non-pending verdict.

Generated by Claude Code

…the snapshot cron off the patrol's hour

The first scheduled run of the archiver walked `state=all` oldest-first: 800
requests bought 137 closed issues and 268 closed pull requests from February
and zero open cards. The records an account suspension destroys are the OPEN
board, so that half was being archived last.

A first walk now runs two phases in one order. The open phase asks the
`/issues` listing for `state=open` (it carries pull requests too) and walks it
to a short page; only when it is complete does the closed-history walk start
from its own cursor. The two cursors are kept apart, so a run that runs out of
budget inside the open set resumes inside it and the history cursor waits
untouched. `walk_phase` and `walk.open_set` in the manifest separate "open set
complete, history resuming" from "still inside the open set". An archive
written before this order existed re-walks the open board first and keeps its
history cursor rather than discarding it.

The count check's pending predicate moves with it: its arithmetic needs only
the open set, so the run that finishes the open phase has a reading even while
the history is still resuming, and a later history run does not — that
enumeration is from an earlier run. Every pending verdict now carries the
reason it is pending.

Also fixed, in the same file and required by the negative control: the nested
`board.read_at` stamp escaped `materialManifest`, so a steady-state run whose
whole job is to write nothing would move it and commit a manifest-only diff on
every scheduled run.

The cron moves from `37 1,7,13,19` — the half-state patrol's cron to the
minute — to `7 2,8,14,20`. Both are scheduled board readers spending one
per-repository `GITHUB_TOKEN` hour of 1,000 requests. The header now names that
pool, the other standing spenders, and why the cap is 800.

Claude-Session: https://claude.ai/code/session_01YKEjmbYNvYWJvWGSWx26zK
Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/cd size/l skip-changeset PR has no user-facing published change; bypasses the changeset gate

Projects

None yet

2 participants