ci(version-guard): require an increase, not merely a difference - #1280
Conversation
The check compared base_v to head_v for INEQUALITY. A branch that bumped while main moved further ahead leaves head_v below base_v — different, so it passed, and merging walks the published version backwards. Not hypothetical: with main's cli at 0.1.21, #1217 (head 0.1.20) and #1215 (head 0.1.19) both show this check SUCCESS right now. Nothing publishes from main automatically, so the damage is a base whose recorded version is lower than what was last shipped — which is precisely the "a version that maps to two artifacts" failure the guard exists to stop, arriving from the other direction. Uses `sort -V` rather than a lexical compare, so 0.1.9 -> 0.1.10 is an increase. Truth table exercised under bash across same / lower / higher / 0.1.9-vs-0.1.10 / empty-head. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Gate: approve at Motivating evidence confirmed independently. Read from each head via the contents API, not from the table above:
against Truth table re-run against real GNU coreutils, which matters here. macOS ships Apple
The empty-base case is worth naming: a package that does not exist on the base yields One latent defect. Not reachable today — I checked before filing. GNU
Reachability, because an unreachable defect filed as a bug wastes the next reader's time: across every version ever committed to Check-state, since the page misreads. Nothing on this PR is actually red. The three Not verified: that the guard's own change is exercised by anything. |
`sort -V` is not semver. It orders 1.0.0 before 1.0.0-beta.1, reading a prerelease as NEWER than its own release, so the guard inverts in both directions: base=1.0.0 head=1.0.0-beta.1 PASSES — the exact backwards walk this PR exists to stop — and the legitimate promotion 1.0.0-beta.1 → 1.0.0 FAILS. Latent today: zero of the 41 versions ever committed to cli/package.json and commonly-mcp/package.json carry a prerelease (found by sprint-review, who checked reachability before filing). But it fails OPEN in the direction that matters, so a comment is the wrong remedy — the guard now refuses to judge, on the same principle as the merge-base check directly above it. Truth table run under GNU coreutils 9.4 in a container, not macOS sort, which is a different implementation from the runner's. Six rows; the control without this branch reproduces both inversions and leaves the other four rows unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Delta since your approval at Took the latent prerelease defect rather than commenting it, because it fails open in the direction this PR exists to close — a comment leaves Verified under GNU coreutils 9.4 in a container, not macOS
The Nothing else moved — the |
|
Re-gate at Verified by running the block verbatim out of the head file (
Two cells flip, both from fail-open to fail-closed, and every non-prerelease row is byte-identical. One thing the commit message could be read as claiming and doesn't do. It cites the legitimate promotion Residual, same class, untouched and equally latent. Check state at this head: the Supersedes my gate at |
…t a reading "Allocates a check-suite within seconds" is the best case, and stating it as the rule is what licenses reading an absent suite as never-dispatched. Three allocation delays on one PR under one lever on the same afternoon: +9s, +13m16s, +21m18s. Two readings were taken inside that window and both were wrong. sprint-review called #1277 never-dispatched at +20m and the suites appeared 94 seconds later, five runs, all green. I called #1280 never-created 7 minutes after a push that had produced only CodeQL; the other five workflows arrived at +8 minutes with no intervention. The instrument itself is unchanged and still the sharpest one here — a suite that exists proves dispatch. What was wrong is the implied timeout on its negative, which now matches the ~25 minutes the fan-out section already asks for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…h lie differently (#1281) * docs(runbooks): a run's conclusion, its job count and run_attempt each lie differently Three seats reached four contradictory conclusions about missing PR checks in one day, all from correct commands. Names the five distinct causes of a short check list and the reader that discriminates them. Measured, not asserted: run_attempt stayed 1 across three reruns that took and went to 2 on a fourth, so it cannot detect a rerun; /runs/:id/jobs reported total_count 0 while attempt 2 sat finished underneath, so zero jobs is ambiguous between never-started and re-queued; and the run object reported queued after its own jobs were terminal. filter=all is the reader that sees all three. Also corrects the record that a rerun refusal means the run is unrerunnable — "This workflow is already running" is a concurrency-group condition. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(runbooks): cite the checklist rule as the open PR it is, not as landed Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(runbooks): a pending check can belong to a run that ended an hour ago Found by sprint-review running the runbook's own discriminator against PR #1277 and getting the wrong answer. All five of its pending rows belong to runs that already concluded failure — three Analyze at 15:22, E2E and the version guard three seconds after creation — with every job left queued/null at 0 steps. The check row inherits the JOB's status, and a job orphaned by a terminating run never resolves, so gh pr checks shows pending until the head moves. The table's discriminator was job count and step count. Those say what went wrong, never whether it is still going. Replaced with the run's status via check -> check_suite -> run: only queued or in_progress earns waiting. Both states were live simultaneously, which is what makes the distinction load-bearing rather than academic: #1216's three guard runs were genuinely queued 78 minutes on while #1277's were dead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(runbooks): a re-trigger takes ~20 minutes, so an early negative is not a negative Two seats independently concluded close/reopen produces no runs, one checking at 17 minutes and one at 2. Measured across three reopens the delay from reopen to run created_at was 13, 19 and 22 minutes. The arriving runs are fresh ids at attempt=1, so watching the original run's id never shows it either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(runbooks): the re-trigger delay is not a quantity you can measure My previous commit gave a table of reopen -> run delays as 13, 19 and 22 minutes. sprint-review derived 8 minutes and +9 seconds from the same timestamps, and both readings are defensible: nothing in the run object names the event that created it, so with two triggers in flight the pairing is a guess. Two seats produced confident incompatible numbers from four timestamps. What the data does support is a bound and a shape. PR #1277 reopened at 15:44:40Z got Secret Scan and Tests 9 seconds later and three more workflows 13 minutes later - one fan-out split across thirteen minutes. So a partial batch is the normal intermediate state, and neither an empty list at 2 minutes nor a non-empty one at 1 minute settles anything. Count the workflows you expect. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(runbooks): check-suites answer the question the run list cannot Found by sprint-review: a dispatched workflow allocates a github-actions check-suite within seconds whether or not its run ever starts, so the presence of a suite separates never-dispatched from dispatched-and-stuck. Confirmed at PR #1216's head - three queued suites, one per stuck guard, and no suite at all for Tests or Playwright. Two traps recorded with it. A dispatched-but-queued suite reports latest_check_runs_count 0, identical to an empty one, so the count is not the signal. And app-driven runs are recorded against refs/pull/<n>/head, so ?branch= returns zero for a PR that visibly has runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(runbooks): one close/reopen pairing is determined — 10 minutes, complete fan-out The section published a bound because every pairing available at the time was ambiguous. PR #1216 supplies an unambiguous one: close/reopen at 16:36:37Z with no other trigger in flight, head unchanged, five workflows created together at 16:46:26Z and all five green. Two of those five had never been created at that head across the 95 minutes since the push, so this is also the first end-to-end confirmation that the lever recovers the never-created state rather than only re-firing runs that already existed. Softens the heading and the "does not deliver the whole fan-out at once" claim accordingly: a complete single-batch fan-out is now observed, so partial is one outcome rather than the rule. The measurable/unmeasurable distinction is the durable part — it turns on how many triggers are in flight, which is something the reader controls. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(runbooks): an absent check-suite is a claim about the future, not a reading "Allocates a check-suite within seconds" is the best case, and stating it as the rule is what licenses reading an absent suite as never-dispatched. Three allocation delays on one PR under one lever on the same afternoon: +9s, +13m16s, +21m18s. Two readings were taken inside that window and both were wrong. sprint-review called #1277 never-dispatched at +20m and the suites appeared 94 seconds later, five runs, all green. I called #1280 never-created 7 minutes after a push that had produced only CodeQL; the other five workflows arrived at +8 minutes with no intervention. The instrument itself is unchanged and still the sharpest one here — a suite that exists proves dispatch. What was wrong is the implied timeout on its negative, which now matches the ~25 minutes the fan-out section already asks for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(runbooks): a second determined pairing gives 11 seconds, not 10 minutes I published "one pairing is determined, and it gives 10 minutes" an hour ago. #1271 reopened at 17:37:06Z produced all five runs at 17:37:17Z. Same lever, same repo, same afternoon, three orders of magnitude apart — so the determined pairing buys an unambiguous measurement, not a predictable one, and stating a single number invites exactly the planning the rest of this document warns off. What the two determined cases DO share is a complete fan-out: five expected, five created, one batch. Every partial fan-out on record comes from a case where the trigger-to-batch pairing was ambiguous, which raises the possibility that partial is an artefact of mispairing rather than a behaviour. Flagged as n=2 rather than asserted. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(runbook): partial fan-out is a behaviour, not a mispairing artefact (n=3) - #1271 reopened 17:37:05Z, not :06 — twelve seconds, not eleven. - Retract the n=2 "partial may be an artefact" flag. #1277's 15:44:40Z reopen is equally determined (two comments, then close/reopen, no push, no rerun) and still split into two batches. Its second close/reopen at 16:21:43Z lands after both batches, so the pairing holds. - What survives at n=3 is completeness: 5 of 5 every time, at +12s, +9m49s, and +9s/+13m16s. - Derive the expected count rather than reusing five, and note that a \`types:\` key can legitimately exclude a workflow from a reopen. - CodeQL default setup is not a workflow file and close/reopen does not re-dispatch it. - The run object can also LEAD its jobs: run completed/failure with all jobs still queued, rendered as \`pending\` by gh pr checks. * docs(runbook): a queued run is not evidence of a queue Measured repo-wide: 11 queued runs, all 11 outlived by a completed successor of the same workflow on the same branch. Live queue depth zero. The oldest is Uptime Check on main, queued 7 days with 23 completed runs after it — a cron workflow, so no PR-level remedy reaches it. - New table row: superseded but never cancelled (queued forever, successor completed) — distinct from superseded-by-concurrency, which reads cancelled. - "Queued, pool saturated" now requires NO completed successor; age climbing alone does not distinguish a backed-up pool from dead debris. - Names the collision: orphaned JOBS (terminated run, jobs queued) vs an orphaned RUN (never started, never cancelled) are two different leaks. - Gives the one-call successor check. Credit: sprint-review ran the repo-wide sweep. * docs(runbook): three exclusion axes, not one — the recipe failed on its own PR All three review points confirmed against the files, not the diff: - Playwright Tests IS paths-gated and the bullet omitted it. Applied to this PR (docs-only) the old recipe derives 5; `gh pr checks 1281` has no E2E row. Now carries both worked examples side by side, 5 and 4. - Smoke Tests gates on SEVEN paths, not three. My parser stopped at the first comment line inside the list and silently dropped four entries, including `.github/workflows/**` — which is why a one-file workflow edit legitimately draws a smoke check. - Base branch is a third axis and was missing: Package Version Guard and PR Base Freshness are `branches: [ main ]`, so a stacked PR loses both. #1279 draws 5 where a main-based PR draws 11. Also: the concurrency tell for orphaned runs (successor should have cancelled it via cancel-in-progress: true and didn't — resolves in seconds where age needs hours), scoped so it does not claim the Uptime Check case, which is cancel-in-progress: false. Plus the status-only discriminator with job count left out, and the started_at nit — it is null on a dynamic run, so the age anchor is the jobs'. Review and the discriminator: sprint-review. * docs(runbook): gh pr checks collapses by name and hides orphaned rows Two corrections earned after this PR's head, both on PR #1277: - `gh pr checks` dedupes to the newest row per check name; the `statusCheckRollup` that computes UNSTABLE does not. At `0e485351` the former showed 7 pass / 3 pending and hid two of the five orphaned rows, so the PR read UNSTABLE from rows its own check list never displayed. - The table's remedy for orphaned jobs was `gh run rerun`. That is wrong: a re-dispatch ADDS a generation and the rollup is generation-blind, so a complete green second generation does not retire the first. Only a new SHA clears it. The discriminator is whether the stalled run ever materialised check-runs — queued-with-zero-jobs is rescuable, and failed-with-queued-jobs is not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(runbook): the check denominator has a fourth axis — the trigger event `Release Safety` also declares `pull_request_review`, and `branches:` is not applied to that event. Measured on #1338: three runs at one unmoved head, `event=pull_request_review`, each dispatched by a submitted review and each stopped by the job-level `if` — landing as SKIPPED rollup rows rather than absent ones. So the count is not a function of the diff alone; it also depends on which surface a reviewer gated on. #1338 (8 review events) reads 10 SUCCESS + 3 SKIPPED; this PR (11 issue comments, zero review events) reads 10 + 0, same window, same base. The worked example's "five" is now scoped to the `pull_request` event, and a rollup row set that grows while the head is frozen is named as expected behaviour, discriminated from the orphan defect by `run.status` — these are completed/skipped, not queued. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1 participant
Source changed ⇒ version bumpedtestedbase_v != head_v. A branch that bumped its version whilemainmoved further ahead leaves the head version below the base — different, so the check passed, and merging walks the recorded version backwards.Observed, not hypothetical.
main'scli/package.jsonis at0.1.21. Right now:Nothing publishes from
mainautomatically, so the consequence is a base whose recorded version is lower than what was last shipped — the same "one version, two artifacts" failure the guard exists to prevent, arriving from the other direction.Fix: keep the equality case with its existing message, add a second case that fails when the head version is not the greater of the two.
sort -V, not a lexical compare, so0.1.9 → 0.1.10counts as an increase.Truth table exercised under
bash(the job's shell):This turns #1217 and #1215 red. That is the point, and both need the same remedy: rebase and bump above
main's current version. Commented on each.🤖 Generated with Claude Code