From 4d3afdf90c1d8f611756930f994675d6998fae74 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Wed, 26 Aug 2026 09:22:14 -0700 Subject: [PATCH 01/14] docs(runbooks): a run's conclusion, its job count and run_attempt each lie differently MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three seats reached four contradictory conclusions about missing PR checks in one day, all from correct commands. Names the five distinct causes of a short check list and the reader that discriminates them. Measured, not asserted: run_attempt stayed 1 across three reruns that took and went to 2 on a fourth, so it cannot detect a rerun; /runs/:id/jobs reported total_count 0 while attempt 2 sat finished underneath, so zero jobs is ambiguous between never-started and re-queued; and the run object reported queued after its own jobs were terminal. filter=all is the reader that sees all three. Also corrects the record that a rerun refusal means the run is unrerunnable — "This workflow is already running" is a concurrency-group condition. Co-Authored-By: Claude Opus 5 --- docs/runbooks/reading-github-actions-state.md | 77 +++++++++++++++++++ 1 file changed, 77 insertions(+) create mode 100644 docs/runbooks/reading-github-actions-state.md diff --git a/docs/runbooks/reading-github-actions-state.md b/docs/runbooks/reading-github-actions-state.md new file mode 100644 index 000000000..8680039df --- /dev/null +++ b/docs/runbooks/reading-github-actions-state.md @@ -0,0 +1,77 @@ +# Reading GitHub Actions state for a PR + +Three seats spent most of 2026-08-26 diagnosing "the checks on my PR are +missing" and reached four mutually contradictory conclusions, all from correct +commands. Every contradiction came from the same place: **the run object and +the PR's check list are summaries, and each one is lossy in a different +direction.** This runbook names which reader answers which question. + +## An absent or short check list has at least five causes + +They render identically on the PR page — a row that is missing, or grey. They +are not the same problem and they do not share a remedy. + +| Cause | How it looks | Discriminator | Remedy | +|---|---|---|---| +| Run never created | check absent from `gh pr checks` | no run at that SHA in `gh run list --branch ` | needs a NEW event: push, or close/reopen | +| `startup_failure` | check absent from `gh pr checks` | run exists, `conclusion=startup_failure`, 0 jobs | close/reopen | +| Queued, pool saturated | grey/pending | run exists, `status=queued`, age climbing | wait — re-triggering adds to the back of the line | +| Superseded by concurrency | run `cancelled` | a NEWER run exists at a newer SHA in the same group | none needed; read the newer run | +| Jobs cancelled at 0 steps | run `failure` | jobs `cancelled`, `steps=0`, and no newer run to have superseded them | `gh run rerun ` | + +The last row is the one that misleads, because a run-level `failure` reads as +"the tests failed" when nothing ever executed. **Discriminate on job count and +step count, not on the run's conclusion.** + +## Three fields that do not mean what their names promise + +**`run_attempt` cannot detect whether a rerun happened.** Measured on four runs +rerun within the same minute: `32985824262`, `32985824328` and `32985899276` +each went `completed/failure` → `queued/null` with `run_attempt` still `1` and +`previous_attempt_url` unset. `32985813845` went to `run_attempt=2`. An audit +that filters on `run_attempt > 1` will conclude no rerun ever happened, on a +repo where reruns are landing. + +**`/runs/:id/jobs` returns only the latest attempt, and returns nothing while +that attempt is queued.** On run `32985813845` it reported `total_count: 0` +minutes after attempt 2 had finished successfully. Zero jobs is therefore +ambiguous between *never started* and *re-queued, jobs not yet created* — and +the first is exactly what a `startup_failure` looks like. Use +`/runs/:id/jobs?filter=all`, which lists every attempt with its own +`run_attempt`, conclusion and step count: + +```bash +gh api "repos///actions/runs//jobs?filter=all" \ + -q '.jobs[] | "\(.run_attempt): \(.name) \(.status)/\(.conclusion) steps=\(.steps|length)"' +# 1: Source changed ⇒ version bumped completed/cancelled steps=0 +# 2: Source changed ⇒ version bumped completed/success steps=5 +``` + +**The run object lags its own jobs.** At the moment the listing above was +captured, `gh api repos///actions/runs/32985813845` still reported +`status=queued conclusion=null`. The job records were already terminal. When +the two disagree, the per-attempt job listing is the one that has run. + +## The rerun refusal is not about the run's conclusion + +`gh run rerun` answering `cannot be rerun; This workflow is already running` +is a **concurrency-group** condition, not a terminal-state one. `tests.yml` +groups on `${{ github.workflow }}-${{ github.event.pull_request.number || +github.sha }}` with `cancel-in-progress: true`, so a queued run for the same PR +blocks a rerun of an older one. Clearing the queued run — or waiting — makes the +same command succeed on the same id. A refusal is not evidence that the class +of run is unrerunnable. + +## Order to work in + +1. `gh run list --branch ` — does a run exist at this head SHA at all? + `gh pr checks` cannot answer this; a never-created run and a queued run are + the same empty row there. +2. If it exists, `gh api .../runs/` for status, and + `.../jobs?filter=all` for what actually executed, per attempt. +3. Join on SHA before concluding anything. A run listed on the branch may + belong to a previous head. + +Related: [`docs/development/review-checklist.md`](../development/review-checklist.md) +rule on joining two measurements over a time-invariant predicate — the same +failure, one layer up. From 7f62deb9a7d57802dfcc31b0357f167e59d5ecce Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Wed, 26 Aug 2026 09:22:25 -0700 Subject: [PATCH 02/14] docs(runbooks): cite the checklist rule as the open PR it is, not as landed Co-Authored-By: Claude Opus 5 --- docs/runbooks/reading-github-actions-state.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/runbooks/reading-github-actions-state.md b/docs/runbooks/reading-github-actions-state.md index 8680039df..079947483 100644 --- a/docs/runbooks/reading-github-actions-state.md +++ b/docs/runbooks/reading-github-actions-state.md @@ -72,6 +72,6 @@ of run is unrerunnable. 3. Join on SHA before concluding anything. A run listed on the branch may belong to a previous head. -Related: [`docs/development/review-checklist.md`](../development/review-checklist.md) -rule on joining two measurements over a time-invariant predicate — the same -failure, one layer up. +Related: PR #1240 proposes a review-checklist rule that a join across two +measurements needs a time-invariant predicate — the same failure one layer up. +Step 3 above is that rule applied to a single PR. From a40cee009736bf0215f457bf4e03c7f8097ef888 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Wed, 26 Aug 2026 09:26:24 -0700 Subject: [PATCH 03/14] docs(runbooks): a pending check can belong to a run that ended an hour ago MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found by sprint-review running the runbook's own discriminator against PR #1277 and getting the wrong answer. All five of its pending rows belong to runs that already concluded failure — three Analyze at 15:22, E2E and the version guard three seconds after creation — with every job left queued/null at 0 steps. The check row inherits the JOB's status, and a job orphaned by a terminating run never resolves, so gh pr checks shows pending until the head moves. The table's discriminator was job count and step count. Those say what went wrong, never whether it is still going. Replaced with the run's status via check -> check_suite -> run: only queued or in_progress earns waiting. Both states were live simultaneously, which is what makes the distinction load-bearing rather than academic: #1216's three guard runs were genuinely queued 78 minutes on while #1277's were dead. Co-Authored-By: Claude Opus 5 --- docs/runbooks/reading-github-actions-state.md | 24 +++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/docs/runbooks/reading-github-actions-state.md b/docs/runbooks/reading-github-actions-state.md index 079947483..f1284d8f4 100644 --- a/docs/runbooks/reading-github-actions-state.md +++ b/docs/runbooks/reading-github-actions-state.md @@ -18,10 +18,24 @@ are not the same problem and they do not share a remedy. | Queued, pool saturated | grey/pending | run exists, `status=queued`, age climbing | wait — re-triggering adds to the back of the line | | Superseded by concurrency | run `cancelled` | a NEWER run exists at a newer SHA in the same group | none needed; read the newer run | | Jobs cancelled at 0 steps | run `failure` | jobs `cancelled`, `steps=0`, and no newer run to have superseded them | `gh run rerun ` | +| Orphaned jobs | check shows **`pending`, forever** | run `completed/failure`, jobs still `queued/null` at `steps=0` | `gh run rerun ` — waiting never resolves it | -The last row is the one that misleads, because a run-level `failure` reads as -"the tests failed" when nothing ever executed. **Discriminate on job count and -step count, not on the run's conclusion.** +Two of these mislead in opposite directions. A run-level `failure` reads as +"the tests failed" when nothing ever executed. And a check row reporting +`pending` can belong to a run that terminated over an hour ago: the row +inherits its **job's** status, and a job orphaned by a terminating run stays +`queued/null` permanently. `gh pr checks` will show it as pending until the +head moves. + +**So the discriminator is the run's `status`, not the check's.** Map check → +`check_suite` → run, and only `status: in_progress` or `queued` earns waiting. +Both states were live on this repo simultaneously on 2026-08-26: PR #1216's +three guard runs were genuinely `queued` 78 minutes after creation, while PR +#1277's five pending rows all belonged to runs that had already concluded +`failure` — one at 15:22, two more three seconds after they were created. + +Job count and step count then tell you *what* went wrong; they cannot tell you +whether it is still going. ## Three fields that do not mean what their names promise @@ -64,7 +78,9 @@ of run is unrerunnable. ## Order to work in -1. `gh run list --branch ` — does a run exist at this head SHA at all? +1. `gh run list --branch ` — or better, + `gh api "repos///actions/runs?head_sha="`. Does a run exist at + this head SHA at all? `gh pr checks` cannot answer this; a never-created run and a queued run are the same empty row there. 2. If it exists, `gh api .../runs/` for status, and From 019e8400175ea5b4b7b2756d451efa7232e74227 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Wed, 26 Aug 2026 09:40:03 -0700 Subject: [PATCH 04/14] docs(runbooks): a re-trigger takes ~20 minutes, so an early negative is not a negative Two seats independently concluded close/reopen produces no runs, one checking at 17 minutes and one at 2. Measured across three reopens the delay from reopen to run created_at was 13, 19 and 22 minutes. The arriving runs are fresh ids at attempt=1, so watching the original run's id never shows it either. Co-Authored-By: Claude Opus 5 --- docs/runbooks/reading-github-actions-state.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/runbooks/reading-github-actions-state.md b/docs/runbooks/reading-github-actions-state.md index f1284d8f4..0f766e67d 100644 --- a/docs/runbooks/reading-github-actions-state.md +++ b/docs/runbooks/reading-github-actions-state.md @@ -66,6 +66,25 @@ captured, `gh api repos///actions/runs/32985813845` still reported `status=queued conclusion=null`. The job records were already terminal. When the two disagree, the per-attempt job listing is the one that has run. +## A re-trigger takes ~20 minutes to produce a run + +Close/reopen re-fires every `pull_request` workflow without moving the head, +which is what makes it the right lever over an empty commit when a run was +never created. But the runs do not appear promptly. Measured on 2026-08-26, +reopen timestamp → run `created_at`: + +| PR | reopened | runs created | delay | +|---|---|---|---| +| #1277 | 15:44:40Z | 15:57:56Z | 13m | +| #1275 | 15:57:30Z | 16:19:58Z | 22m | +| #1275 | 16:11:42Z | 16:30:30Z | 19m | + +Two of us independently concluded "close/reopen produces no runs" by checking +at 2 and 17 minutes. **A negative measured inside ~25 minutes is not a +negative.** The runs that do arrive are fresh ids with `event=pull_request` and +`run_attempt=1` — they do not reuse the run you were looking at, so watching +the old run's id will never show you the answer either. + ## The rerun refusal is not about the run's conclusion `gh run rerun` answering `cannot be rerun; This workflow is already running` From 26e425046f7fd511d89ca2be4ab83c9e03f9ff28 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Wed, 26 Aug 2026 09:41:46 -0700 Subject: [PATCH 05/14] docs(runbooks): the re-trigger delay is not a quantity you can measure My previous commit gave a table of reopen -> run delays as 13, 19 and 22 minutes. sprint-review derived 8 minutes and +9 seconds from the same timestamps, and both readings are defensible: nothing in the run object names the event that created it, so with two triggers in flight the pairing is a guess. Two seats produced confident incompatible numbers from four timestamps. What the data does support is a bound and a shape. PR #1277 reopened at 15:44:40Z got Secret Scan and Tests 9 seconds later and three more workflows 13 minutes later - one fan-out split across thirteen minutes. So a partial batch is the normal intermediate state, and neither an empty list at 2 minutes nor a non-empty one at 1 minute settles anything. Count the workflows you expect. Co-Authored-By: Claude Opus 5 --- docs/runbooks/reading-github-actions-state.md | 48 ++++++++++++------- 1 file changed, 30 insertions(+), 18 deletions(-) diff --git a/docs/runbooks/reading-github-actions-state.md b/docs/runbooks/reading-github-actions-state.md index 0f766e67d..504573896 100644 --- a/docs/runbooks/reading-github-actions-state.md +++ b/docs/runbooks/reading-github-actions-state.md @@ -66,24 +66,36 @@ captured, `gh api repos///actions/runs/32985813845` still reported `status=queued conclusion=null`. The job records were already terminal. When the two disagree, the per-attempt job listing is the one that has run. -## A re-trigger takes ~20 minutes to produce a run - -Close/reopen re-fires every `pull_request` workflow without moving the head, -which is what makes it the right lever over an empty commit when a run was -never created. But the runs do not appear promptly. Measured on 2026-08-26, -reopen timestamp → run `created_at`: - -| PR | reopened | runs created | delay | -|---|---|---|---| -| #1277 | 15:44:40Z | 15:57:56Z | 13m | -| #1275 | 15:57:30Z | 16:19:58Z | 22m | -| #1275 | 16:11:42Z | 16:30:30Z | 19m | - -Two of us independently concluded "close/reopen produces no runs" by checking -at 2 and 17 minutes. **A negative measured inside ~25 minutes is not a -negative.** The runs that do arrive are fresh ids with `event=pull_request` and -`run_attempt=1` — they do not reuse the run you were looking at, so watching -the old run's id will never show you the answer either. +## A re-trigger fans out partially, and stragglers arrive minutes later + +Close/reopen re-fires `pull_request` workflows without moving the head, which is +what makes it the right lever over an empty commit when a run was never created. +It does work. But it does **not** deliver the whole fan-out at once, so an early +check tells you almost nothing. + +Measured on 2026-08-26. PR #1277 reopened at 15:44:40Z: `Secret Scan` and +`Tests` were created 9 seconds later, and three more workflows — +`Package Version Guard`, `Playwright Tests`, `PR Base Freshness` — only at +15:57:56Z, 13 minutes on. Same trigger, same PR, one fan-out split across +thirteen minutes. PR #1275 had two reopens (15:57:30Z, 16:11:42Z) and two run +batches (16:19:58Z, 16:30:30Z); depending on how you pair them the delay is +either 8 and 19 minutes or 22 and 19. + +**That pairing is the trap.** Nothing in the run object names the event that +created it, so with more than one trigger in flight the delay is not a quantity +you can measure — two of us independently derived confident and incompatible +numbers from the same four timestamps. What the data supports is a bound and a +shape: **some runs land in seconds, some take up to ~20 minutes, and a partial +batch is the normal intermediate state, not evidence of a failure.** + +Practical consequences: + +- Do not conclude "the re-trigger did nothing" inside ~25 minutes. Both of us + did, at 2 and 17 minutes. +- Do not conclude it worked because *some* runs appeared. Count the workflows + you expect, not whether the list is non-empty. +- The arriving runs are fresh ids at `run_attempt=1`; they never reuse the run + you were watching, so watching that id shows you nothing either way. ## The rerun refusal is not about the run's conclusion From 146ba0e84510d3c77878b1ce70246f7cefd79319 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Wed, 26 Aug 2026 09:43:49 -0700 Subject: [PATCH 06/14] docs(runbooks): check-suites answer the question the run list cannot Found by sprint-review: a dispatched workflow allocates a github-actions check-suite within seconds whether or not its run ever starts, so the presence of a suite separates never-dispatched from dispatched-and-stuck. Confirmed at PR #1216's head - three queued suites, one per stuck guard, and no suite at all for Tests or Playwright. Two traps recorded with it. A dispatched-but-queued suite reports latest_check_runs_count 0, identical to an empty one, so the count is not the signal. And app-driven runs are recorded against refs/pull//head, so ?branch= returns zero for a PR that visibly has runs. Co-Authored-By: Claude Opus 5 --- docs/runbooks/reading-github-actions-state.md | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/docs/runbooks/reading-github-actions-state.md b/docs/runbooks/reading-github-actions-state.md index 504573896..547db9279 100644 --- a/docs/runbooks/reading-github-actions-state.md +++ b/docs/runbooks/reading-github-actions-state.md @@ -66,6 +66,33 @@ captured, `gh api repos///actions/runs/32985813845` still reported `status=queued conclusion=null`. The job records were already terminal. When the two disagree, the per-attempt job listing is the one that has run. +## Check-suites separate never-dispatched from dispatched-and-stuck + +This is the sharpest instrument in this document, and it answers the question +the run list cannot: was the workflow ever dispatched at all? Every dispatched +workflow allocates a `github-actions` check-suite within seconds, *whether or +not its run ever starts*. So: + +```bash +gh api "repos///commits//check-suites?per_page=50" \ + -q '.check_suites[] | "\(.created_at) app=\(.app.slug) \(.status)/\(.conclusion) runs=\(.latest_check_runs_count)"' +``` + +Measured at PR #1216's head on 2026-08-26: three `github-actions` suites created +15:08:11, 15:08:13 and 15:09:50, all `queued`, one per stuck guard workflow — +and **no suite at all** for `Tests` or `Playwright Tests`. Those two were never +dispatched. At PR #1277's head, two suites sit at `completed/startup_failure` +with zero runs, which is the terminal state-1 case wearing the same face. + +**Do not key on `latest_check_runs_count`.** A dispatched-but-queued suite +reports `runs: 0`, identical to an empty one. The suite's *existence at that +sha* is the signal; its count is not. + +One collection caveat that cost time here: CodeQL and other app-driven runs are +recorded against `refs/pull//head`, not the branch, so +`?branch=` can return zero for a PR that visibly has runs. Query +by `head_sha` or via the commit's check-suites instead. + ## A re-trigger fans out partially, and stragglers arrive minutes later Close/reopen re-fires `pull_request` workflows without moving the head, which is From 5d9c80ac7d4b2173f3c5042ea74a4f2caf0160e7 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Wed, 26 Aug 2026 10:16:08 -0700 Subject: [PATCH 07/14] =?UTF-8?q?docs(runbooks):=20one=20close/reopen=20pa?= =?UTF-8?q?iring=20is=20determined=20=E2=80=94=2010=20minutes,=20complete?= =?UTF-8?q?=20fan-out?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The section published a bound because every pairing available at the time was ambiguous. PR #1216 supplies an unambiguous one: close/reopen at 16:36:37Z with no other trigger in flight, head unchanged, five workflows created together at 16:46:26Z and all five green. Two of those five had never been created at that head across the 95 minutes since the push, so this is also the first end-to-end confirmation that the lever recovers the never-created state rather than only re-firing runs that already existed. Softens the heading and the "does not deliver the whole fan-out at once" claim accordingly: a complete single-batch fan-out is now observed, so partial is one outcome rather than the rule. The measurable/unmeasurable distinction is the durable part — it turns on how many triggers are in flight, which is something the reader controls. Co-Authored-By: Claude Opus 5 --- docs/runbooks/reading-github-actions-state.md | 21 ++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/docs/runbooks/reading-github-actions-state.md b/docs/runbooks/reading-github-actions-state.md index 547db9279..703b7b78c 100644 --- a/docs/runbooks/reading-github-actions-state.md +++ b/docs/runbooks/reading-github-actions-state.md @@ -93,12 +93,13 @@ recorded against `refs/pull//head`, not the branch, so `?branch=` can return zero for a PR that visibly has runs. Query by `head_sha` or via the commit's check-suites instead. -## A re-trigger fans out partially, and stragglers arrive minutes later +## A re-trigger may fan out partially, and stragglers arrive minutes later Close/reopen re-fires `pull_request` workflows without moving the head, which is what makes it the right lever over an empty commit when a run was never created. -It does work. But it does **not** deliver the whole fan-out at once, so an early -check tells you almost nothing. +It does work. But it does **not reliably** deliver the whole fan-out at once — +sometimes it does and sometimes it does not — so an early check tells you almost +nothing either way. Measured on 2026-08-26. PR #1277 reopened at 15:44:40Z: `Secret Scan` and `Tests` were created 9 seconds later, and three more workflows — @@ -115,6 +116,20 @@ numbers from the same four timestamps. What the data supports is a bound and a shape: **some runs land in seconds, some take up to ~20 minutes, and a partial batch is the normal intermediate state, not evidence of a failure.** +**One pairing is determined, and it gives 10 minutes.** PR #1216 was +close/reopened at 16:36:37Z with no other trigger in flight — head unchanged +throughout, no pushes, no reruns. At 16:46:26Z all five workflows were created +in a single batch, and all five concluded `success`. Two of them, `Tests` and +`Playwright Tests`, had never been created at that head at all, 95 minutes +after the push that should have produced them. So this is also the only +end-to-end confirmation in this document that the lever recovers the +never-created case rather than merely re-firing what already existed. + +Note what made it measurable: exactly one trigger and exactly one batch. The +delay is not unknowable in general — it is unknowable whenever you have more +triggers in flight than batches to match them to, which is the situation you +create by re-firing a second time while waiting. + Practical consequences: - Do not conclude "the re-trigger did nothing" inside ~25 minutes. Both of us From c27e3ee1bba2ccabea7e8d083cbf05586eb26440 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Wed, 26 Aug 2026 10:18:02 -0700 Subject: [PATCH 08/14] docs(runbooks): an absent check-suite is a claim about the future, not a reading MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "Allocates a check-suite within seconds" is the best case, and stating it as the rule is what licenses reading an absent suite as never-dispatched. Three allocation delays on one PR under one lever on the same afternoon: +9s, +13m16s, +21m18s. Two readings were taken inside that window and both were wrong. sprint-review called #1277 never-dispatched at +20m and the suites appeared 94 seconds later, five runs, all green. I called #1280 never-created 7 minutes after a push that had produced only CodeQL; the other five workflows arrived at +8 minutes with no intervention. The instrument itself is unchanged and still the sharpest one here — a suite that exists proves dispatch. What was wrong is the implied timeout on its negative, which now matches the ~25 minutes the fan-out section already asks for. Co-Authored-By: Claude Opus 5 --- docs/runbooks/reading-github-actions-state.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/docs/runbooks/reading-github-actions-state.md b/docs/runbooks/reading-github-actions-state.md index 703b7b78c..075cca341 100644 --- a/docs/runbooks/reading-github-actions-state.md +++ b/docs/runbooks/reading-github-actions-state.md @@ -70,8 +70,9 @@ the two disagree, the per-attempt job listing is the one that has run. This is the sharpest instrument in this document, and it answers the question the run list cannot: was the workflow ever dispatched at all? Every dispatched -workflow allocates a `github-actions` check-suite within seconds, *whether or -not its run ever starts*. So: +workflow allocates a `github-actions` check-suite *whether or not its run ever +starts*, so a suite that exists proves dispatch. **An absent suite proves +nothing until you have waited — see the timeout below.** So: ```bash gh api "repos///commits//check-suites?per_page=50" \ @@ -84,6 +85,17 @@ and **no suite at all** for `Tests` or `Playwright Tests`. Those two were never dispatched. At PR #1277's head, two suites sit at `completed/startup_failure` with zero runs, which is the terminal state-1 case wearing the same face. +**The absence of a suite is only evidence after ~25 minutes, and reading it +sooner is the mistake this instrument invites.** Allocation is usually fast and +occasionally is not; measured on one PR, one lever, on the same afternoon: +`+9s`, `+13m16s`, `+21m18s`. A reading taken at +20 minutes and called +never-dispatched was contradicted 94 seconds later by five green runs. A second +reading, taken 7 minutes after a push that had produced only CodeQL, called it +never-created; the other five workflows arrived at +8 minutes untouched. The +instrument is sound — a suite that exists really does prove dispatch — but the +*absence* of one is a claim about the future, so give it the same ~25 minutes +the fan-out section asks for before acting on it. + **Do not key on `latest_check_runs_count`.** A dispatched-but-queued suite reports `runs: 0`, identical to an empty one. The suite's *existence at that sha* is the signal; its count is not. From f88fcdf8001352334eeb13c2d12de6bc9b6d8cd6 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Wed, 26 Aug 2026 10:44:49 -0700 Subject: [PATCH 09/14] docs(runbooks): a second determined pairing gives 11 seconds, not 10 minutes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I published "one pairing is determined, and it gives 10 minutes" an hour ago. #1271 reopened at 17:37:06Z produced all five runs at 17:37:17Z. Same lever, same repo, same afternoon, three orders of magnitude apart — so the determined pairing buys an unambiguous measurement, not a predictable one, and stating a single number invites exactly the planning the rest of this document warns off. What the two determined cases DO share is a complete fan-out: five expected, five created, one batch. Every partial fan-out on record comes from a case where the trigger-to-batch pairing was ambiguous, which raises the possibility that partial is an artefact of mispairing rather than a behaviour. Flagged as n=2 rather than asserted. Co-Authored-By: Claude Opus 5 --- docs/runbooks/reading-github-actions-state.md | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/docs/runbooks/reading-github-actions-state.md b/docs/runbooks/reading-github-actions-state.md index 075cca341..7e92638f2 100644 --- a/docs/runbooks/reading-github-actions-state.md +++ b/docs/runbooks/reading-github-actions-state.md @@ -128,7 +128,7 @@ numbers from the same four timestamps. What the data supports is a bound and a shape: **some runs land in seconds, some take up to ~20 minutes, and a partial batch is the normal intermediate state, not evidence of a failure.** -**One pairing is determined, and it gives 10 minutes.** PR #1216 was +**Two pairings are determined, and they give 10 minutes and 11 seconds.** PR #1216 was close/reopened at 16:36:37Z with no other trigger in flight — head unchanged throughout, no pushes, no reruns. At 16:46:26Z all five workflows were created in a single batch, and all five concluded `success`. Two of them, `Tests` and @@ -137,8 +137,19 @@ after the push that should have produced them. So this is also the only end-to-end confirmation in this document that the lever recovers the never-created case rather than merely re-firing what already existed. -Note what made it measurable: exactly one trigger and exactly one batch. The -delay is not unknowable in general — it is unknowable whenever you have more +PR #1271, same lever under the same conditions, went the other way: reopened +17:37:06Z, five runs created 17:37:17Z — **eleven seconds**, four of them +already `success` seven minutes later. + +So a determined pairing does not buy you a predictable delay; the two +unambiguous measurements are 10 minutes and 11 seconds. What both share, and +what the ambiguous cases never showed, is that the fan-out was **complete** — +five expected workflows, five created, one batch. On this evidence (n=2) the +partial fan-outs above may be an artefact of pairing runs to the wrong trigger +rather than a behaviour of the lever. Do not plan around either number. + +Note what made these two measurable: exactly one trigger and exactly one batch. +The delay is not unknowable in general — it is unknowable whenever you have more triggers in flight than batches to match them to, which is the situation you create by re-firing a second time while waiting. From f33f785a60ac96ade8c4772732cc8874dc698898 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Wed, 26 Aug 2026 10:54:11 -0700 Subject: [PATCH 10/14] docs(runbook): partial fan-out is a behaviour, not a mispairing artefact (n=3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - #1271 reopened 17:37:05Z, not :06 — twelve seconds, not eleven. - Retract the n=2 "partial may be an artefact" flag. #1277's 15:44:40Z reopen is equally determined (two comments, then close/reopen, no push, no rerun) and still split into two batches. Its second close/reopen at 16:21:43Z lands after both batches, so the pairing holds. - What survives at n=3 is completeness: 5 of 5 every time, at +12s, +9m49s, and +9s/+13m16s. - Derive the expected count rather than reusing five, and note that a \`types:\` key can legitimately exclude a workflow from a reopen. - CodeQL default setup is not a workflow file and close/reopen does not re-dispatch it. - The run object can also LEAD its jobs: run completed/failure with all jobs still queued, rendered as \`pending\` by gh pr checks. --- docs/runbooks/reading-github-actions-state.md | 53 +++++++++++++++---- 1 file changed, 43 insertions(+), 10 deletions(-) diff --git a/docs/runbooks/reading-github-actions-state.md b/docs/runbooks/reading-github-actions-state.md index 7e92638f2..4b9b49963 100644 --- a/docs/runbooks/reading-github-actions-state.md +++ b/docs/runbooks/reading-github-actions-state.md @@ -63,8 +63,19 @@ gh api "repos///actions/runs//jobs?filter=all" \ **The run object lags its own jobs.** At the moment the listing above was captured, `gh api repos///actions/runs/32985813845` still reported -`status=queued conclusion=null`. The job records were already terminal. When -the two disagree, the per-attempt job listing is the one that has run. +`status=queued conclusion=null`. The job records were already terminal. + +**And it leads them, too — so neither field is authoritative on its own.** Run +`32985816249` (#1271's CodeQL, `event: dynamic`) reads `completed/failure` while +all three of its jobs are still `queued`, `conclusion: null`, `completed_at: +null`, hours after `started_at`. `gh pr checks` renders those as `pending` with +duration `0`, which is indistinguishable from a job that is genuinely about to +run. The sibling run `32984068926` on #1216 has the same shape one step later — +jobs `completed/cancelled` after 15m4s — and `gh pr checks` renders *those* as +`fail`. So a `pending` row can belong to a run that has already failed, and a +`fail` row can be a cancellation rather than a test failure. Read the run +conclusion and the per-attempt job listing together; when they disagree, the +disagreement is the finding. ## Check-suites separate never-dispatched from dispatched-and-stuck @@ -138,15 +149,24 @@ end-to-end confirmation in this document that the lever recovers the never-created case rather than merely re-firing what already existed. PR #1271, same lever under the same conditions, went the other way: reopened -17:37:06Z, five runs created 17:37:17Z — **eleven seconds**, four of them +17:37:05Z, five runs created 17:37:17Z — **twelve seconds**, four of them already `success` seven minutes later. -So a determined pairing does not buy you a predictable delay; the two -unambiguous measurements are 10 minutes and 11 seconds. What both share, and -what the ambiguous cases never showed, is that the fan-out was **complete** — -five expected workflows, five created, one batch. On this evidence (n=2) the -partial fan-outs above may be an artefact of pairing runs to the wrong trigger -rather than a behaviour of the lever. Do not plan around either number. +So a determined pairing does not buy you a predictable delay; the unambiguous +measurements are 10 minutes and 12 seconds. An earlier draft of this section +added a third claim — that they also shared a *single* batch, and that partial +fan-outs might therefore be an artefact of pairing runs to the wrong trigger. +**That is false.** #1277's 15:44:40Z reopen is equally determined: two +`commented` events, then close/reopen, no push and no rerun in between. It +still split into two batches, 9 seconds and 13 minutes apart. (#1277 does have +a second close/reopen, at 16:21:43Z/16:21:44Z — it lands *after* both batches +and so leaves the pairing intact.) Partial fan-out is a behaviour of the lever, +not a measurement error. + +What does survive at n=3 is **completeness**: all three determined pairings +delivered all five expected workflows eventually — one batch at +12s, one batch +at +9m49s, two batches at +9s and +13m16s. Delay and batch count are both +unpredictable; the final count is not. Do not plan around any of the numbers. Note what made these two measurable: exactly one trigger and exactly one batch. The delay is not unknowable in general — it is unknowable whenever you have more @@ -158,7 +178,20 @@ Practical consequences: - Do not conclude "the re-trigger did nothing" inside ~25 minutes. Both of us did, at 2 and 17 minutes. - Do not conclude it worked because *some* runs appeared. Count the workflows - you expect, not whether the list is non-empty. + you expect, not whether the list is non-empty. Derive that number rather than + reusing this document's five: eight workflow files declare `pull_request`, and + for these three PRs `Deploy Docs` (`paths: docs-site/**`) and `Smoke Tests` + (`paths: k8s/**`, the two Dockerfiles) did not match the diff while + `Release Safety` is `branches: [ v1.0.x ]`, leaving five. A workflow with no + `types:` key defaults to `[opened, synchronize, reopened]`, so none of the + five is excluded from a reopen — but check, because one that pinned + `types: [opened, synchronize]` would legitimately never come back and would + read as a missing run forever. +- **Your expected count is not the whole check list.** CodeQL default setup runs + as `path: dynamic/github-code-scanning/codeql`, `event: dynamic`, with no file + in `.github/workflows/`. Close/reopen does not re-dispatch it, so its three + `Analyze` checks stay on whatever state they were already in while the five + workflow runs come back green. - The arriving runs are fresh ids at `run_attempt=1`; they never reuse the run you were watching, so watching that id shows you nothing either way. From f56061855fb25e951342f6b59b8163598cde17c8 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Wed, 26 Aug 2026 11:06:44 -0700 Subject: [PATCH 11/14] docs(runbook): a queued run is not evidence of a queue MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Measured repo-wide: 11 queued runs, all 11 outlived by a completed successor of the same workflow on the same branch. Live queue depth zero. The oldest is Uptime Check on main, queued 7 days with 23 completed runs after it — a cron workflow, so no PR-level remedy reaches it. - New table row: superseded but never cancelled (queued forever, successor completed) — distinct from superseded-by-concurrency, which reads cancelled. - "Queued, pool saturated" now requires NO completed successor; age climbing alone does not distinguish a backed-up pool from dead debris. - Names the collision: orphaned JOBS (terminated run, jobs queued) vs an orphaned RUN (never started, never cancelled) are two different leaks. - Gives the one-call successor check. Credit: sprint-review ran the repo-wide sweep. --- docs/runbooks/reading-github-actions-state.md | 33 ++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/docs/runbooks/reading-github-actions-state.md b/docs/runbooks/reading-github-actions-state.md index 4b9b49963..59dd9cf4e 100644 --- a/docs/runbooks/reading-github-actions-state.md +++ b/docs/runbooks/reading-github-actions-state.md @@ -15,8 +15,9 @@ are not the same problem and they do not share a remedy. |---|---|---|---| | Run never created | check absent from `gh pr checks` | no run at that SHA in `gh run list --branch ` | needs a NEW event: push, or close/reopen | | `startup_failure` | check absent from `gh pr checks` | run exists, `conclusion=startup_failure`, 0 jobs | close/reopen | -| Queued, pool saturated | grey/pending | run exists, `status=queued`, age climbing | wait — re-triggering adds to the back of the line | +| Queued, pool saturated | grey/pending | run exists, `status=queued`, age climbing, **and no completed successor** | wait — re-triggering adds to the back of the line | | Superseded by concurrency | run `cancelled` | a NEWER run exists at a newer SHA in the same group | none needed; read the newer run | +| Superseded but never cancelled | grey/pending, indefinitely | run `status=queued` **and** a later run of the same workflow on the same branch has `completed` | none — it is dead; read the successor | | Jobs cancelled at 0 steps | run `failure` | jobs `cancelled`, `steps=0`, and no newer run to have superseded them | `gh run rerun ` | | Orphaned jobs | check shows **`pending`, forever** | run `completed/failure`, jobs still `queued/null` at `steps=0` | `gh run rerun ` — waiting never resolves it | @@ -27,6 +28,36 @@ inherits its **job's** status, and a job orphaned by a terminating run stays `queued/null` permanently. `gh pr checks` will show it as pending until the head moves. +## A queued run is not evidence of a queue + +`status=queued` is the one state this document tells you to wait on, so it is +worth knowing that most queued runs on this repo are not waiting for anything. + +Measured 2026-08-26: `?status=queued` returned `total_count: 11` repo-wide, and +**all eleven had a later run of the same workflow on the same branch already +completed.** Live queue depth was zero. Ten sat across four PR branches; the +eleventh was `Uptime Check` on `main`, queued since 2026-08-19 with 23 completed +runs after it — seven days, on a cron workflow, invisible from any PR page and +untouchable by a PR-level remedy. + +These are *not* the same thing as the orphaned **jobs** in the table above. That +state is a terminated run whose jobs never left `queued`; this one is a whole +run that never started and never got cancelled either, despite the concurrency +group that should have swept it. Two different leaks, and "orphaned" gets used +for both — say which layer you mean. + +The check is one call, and it is the difference between "the pool is backed up, +wait" and "this run is dead, read the successor": + +```bash +gh api "repos///actions/runs?branch=&per_page=100" \ + --jq '[.workflow_runs[] | select(.name=="" and .created_at > "" and .status=="completed")] | length' +``` + +Non-zero means the queued run you are watching has already been outlived. A +repo-wide queue-depth number that has not been through this filter measures +accumulated debris, not load. + **So the discriminator is the run's `status`, not the check's.** Map check → `check_suite` → run, and only `status: in_progress` or `queued` earns waiting. Both states were live on this repo simultaneously on 2026-08-26: PR #1216's From 9e8ec060b9bf9fb58f56fe623745f680487f2681 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Wed, 26 Aug 2026 11:08:59 -0700 Subject: [PATCH 12/14] =?UTF-8?q?docs(runbook):=20three=20exclusion=20axes?= =?UTF-8?q?,=20not=20one=20=E2=80=94=20the=20recipe=20failed=20on=20its=20?= =?UTF-8?q?own=20PR?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All three review points confirmed against the files, not the diff: - Playwright Tests IS paths-gated and the bullet omitted it. Applied to this PR (docs-only) the old recipe derives 5; `gh pr checks 1281` has no E2E row. Now carries both worked examples side by side, 5 and 4. - Smoke Tests gates on SEVEN paths, not three. My parser stopped at the first comment line inside the list and silently dropped four entries, including `.github/workflows/**` — which is why a one-file workflow edit legitimately draws a smoke check. - Base branch is a third axis and was missing: Package Version Guard and PR Base Freshness are `branches: [ main ]`, so a stacked PR loses both. #1279 draws 5 where a main-based PR draws 11. Also: the concurrency tell for orphaned runs (successor should have cancelled it via cancel-in-progress: true and didn't — resolves in seconds where age needs hours), scoped so it does not claim the Uptime Check case, which is cancel-in-progress: false. Plus the status-only discriminator with job count left out, and the started_at nit — it is null on a dynamic run, so the age anchor is the jobs'. Review and the discriminator: sprint-review. --- docs/runbooks/reading-github-actions-state.md | 58 +++++++++++++++---- 1 file changed, 48 insertions(+), 10 deletions(-) diff --git a/docs/runbooks/reading-github-actions-state.md b/docs/runbooks/reading-github-actions-state.md index 59dd9cf4e..21852a41b 100644 --- a/docs/runbooks/reading-github-actions-state.md +++ b/docs/runbooks/reading-github-actions-state.md @@ -58,6 +58,26 @@ Non-zero means the queued run you are watching has already been outlived. A repo-wide queue-depth number that has not been through this filter measures accumulated debris, not load. +**A sharper tell than age: the successor should have cancelled it, and didn't.** +All five PR workflows declare `cancel-in-progress: true` on a group keyed by PR +number, so a successor lands in the same group as its queued predecessor and +should sweep it to `cancelled`. Every one of the eleven is still `queued`. An +orphan is not losing the concurrency race — it is absent from the bookkeeping +that would have cancelled it. This tell resolves in seconds where age needs +hours, so reach for it first. It does **not** cover the `Uptime Check` case: +that workflow uses a static group with `cancel-in-progress: false`, so no +successor was ever going to cancel it and age is the only evidence you have. +Keep both. + +So, discriminating on `status` alone, with job count deliberately left out (it +is `0` for orphans *and* for a genuine `startup_failure`): + +``` +completed -> terminal; read the conclusion, ignore age +queued, successor completed -> orphaned; it will never run, ignore it +queued, no successor yet -> unknown; re-check, or force one via close/reopen +``` + **So the discriminator is the run's `status`, not the check's.** Map check → `check_suite` → run, and only `status: in_progress` or `queued` earns waiting. Both states were live on this repo simultaneously on 2026-08-26: PR #1216's @@ -99,7 +119,8 @@ captured, `gh api repos///actions/runs/32985813845` still reported **And it leads them, too — so neither field is authoritative on its own.** Run `32985816249` (#1271's CodeQL, `event: dynamic`) reads `completed/failure` while all three of its jobs are still `queued`, `conclusion: null`, `completed_at: -null`, hours after `started_at`. `gh pr checks` renders those as `pending` with +null`, hours after the *jobs'* `started_at` (the run's own `started_at` is +`null` on a `dynamic` run — do not anchor the age to it). `gh pr checks` renders those as `pending` with duration `0`, which is indistinguishable from a job that is genuinely about to run. The sibling run `32984068926` on #1216 has the same shape one step later — jobs `completed/cancelled` after 15m4s — and `gh pr checks` renders *those* as @@ -209,15 +230,32 @@ Practical consequences: - Do not conclude "the re-trigger did nothing" inside ~25 minutes. Both of us did, at 2 and 17 minutes. - Do not conclude it worked because *some* runs appeared. Count the workflows - you expect, not whether the list is non-empty. Derive that number rather than - reusing this document's five: eight workflow files declare `pull_request`, and - for these three PRs `Deploy Docs` (`paths: docs-site/**`) and `Smoke Tests` - (`paths: k8s/**`, the two Dockerfiles) did not match the diff while - `Release Safety` is `branches: [ v1.0.x ]`, leaving five. A workflow with no - `types:` key defaults to `[opened, synchronize, reopened]`, so none of the - five is excluded from a reopen — but check, because one that pinned - `types: [opened, synchronize]` would legitimately never come back and would - read as a missing run forever. + you expect, not whether the list is non-empty — and **derive that number for + your own PR**. Do not reuse the five below. Eight workflow files declare + `pull_request`, and a workflow is excluded by any of **three** independent + axes: + + | Axis | Who declares it | What it costs you if you forget | + |---|---|---| + | `branches:` | `Package Version Guard`, `PR Base Freshness` (`main`); `Release Safety` (`v1.0.x`) | a **stacked** PR based on another feature branch loses both guards legitimately — #1279 draws 5 checks where a main-based PR draws 11 | + | `paths:` | `Deploy Docs`, `Playwright Tests`, `Smoke Tests` | see below — this is the one most often mis-enumerated | + | `types:` | `Package Version Guard`, `PR Base Freshness`, `Release Safety` | no key defaults to `[opened, synchronize, reopened]`; one pinning `[opened, synchronize]` would never return from a reopen and would read as permanently missing | + + Read the `paths:` lists from the file, in full, every time. `Playwright Tests` + is paths-gated (`frontend/**`, `backend/**`, `e2e/**`, `playwright.config.*`) + and is easy to forget because most PRs match it. `Smoke Tests` gates on seven + entries, not the three you might skim — `k8s/**`, both Dockerfiles, + `_external/clawdbot`, `_external/clawdbot/**`, `dev.sh`, **and + `.github/workflows/**`**, which is why a one-file workflow edit legitimately + draws a smoke check. + + Worked example, and note that the two answers differ. The three incident PRs + each touch `backend/**` on a `main` base: `Deploy Docs` and `Smoke Tests` miss + on paths, `Release Safety` misses on base — **five**. *This* document's PR + touches only `docs/runbooks/*.md`: `Playwright Tests` and `Smoke Tests` also + miss on paths — **four**, and `gh pr checks` on it has no `E2E Tests` row at + all. A recipe that named only Deploy Docs and Smoke Tests as paths-gated would + score that missing row as a fault on the very PR that carries the recipe. - **Your expected count is not the whole check list.** CodeQL default setup runs as `path: dynamic/github-code-scanning/codeql`, `event: dynamic`, with no file in `.github/workflows/`. Close/reopen does not re-dispatch it, so its three From c018ef558ac7a1c4b87aae3102c363f256384333 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Wed, 26 Aug 2026 18:12:21 -0700 Subject: [PATCH 13/14] docs(runbook): gh pr checks collapses by name and hides orphaned rows MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two corrections earned after this PR's head, both on PR #1277: - `gh pr checks` dedupes to the newest row per check name; the `statusCheckRollup` that computes UNSTABLE does not. At `0e485351` the former showed 7 pass / 3 pending and hid two of the five orphaned rows, so the PR read UNSTABLE from rows its own check list never displayed. - The table's remedy for orphaned jobs was `gh run rerun`. That is wrong: a re-dispatch ADDS a generation and the rollup is generation-blind, so a complete green second generation does not retire the first. Only a new SHA clears it. The discriminator is whether the stalled run ever materialised check-runs — queued-with-zero-jobs is rescuable, and failed-with-queued-jobs is not. Co-Authored-By: Claude Opus 5 --- docs/runbooks/reading-github-actions-state.md | 41 +++++++++++++++++-- 1 file changed, 38 insertions(+), 3 deletions(-) diff --git a/docs/runbooks/reading-github-actions-state.md b/docs/runbooks/reading-github-actions-state.md index 21852a41b..4236fa6b1 100644 --- a/docs/runbooks/reading-github-actions-state.md +++ b/docs/runbooks/reading-github-actions-state.md @@ -19,14 +19,49 @@ are not the same problem and they do not share a remedy. | Superseded by concurrency | run `cancelled` | a NEWER run exists at a newer SHA in the same group | none needed; read the newer run | | Superseded but never cancelled | grey/pending, indefinitely | run `status=queued` **and** a later run of the same workflow on the same branch has `completed` | none — it is dead; read the successor | | Jobs cancelled at 0 steps | run `failure` | jobs `cancelled`, `steps=0`, and no newer run to have superseded them | `gh run rerun ` | -| Orphaned jobs | check shows **`pending`, forever** | run `completed/failure`, jobs still `queued/null` at `steps=0` | `gh run rerun ` — waiting never resolves it | +| Orphaned jobs | `pending` forever in `statusCheckRollup` — and possibly **not visible at all** in `gh pr checks` | run `completed/failure`, jobs still `queued/null` at `steps=0` | **a new SHA.** `gh run rerun` and close/reopen both ADD a generation; neither replaces one | Two of these mislead in opposite directions. A run-level `failure` reads as "the tests failed" when nothing ever executed. And a check row reporting `pending` can belong to a run that terminated over an hour ago: the row inherits its **job's** status, and a job orphaned by a terminating run stays -`queued/null` permanently. `gh pr checks` will show it as pending until the -head moves. +`queued/null` permanently. It will read as pending until the head moves — but +only in a reader that shows you every row. + +## `gh pr checks` collapses by name; the rollup that computes UNSTABLE does not + +`gh pr checks` dedupes to the newest row per check name. `statusCheckRollup` — +the field GitHub itself uses to decide `UNSTABLE` — enumerates the jobs of +**every** run at that SHA, including the ones a later generation superseded. + +Those two disagree exactly when it matters. Measured on PR #1277 at +`0e485351`: `gh pr checks` reported 7 pass / 3 pending, and **hid two orphaned +rows entirely**, because a later green run of the same workflow had taken over +the name. The PR still read `UNSTABLE`, from rows its own check list did not +show. Five rows were orphaned; the friendlier instrument could see three. + +So: **`gh pr checks` is the wrong reader for diagnosing UNSTABLE.** A tool that +dedupes by name cannot show you a stale generation sitting beside a fresh one, +and that stale generation is the whole defect. Use +`gh pr view --json statusCheckRollup`, or the jobs endpoint per run. + +## A re-dispatch adds a generation; it never replaces one + +The rollup is SHA-scoped and generation-blind, so a second, wholly green +generation does not retire the first. On #1277 a close/reopen at 16:43Z +produced a complete green set and the PR stayed `UNSTABLE` regardless — the +orphans sat beside the green rows and outvoted them. It cleared only when the +head moved to `489d9847`. + +The discriminator for whether a re-dispatch can rescue a PR is **whether the +stalled run ever materialised check-runs**, not whether it failed: + +- run `queued` with **zero** check-runs — invisible to the rollup; a + re-dispatch genuinely rescues it (PR #1271). +- run `completed/failure` with its **jobs** still `queued` — enumerated + forever; only a new SHA clears it (PR #1277). + +Same symptom, opposite remedy. Enumerate the run set before choosing one. ## A queued run is not evidence of a queue From 69a73f7f739b9f7cee86d96d1ae552edefa9f9ce Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Sat, 29 Aug 2026 07:40:34 -0700 Subject: [PATCH 14/14] =?UTF-8?q?docs(runbook):=20the=20check=20denominato?= =?UTF-8?q?r=20has=20a=20fourth=20axis=20=E2=80=94=20the=20trigger=20event?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `Release Safety` also declares `pull_request_review`, and `branches:` is not applied to that event. Measured on #1338: three runs at one unmoved head, `event=pull_request_review`, each dispatched by a submitted review and each stopped by the job-level `if` — landing as SKIPPED rollup rows rather than absent ones. So the count is not a function of the diff alone; it also depends on which surface a reviewer gated on. #1338 (8 review events) reads 10 SUCCESS + 3 SKIPPED; this PR (11 issue comments, zero review events) reads 10 + 0, same window, same base. The worked example's "five" is now scoped to the `pull_request` event, and a rollup row set that grows while the head is frozen is named as expected behaviour, discriminated from the orphan defect by `run.status` — these are completed/skipped, not queued. Co-Authored-By: Claude Opus 5 --- docs/runbooks/reading-github-actions-state.md | 21 +++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/docs/runbooks/reading-github-actions-state.md b/docs/runbooks/reading-github-actions-state.md index 4236fa6b1..1e8f750f2 100644 --- a/docs/runbooks/reading-github-actions-state.md +++ b/docs/runbooks/reading-github-actions-state.md @@ -267,7 +267,7 @@ Practical consequences: - Do not conclude it worked because *some* runs appeared. Count the workflows you expect, not whether the list is non-empty — and **derive that number for your own PR**. Do not reuse the five below. Eight workflow files declare - `pull_request`, and a workflow is excluded by any of **three** independent + `pull_request`, and a workflow is excluded by any of **four** independent axes: | Axis | Who declares it | What it costs you if you forget | @@ -275,6 +275,7 @@ Practical consequences: | `branches:` | `Package Version Guard`, `PR Base Freshness` (`main`); `Release Safety` (`v1.0.x`) | a **stacked** PR based on another feature branch loses both guards legitimately — #1279 draws 5 checks where a main-based PR draws 11 | | `paths:` | `Deploy Docs`, `Playwright Tests`, `Smoke Tests` | see below — this is the one most often mis-enumerated | | `types:` | `Package Version Guard`, `PR Base Freshness`, `Release Safety` | no key defaults to `[opened, synchronize, reopened]`; one pinning `[opened, synchronize]` would never return from a reopen and would read as permanently missing | + | `event:` | `Release Safety` also declares `pull_request_review` | **`branches:` does not filter that event.** Measured on #1338: three `Release Safety` runs at one unmoved head, `event=pull_request_review`, each dispatched by a submitted review and each stopped by the job-level `if` — so they land as `SKIPPED` rollup rows, not absent ones | Read the `paths:` lists from the file, in full, every time. `Playwright Tests` is paths-gated (`frontend/**`, `backend/**`, `e2e/**`, `playwright.config.*`) @@ -286,11 +287,27 @@ Practical consequences: Worked example, and note that the two answers differ. The three incident PRs each touch `backend/**` on a `main` base: `Deploy Docs` and `Smoke Tests` miss - on paths, `Release Safety` misses on base — **five**. *This* document's PR + on paths, `Release Safety` misses on base — **five**, *for the `pull_request` + event*. *This* document's PR touches only `docs/runbooks/*.md`: `Playwright Tests` and `Smoke Tests` also miss on paths — **four**, and `gh pr checks` on it has no `E2E Tests` row at all. A recipe that named only Deploy Docs and Smoke Tests as paths-gated would score that missing row as a fault on the very PR that carries the recipe. + + **And the same trap has a second form, on an axis this table did not have + until now: the count is not a function of the diff alone — it also depends on + where your reviewer chose to write.** A submitted review dispatches + `Release Safety` regardless of base, because `branches:` is not applied to + `pull_request_review`; the job's `if: base.ref == 'v1.0.x'` is what stops it, + and a stopped job is a `SKIPPED` row rather than no row. So a PR gated three + times through the *reviews* surface carries three more rollup rows than an + identical PR gated through *issue comments*, at the same head, with the same + diff. Measured across two PRs in one window: #1338 (8 review events) reads + 10 SUCCESS + 3 SKIPPED; this document's PR (11 issue comments, zero review + events) reads 10 SUCCESS + 0 SKIPPED. Neither number is wrong. **A rollup row + set that grows while the head is frozen is the expected behaviour here, not + the orphan defect** — the discriminator is the same one this document already + gives: read `run.status`. These are `completed/skipped`, not `queued`. - **Your expected count is not the whole check list.** CodeQL default setup runs as `path: dynamic/github-code-scanning/codeql`, `event: dynamic`, with no file in `.github/workflows/`. Close/reopen does not re-dispatch it, so its three