From 2116cb9ca26242bdd8aeaf3a2bcbbb90a14d5ac9 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Sat, 22 Aug 2026 12:08:01 -0700 Subject: [PATCH 01/12] =?UTF-8?q?docs(ax):=20entry=2041=20=E2=80=94=20a=20?= =?UTF-8?q?conflicting=20PR=20loses=20its=20suite=20and=20renders=20green?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three PRs reported as green at their heads. #1109 had squash-merged four minutes before I pushed two commits to its branch, so the PR reported a green rollup for the parent commit while the fix sat untested on a closed branch. #1120 and #1128 were CONFLICTING, which means no merge ref, which means tests.yml never dispatched — leaving only the push-triggered CodeQL jobs and a four-for-four green page. Both failures are absences, and the rollup folds over the checks that exist rather than the checks that should. Records the sha-level check instead, and the rule that a stack must be rebased the moment its base squash-merges. Co-Authored-By: Claude Opus 5 --- docs/development/agent-experience-audit.md | 60 ++++++++++++++++++++++ 1 file changed, 60 insertions(+) diff --git a/docs/development/agent-experience-audit.md b/docs/development/agent-experience-audit.md index 5952d7105..6f3e1d998 100644 --- a/docs/development/agent-experience-audit.md +++ b/docs/development/agent-experience-audit.md @@ -2240,3 +2240,63 @@ as "older response shape," not "your message reached nobody." the twenty minutes this cost tracing runs, events, and a just-landed deploy that had nothing to do with it — proximity to a deploy makes every bug look like a regression. + +## 41. A conflicting PR loses its whole test suite, and renders as green (2026-08-22, pod-architect) + +> Numbering assumes #1122 (entry 39) and #1132 (entry 40) land first. If they +> merge in a different order, renumber this one rather than them. + +Three stacked threading PRs sat at heads I had just pushed. Their pages showed +green ticks. I reported them as green in a task update. All three reports were +wrong, in two different ways, and both ways look identical to a passing build. + +**What was actually true.** + +`#1109` had squash-merged at 18:50:59Z. At ~18:55 I pushed two more commits to +its branch — a real fix, with tests. The branch ref moved; the PR was already +closed, so nothing dispatched and nothing merged them. `gh pr view` kept +reporting `headRefOid` as the pre-merge commit, so the PR page showed a full +green rollup **for the parent commit**, three commits behind the branch. + +`#1120` and `#1128` were `CONFLICTING/DIRTY`: `main` had taken the squashed +2/4 while their branches still carried the unsquashed originals. GitHub builds +`pull_request` events against a *merge ref*. A conflicting PR has no merge ref, +so `tests.yml` never dispatched at all. What remained was CodeQL — which +triggers on `push`, needs no merge ref, and passed. **Four checks, four green +ticks, zero tests.** + +**Why the instrument can't tell you.** + +Both failures are absences. A commit with no check runs has no failures. A +workflow that never dispatched leaves no red X — it leaves nothing, and the UI +renders nothing as clean. The rollup is a fold over the checks that exist; it +has no opinion about the checks that should exist. Reading it answers "did +anything fail," which is not the question — the question is "did the suite run +on *this* commit." + +The same shape has now been recorded here four times under different names +(entries 34, 35, 37). It keeps recurring because every instance is a status +read against the wrong object: the registry instead of the consumer, the +workflow's clock instead of the pod's, the parent commit instead of the head. + +**Rules earned.** + +- **Verify by sha, not by PR.** `gh api repos/:o/:r/commits//check-runs` + and compare the sha you queried to the branch tip you pushed. A PR's + `headRefOid` can lag its own ref, and on a merged PR it stops updating + entirely. If the count of check runs is zero, that is the loudest possible + signal, and it prints as silence. +- **Check `mergeable` before reading `statusCheckRollup`.** `CONFLICTING` + means the rollup is describing a smaller set of checks than you think, so + the rollup is not evidence about the suite. Read the two together or + neither. +- **A squash merge orphans anything pushed to that branch afterwards.** The + window is as long as your commit takes. Before pushing to a branch you + believe is open, confirm it: a closed PR accepts the push, moves the ref, + and reports nothing. Recovery is a cherry-pick onto a fresh branch off + `main` — cheap, but only if you notice. +- **A stack does not survive its own base squash-merging.** The child now + conflicts on every file the squash touched, and — per the above — goes + quiet rather than red. Rebase children onto `main` immediately after a + parent lands; do not wait for a review to surface it, because the review + will be looking at the same green ticks. From 6ee44bb36bb152ecadb44ef25b01bfe62c7567bb Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Sat, 22 Aug 2026 13:19:06 -0700 Subject: [PATCH 02/12] =?UTF-8?q?docs(ax):=20entry=2041=20was=20wrong=20ab?= =?UTF-8?q?out=20the=20mechanism=20=E2=80=94=20there=20are=20two?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit @sprint-review falsified the sentence "a conflicting PR never dispatches tests" by finding eight green runs on 3f31d103, a conflicting head, Tier 1 among them. Their measurement was right and my sentence was wrong. Resolved by timestamps rather than by argument, and the answer is that both of us had half of it: 9366e11e pushed 18:53:21Z, conflict began 18:50:59Z but was not yet computed -> full 11-check dispatch 4942ad3d pushed 18:57:12Z, conflict known by then -> CodeQL only, no pull_request dispatch 3f31d103 pushed 19:38:29Z, became conflicting 19:44:07Z -> full dispatch, then the conflict arrived after So: push while KNOWN-conflicting and pull_request does not dispatch. Become conflicting AFTERWARDS and every check stays attached to the sha, still green, now describing a tree that no longer exists — nothing re-runs, because nothing was pushed. The second is the worse one and the entry did not have it. Mechanism 1 at least leaves a suspiciously thin check list. Mechanism 2 leaves a complete, genuinely-passing 11-check rollup on a PR that cannot be merged and whose tests never ran against the tree merging would produce. No artifact anywhere records that. It is also why mechanism 1 reads as false to anyone measuring after the fact — which is exactly what happened here. Row 1 is worth keeping for the window it exposes: a push 142 seconds after the merge that broke it still got a full dispatch, because mergeability had not been recomputed yet. Which side of that recomputation your push lands on decides which mechanism you get, and nothing in the UI marks the boundary. Title and rule updated: a COMPLETE green rollup on a conflicting PR is the worse signal, not the reassuring one. Co-Authored-By: Claude Opus 5 --- docs/development/agent-experience-audit.md | 50 ++++++++++++++++++---- 1 file changed, 41 insertions(+), 9 deletions(-) diff --git a/docs/development/agent-experience-audit.md b/docs/development/agent-experience-audit.md index 6f3e1d998..ac6f2cbe2 100644 --- a/docs/development/agent-experience-audit.md +++ b/docs/development/agent-experience-audit.md @@ -2241,7 +2241,7 @@ as "older response shape," not "your message reached nobody." deploy that had nothing to do with it — proximity to a deploy makes every bug look like a regression. -## 41. A conflicting PR loses its whole test suite, and renders as green (2026-08-22, pod-architect) +## 41. A conflicting PR's checks describe a tree that will never exist (2026-08-22, pod-architect + sprint-review) > Numbering assumes #1122 (entry 39) and #1132 (entry 40) land first. If they > merge in a different order, renumber this one rather than them. @@ -2260,10 +2260,40 @@ green rollup **for the parent commit**, three commits behind the branch. `#1120` and `#1128` were `CONFLICTING/DIRTY`: `main` had taken the squashed 2/4 while their branches still carried the unsquashed originals. GitHub builds -`pull_request` events against a *merge ref*. A conflicting PR has no merge ref, -so `tests.yml` never dispatched at all. What remained was CodeQL — which -triggers on `push`, needs no merge ref, and passed. **Four checks, four green -ticks, zero tests.** +`pull_request` events against a *merge ref*, and a PR that is conflicting **at +the moment of the event** has none — so `tests.yml` did not dispatch. What +remained was CodeQL, which triggers on `push`, needs no merge ref, and passed. +**Four checks, four green ticks, zero tests.** + +Measured, because the first draft of this entry said "a conflicting PR never +dispatches tests" and @sprint-review falsified it by finding eight green runs +— Tier 1 among them — on `3f31d103`, a conflicting head. Their measurement was +right and the sentence was wrong. Timestamps settle it: + +| Head | Pushed | Conflict began | `pull_request` dispatched? | +|---|---|---|---| +| `9366e11e` (#1120) | 18:53:21Z | 18:50:59Z, not yet computed | yes — 11 checks | +| `4942ad3d` (#1120) | 18:57:12Z | known by then | **no** — CodeQL only | +| `3f31d103` (#1136) | 19:38:29Z | 19:44:07Z, six minutes later | yes — 11 checks | + +**So there are two mechanisms, not one, and the second is the worse of them.** + +1. *Push while the PR is known-conflicting* → no `pull_request` dispatch. The + checks that appear are the push-triggered ones. This is what hit `4942ad3d`. +2. *Become conflicting after the checks ran* → every check stays attached to + the sha, still green, now describing a state that no longer exists. Nothing + re-runs, because nothing was pushed. This is what hit `3f31d103`, and it is + what makes the first mechanism look false to anyone who measures afterwards. + +The second is worse because the first at least leaves a thin check list as a +hint. The second leaves a **complete, genuinely-passing 11-check rollup** on a +PR that can no longer be merged and whose tests have never run against the +tree it would produce. There is no artifact anywhere that says so. + +Note the window in row 1: `9366e11e` was pushed 142 seconds after the merge +that broke it and still got a full dispatch, because GitHub had not recomputed +mergeability yet. Whether your push lands before or after that recomputation +decides which mechanism you get, and nothing in the UI marks the boundary. **Why the instrument can't tell you.** @@ -2286,10 +2316,12 @@ workflow's clock instead of the pod's, the parent commit instead of the head. `headRefOid` can lag its own ref, and on a merged PR it stops updating entirely. If the count of check runs is zero, that is the loudest possible signal, and it prints as silence. -- **Check `mergeable` before reading `statusCheckRollup`.** `CONFLICTING` - means the rollup is describing a smaller set of checks than you think, so - the rollup is not evidence about the suite. Read the two together or - neither. +- **Check `mergeable` before reading `statusCheckRollup`.** On `CONFLICTING` + the rollup is either a smaller set than you think (mechanism 1) or a full + set measured against a tree that no longer exists (mechanism 2). Neither is + evidence about what merging would do. Read the two together or neither — + and note that a *complete* green rollup on a conflicting PR is the worse + signal, not the reassuring one. - **A squash merge orphans anything pushed to that branch afterwards.** The window is as long as your commit takes. Before pushing to a branch you believe is open, confirm it: a closed PR accepts the push, moves the ref, From 4cf588d4876b88b377f09734e9585c137536c02c Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Sat, 22 Aug 2026 13:29:17 -0700 Subject: [PATCH 03/12] docs(ax): entry 41's own table carried a wrong count MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit @sprint-review (57336): row 3 said "11 checks" for 3f31d103. Measured, it is 8 runs across 5 unique names. Re-derived rather than taken: 9366e11e total=11 unique=11 4942ad3d total=4 unique=4 3f31d103 total=8 unique=5 They are right, and the sting is that 8-across-5 is a number I had counted correctly myself an hour earlier, in this pod, before writing 11 into the table. The entry is about instruments whose output does not mean what it appears to; the table was mine and it did the same thing. Run count and name count disagree by three because sprint-impl's manual re-dispatch landed on the same sha as the automatic run, so the split is now stated explicitly: 5 pull_request + 3 workflow_dispatch. The mechanism claim is unaffected — pull_request DID dispatch at 19:38, while the PR was still mergeable — but "11 checks" was doing rhetorical work the evidence did not support. Also dropped "11-check rollup" from the mechanism-2 paragraph for the same reason. Added what I did NOT measure, because the difference is more interesting than the count: the two PRs' pull_request sets differ in MEMBERSHIP, 11 names against 5, not merely in size. #1120 targets main and #1136 targeted a feature branch, and several workflows filter on base — but that is a hypothesis about the difference and it is labelled as one rather than slipped in as a finding. Co-Authored-By: Claude Opus 5 --- docs/development/agent-experience-audit.md | 23 ++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/docs/development/agent-experience-audit.md b/docs/development/agent-experience-audit.md index ac6f2cbe2..c5832083a 100644 --- a/docs/development/agent-experience-audit.md +++ b/docs/development/agent-experience-audit.md @@ -2270,11 +2270,22 @@ dispatches tests" and @sprint-review falsified it by finding eight green runs — Tier 1 among them — on `3f31d103`, a conflicting head. Their measurement was right and the sentence was wrong. Timestamps settle it: -| Head | Pushed | Conflict began | `pull_request` dispatched? | -|---|---|---|---| -| `9366e11e` (#1120) | 18:53:21Z | 18:50:59Z, not yet computed | yes — 11 checks | -| `4942ad3d` (#1120) | 18:57:12Z | known by then | **no** — CodeQL only | -| `3f31d103` (#1136) | 19:38:29Z | 19:44:07Z, six minutes later | yes — 11 checks | +| Head | Pushed | Conflict began | `pull_request` dispatched? | Runs (unique names) | +|---|---|---|---|---| +| `9366e11e` (#1120) | 18:53:21Z | 18:50:59Z, not yet computed | yes | 11 (11) | +| `4942ad3d` (#1120) | 18:57:12Z | known by then | **no** | 4 (4) — CodeQL family only | +| `3f31d103` (#1136) | 19:38:29Z | 19:44:07Z, six minutes later | yes | 8 (5) — 5 `pull_request` + 3 `workflow_dispatch` | + +The last row said "11 checks" in the first version of this entry. @sprint-review +measured it at 8 runs across 5 names and was right — and it was a number I had +myself counted correctly earlier the same hour before contradicting it here. +The duplication is sprint-impl's manual re-dispatch landing on the same sha as +the automatic run, so a name-count and a run-count disagree by three. + +NOT chased, and so not claimed: the two PRs' `pull_request` sets differ in +membership (11 names vs 5), not just in size. #1120 targets `main` and #1136 +targeted a feature branch, and several workflows filter on base — but that is +a hypothesis about the difference, not a measurement of it. **So there are two mechanisms, not one, and the second is the worse of them.** @@ -2286,7 +2297,7 @@ right and the sentence was wrong. Timestamps settle it: what makes the first mechanism look false to anyone who measures afterwards. The second is worse because the first at least leaves a thin check list as a -hint. The second leaves a **complete, genuinely-passing 11-check rollup** on a +hint. The second leaves a **complete, genuinely-passing rollup** on a PR that can no longer be merged and whose tests have never run against the tree it would produce. There is no artifact anywhere that says so. From f7eed084629023fe39a069aa13535a92c0a9dce8 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Sat, 22 Aug 2026 13:34:04 -0700 Subject: [PATCH 04/12] docs(ax): the check-set denominator is a property of the base, not the repo MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit @sprint-review (57341) settled the membership question I had labelled as an unchased hypothesis. Re-derived rather than accepted: the 5 names on 3f31d103 are a STRICT subset of the 11 on 9366e11e, with exactly six extras — CodeQL plus its three Analyze jobs, the version-bump guard, and the stale-base guard. Two of those are merge-to-main guards and are correctly base-scoped: package-version-guard.yml and pr-base-freshness.yml both declare `pull_request: branches: [ main ]`, so a PR onto a feature branch is outside their remit by design. The other four are CodeQL's, and there is no codeql.yml in this repo at all — GitHub default setup, scoped somewhere our workflow files cannot show you. The consequence is bigger than the correction that prompted it, and it is now the paragraph's point: a PR's "full" check set is a property of its BASE. Eleven is complete onto main; five is complete onto a feature branch. There is no fixed denominator, so a check COUNT means nothing on its own — and a stacked PR retargeted to main at merge time will be judged by guards that never ran against it. That is the same error I made one paragraph earlier when I wrote "11 checks" for a row that had 8, inside an entry about instruments that report numbers meaning something other than they appear to. Co-Authored-By: Claude Opus 5 --- docs/development/agent-experience-audit.md | 28 ++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-) diff --git a/docs/development/agent-experience-audit.md b/docs/development/agent-experience-audit.md index c5832083a..da1e753ec 100644 --- a/docs/development/agent-experience-audit.md +++ b/docs/development/agent-experience-audit.md @@ -2282,10 +2282,30 @@ myself counted correctly earlier the same hour before contradicting it here. The duplication is sprint-impl's manual re-dispatch landing on the same sha as the automatic run, so a name-count and a run-count disagree by three. -NOT chased, and so not claimed: the two PRs' `pull_request` sets differ in -membership (11 names vs 5), not just in size. #1120 targets `main` and #1136 -targeted a feature branch, and several workflows filter on base — but that is -a hypothesis about the difference, not a measurement of it. +The two PRs' `pull_request` sets also differ in **membership**, 11 names +against 5, not merely in size. I left that labelled as an unchased hypothesis; +@sprint-review measured it, and re-deriving it confirms the 5 are a strict +subset of the 11 with exactly six extras: + +``` +CodeQL · Analyze (actions) · Analyze (javascript-typescript) · Analyze (python) +Source changed ⇒ version bumped · Stale-base merge guard +``` + +Two are merge-to-main guards and are correctly base-scoped — both +`package-version-guard.yml` and `pr-base-freshness.yml` declare +`pull_request: branches: [ main ]`, so a PR onto a feature branch is outside +their remit by design. The other four are CodeQL's, and there is no +`codeql.yml` in the repo: that is GitHub default setup, scoped outside our +workflow files entirely. + +**Which means a PR's "full" check set is a property of its BASE, not of the +repo.** There is no fixed number to compare against. Eleven is complete for a +PR onto `main`; five is complete for a PR onto a feature branch; and a stacked +PR that gets retargeted to `main` at merge time will be judged by guards that +never ran on it. Counting checks tells you nothing unless you know what the +denominator should have been — which was the mistake one paragraph up, made +in an entry about exactly this. **So there are two mechanisms, not one, and the second is the worse of them.** From 60e9e9adc6350acb7a41e6c8db7c736afbd49b88 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Sat, 22 Aug 2026 13:42:04 -0700 Subject: [PATCH 05/12] docs(ax): a base retarget fires nothing, and that is the worst of the three MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit @Sam (56969) named this mechanism in advance, from the 2026-08-04 incident, and it was missing from the entry. Confirmed against what actually happened: #1106 merged at 15:15:08Z, GitHub retargeted #1109 from the parent branch to main, and the next workflow run on that branch was 15:53:20Z — 38 minutes later, triggered by a push. Zero runs at 15:15. So mechanism 3: the base changes, no event fires, and every existing check silently becomes a check against a base the PR no longer has. It is the worst of the three because nothing about it looks wrong. No conflict, no thin list, no red. The PR is green and mergeable, and it now MEANS something different from what was tested — it merges into main rather than into its parent — while every check on it was computed against the old base. A green rollup is exactly what you would expect to see, and exactly what you get. Two rules added. Sam's press plan is the working mitigation and is recorded as such: merge the parent, verify the child's baseRefName flipped and the PR did not auto-close, then force a head event so CI runs against main. The second is a way to check it after the fact, which the entry needed and did not have. The base-scoped guards are the tell — Package Version Guard and PR Base Freshness both declare `pull_request: branches: [ main ]`, so their PRESENCE certifies a run happened with main as base. That is how #1109 is confirmed to have satisfied the rule before merging: both guards appear in its 18:30 run. Reading the check NAMES answers a question the check COUNT cannot — which is the same lesson as the denominator paragraph above, arrived at from the opposite direction. Co-Authored-By: Claude Opus 5 --- docs/development/agent-experience-audit.md | 26 ++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/docs/development/agent-experience-audit.md b/docs/development/agent-experience-audit.md index da1e753ec..efb6f99bd 100644 --- a/docs/development/agent-experience-audit.md +++ b/docs/development/agent-experience-audit.md @@ -2315,6 +2315,19 @@ in an entry about exactly this. the sha, still green, now describing a state that no longer exists. Nothing re-runs, because nothing was pushed. This is what hit `3f31d103`, and it is what makes the first mechanism look false to anyone who measures afterwards. +3. *The base auto-retargets when the parent merges* → **no event fires at all.** + Named in advance by Sam (56969) from an 2026-08-04 incident, and confirmed + here: `#1106` merged at 15:15:08Z, GitHub retargeted `#1109` from the parent + branch to `main`, and the next workflow run on that branch was at 15:53:20Z + — 38 minutes later, triggered by a push, not by the retarget. Zero runs at + 15:15. + +Mechanism 3 is the worst of the three for a stacked PR, because nothing about +it looks wrong. There is no conflict, no thin check list, no red. The PR is +green and mergeable — and it now means something different from what was +tested, since it merges into `main` rather than into its parent branch, and +every check on it was computed against the old base. A green rollup is exactly +what you would expect to see, and exactly what you get. The second is worse because the first at least leaves a thin check list as a hint. The second leaves a **complete, genuinely-passing rollup** on a @@ -2363,3 +2376,16 @@ workflow's clock instead of the pod's, the parent commit instead of the head. quiet rather than red. Rebase children onto `main` immediately after a parent lands; do not wait for a review to surface it, because the review will be looking at the same green ticks. +- **A retargeted PR needs a new head event before its checks mean anything.** + Sam's press plan (56969) is the working form: merge the parent, verify the + child's `baseRefName` actually flipped and the PR did not auto-close, then + force a head event (`gh pr update-branch`) so CI runs against `main`. No + stacked PR merges without its own green run *at the new base*. +- **You can tell which base a run was against, from the run list.** The + base-scoped guards are the tell: `Package Version Guard` and + `PR Base Freshness` both declare `pull_request: branches: [ main ]`, so their + PRESENCE certifies the run happened with `main` as base and their absence + says it did not. That is how `#1109` was confirmed to have satisfied the rule + before it merged — both guards appear in its 18:30 run, three hours after the + retarget. Reading the check *names* answers a question the check *count* + cannot. From 2036382e13c4218f1275152b86f049e273449b98 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Sat, 22 Aug 2026 13:52:16 -0700 Subject: [PATCH 06/12] =?UTF-8?q?docs(ax):=20base=20is=20necessary=20and?= =?UTF-8?q?=20not=20sufficient=20=E2=80=94=20paths=20move=20the=20set=20to?= =?UTF-8?q?o?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found by applying this entry's own denominator paragraph to the PR carrying it. #1135 is docs-only, targets main, and reports 10 checks where #1120 reports 11. The missing one is E2E Tests: playwright.yml filters on frontend/**, backend/**, e2e/** and playwright.config.*, and a docs/** diff matches none, so the workflow never dispatches and the check never exists. I had written that the check set is a property of the BASE. That is true and incomplete — it is a function of (base, paths touched), and I only found the second half because the claim was sitting on a PR that contradicted it. The visible consequence is worth stating because it will be read wrong: a docs-only PR settles at MERGEABLE/UNSTABLE rather than CLEAN, permanently. No future event produces the absent check. UNSTABLE here means "a check you might expect is not present", not "something failed" — all ten that ran are green. A "merge only when CLEAN" rule waits forever on documentation, and anyone reading UNSTABLE as breakage goes hunting a failure that does not exist. Three of this entry's corrections have now come from treating a check count as comparable across PRs that were never comparable. Co-Authored-By: Claude Opus 5 --- docs/development/agent-experience-audit.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/development/agent-experience-audit.md b/docs/development/agent-experience-audit.md index efb6f99bd..4a6f1f76a 100644 --- a/docs/development/agent-experience-audit.md +++ b/docs/development/agent-experience-audit.md @@ -2307,6 +2307,25 @@ never ran on it. Counting checks tells you nothing unless you know what the denominator should have been — which was the mistake one paragraph up, made in an entry about exactly this. +**And base is necessary, not sufficient — the CHANGED PATHS move it too.** +Found by applying the paragraph above to this very PR. `#1135` is docs-only, +targets `main`, and reports 10 checks against `#1120`'s 11; the missing one is +`E2E Tests`, because `playwright.yml` filters on +`frontend/** · backend/** · e2e/** · playwright.config.*` and a `docs/**` diff +matches none of them. The workflow never dispatches, so the check never exists. + +The visible consequence is that a docs-only PR settles at `MERGEABLE/UNSTABLE` +rather than `MERGEABLE/CLEAN`, permanently — there is no future event that will +produce the absent check. `UNSTABLE` here means "a check you might expect is +not present", not "something failed": all ten that ran are green. Anyone +following a "merge only when CLEAN" rule will wait forever on documentation, +and anyone reading `UNSTABLE` as "something broke" will go looking for a +failure that does not exist. + +So the denominator is a function of *(base, paths touched)*. Three of this +entry's corrections have now come from treating a check count as comparable +across PRs that were never comparable. + **So there are two mechanisms, not one, and the second is the worse of them.** 1. *Push while the PR is known-conflicting* → no `pull_request` dispatch. The From cea3c26c38583c195ed8a8f375d2ea1e664e5256 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Sat, 22 Aug 2026 14:01:32 -0700 Subject: [PATCH 07/12] docs(ax): I read a pending check as a missing one and built a theory on it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Retracting a consequence I committed to this entry an hour ago and posted to the pod as actionable: that a docs-only PR settles at MERGEABLE/UNSTABLE permanently, because the paths-filtered E2E check never dispatches, and that a merge-only-when-CLEAN rule therefore deadlocks on documentation. False. Waited for the runs to finish instead of reading a snapshot: #1142 MERGEABLE/CLEAN checks=10 E2E absent #1143 MERGEABLE/CLEAN checks=10 E2E absent An absent E2E Tests does not prevent CLEAN — it is not required, so its non-existence costs nothing. The UNSTABLE I saw on #1135 was a check still PENDING, not a check MISSING, and it cleared by itself. The membership finding survives and is independently verified from playwright.yml and from E2E=0 on all three PRs. Only the consequence I hung off it was fabricated. Keeping the retraction in the entry rather than deleting the paragraph, because the failure mode is the entry's own subject. I read a transient as structural and then built a mechanism to explain it — paths filter, no dispatch, no check, never CLEAN. Every step of that chain is true in isolation. What made it convincing enough to commit and broadcast was the coherence, not the evidence, and none of it was ever connected to the observation it claimed to explain. Co-Authored-By: Claude Opus 5 --- docs/development/agent-experience-audit.md | 33 +++++++++++++++++----- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/docs/development/agent-experience-audit.md b/docs/development/agent-experience-audit.md index 4a6f1f76a..1a44697af 100644 --- a/docs/development/agent-experience-audit.md +++ b/docs/development/agent-experience-audit.md @@ -2314,13 +2314,32 @@ targets `main`, and reports 10 checks against `#1120`'s 11; the missing one is `frontend/** · backend/** · e2e/** · playwright.config.*` and a `docs/**` diff matches none of them. The workflow never dispatches, so the check never exists. -The visible consequence is that a docs-only PR settles at `MERGEABLE/UNSTABLE` -rather than `MERGEABLE/CLEAN`, permanently — there is no future event that will -produce the absent check. `UNSTABLE` here means "a check you might expect is -not present", not "something failed": all ten that ran are green. Anyone -following a "merge only when CLEAN" rule will wait forever on documentation, -and anyone reading `UNSTABLE` as "something broke" will go looking for a -failure that does not exist. +**I then attached a consequence to that finding which was false, and it is +worth keeping the wreckage visible.** I wrote that a docs-only PR therefore +settles at `MERGEABLE/UNSTABLE` permanently, that no future event produces the +absent check, and that a "merge only when CLEAN" rule would deadlock on +documentation. I posted that to the pod as something to act on. + +It is wrong. Waiting for the runs to finish and re-reading: + +``` +#1142 MERGEABLE/CLEAN checks=10 E2E absent +#1143 MERGEABLE/CLEAN checks=10 E2E absent +``` + +An absent `E2E Tests` does not prevent `CLEAN` — it is not a required check, +so its non-existence costs nothing. The `UNSTABLE` I had seen on `#1135` was a +check still **pending**, not a check **missing**, and it resolved on its own. + +So the mistake was reading a transient state as a structural one, and then +inventing a mechanism to explain it. The invented mechanism was internally +coherent — paths filter, no dispatch, no check, never CLEAN — which is exactly +what made it convincing enough to commit and to broadcast. Every step was true +except the one connecting them to the observation. + +The membership finding above survives intact and was independently verified +from `playwright.yml` and from `E2E=0` on all three PRs. Only the consequence +was fabricated. So the denominator is a function of *(base, paths touched)*. Three of this entry's corrections have now come from treating a check count as comparable From 5a9c11cda6a85c4b37f8b7d04c3ba45ae2b5912a Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Sat, 22 Aug 2026 15:31:26 -0700 Subject: [PATCH 08/12] docs(ax): why the base flip fires nothing, not just that it fired nothing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit @sprint-review (57010) supplied the cause behind the measurement. Re-derived rather than accepted, and every part holds: release-safety.yml [opened, synchronize, reopened, ready_for_review] package-version-guard.yml [opened, synchronize, reopened, ready_for_review] pr-base-freshness.yml [opened, synchronize, reopened] Four others on pull_request — tests, playwright, secret-scan, mintlify — take the default set, which is that list minus ready_for_review. `grep -rn edited .github/workflows/` returns nothing at all. So the retarget does fire an event. It fires one nothing is subscribed to, which is exactly why update-branch works where a base flip does not: it pushes a head commit and produces synchronize. Kept the two halves separate because their evidence differs. Zero runs at the retarget, and no subscriber to `edited`, are both measured here. That GitHub emits `edited` specifically on a base change is documentation, not an event payload I captured — consistent with the observation rather than demonstrated by it. That distinction is rule 16's, applied to my own addendum. Also recorded the fix that was considered and declined: adding `edited` to those types lists would make retargets re-run CI by themselves, and would also re-run CI on every title and body edit, which is a bad trade here. The mitigation stays a deliberate head event. Writing down the rejected option so the next reader does not re-derive it and reach the other conclusion. Co-Authored-By: Claude Opus 5 --- docs/development/agent-experience-audit.md | 32 ++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/docs/development/agent-experience-audit.md b/docs/development/agent-experience-audit.md index 1a44697af..83b34d1cb 100644 --- a/docs/development/agent-experience-audit.md +++ b/docs/development/agent-experience-audit.md @@ -2360,6 +2360,38 @@ across PRs that were never comparable. — 38 minutes later, triggered by a push, not by the retarget. Zero runs at 15:15. +**Why nothing fires, not just that nothing fired.** @sprint-review (57010) +supplied the cause behind the measurement, and re-deriving it confirms every +part. A base change emits `pull_request.edited`, and **no workflow in this repo +subscribes to `edited`** — `grep -rn 'edited' .github/workflows/` returns +nothing. Three declare their types explicitly: + +``` +release-safety.yml [opened, synchronize, reopened, ready_for_review] +package-version-guard.yml [opened, synchronize, reopened, ready_for_review] +pr-base-freshness.yml [opened, synchronize, reopened] +``` + +and the four others on `pull_request` — `tests.yml`, `playwright.yml`, +`secret-scan.yml`, `mintlify.yml` — take GitHub's default set, which is the +same list minus `ready_for_review`. `edited` is in none of them. So the +retarget does fire an event; it fires one that nothing is listening for, which +is why `update-branch` works and a base flip does not: the former pushes a head +commit and produces `synchronize`. + +Stated precisely, because the two halves have different evidence: *zero runs at +the retarget* and *no subscriber to `edited`* are both measured here. That +GitHub emits `edited` specifically on a base change is from its documentation, +not from an event payload I captured — consistent with the observation rather +than demonstrated by it. + +**The root fix exists and is probably not worth taking.** Adding `edited` to +those `types` lists would make a retarget re-run CI on its own. It would also +re-run CI on every title and body edit, since `edited` covers those too. That +is a bad trade on a busy repo, so the mitigation stays where Sam put it: force +a head event deliberately. Worth writing down that the alternative was +considered and declined, or the next reader re-derives it. + Mechanism 3 is the worst of the three for a stacked PR, because nothing about it looks wrong. There is no conflict, no thin check list, no red. The PR is green and mergeable — and it now means something different from what was From eece804300c385dbaf8cd086a9ab482ade6c1084 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Sat, 22 Aug 2026 15:33:02 -0700 Subject: [PATCH 09/12] docs(ax): a push cadence under the suite's runtime never lands on the tip MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit @sprint-review (57014): 3/4 took four heads in twelve minutes while tests.yml runs 5-6, so no run could ever cover the tip. Folded into the existing verify-by-sha rule rather than added as a fourth mechanism, because it is not one — it is the same "checks describe a superseded state", self-inflicted. What the rule did not carry is the difference between detecting it and escaping it: comparing sha-to-tip finds the mismatch every time and suggests re-dispatching, which reproduces it. Only noticing the RATE tells you the fix is to stop pushing. Co-Authored-By: Claude Opus 5 --- docs/development/agent-experience-audit.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/development/agent-experience-audit.md b/docs/development/agent-experience-audit.md index 83b34d1cb..40cc7524e 100644 --- a/docs/development/agent-experience-audit.md +++ b/docs/development/agent-experience-audit.md @@ -2430,6 +2430,14 @@ workflow's clock instead of the pod's, the parent commit instead of the head. `headRefOid` can lag its own ref, and on a merged PR it stops updating entirely. If the count of check runs is zero, that is the loudest possible signal, and it prints as silence. + + **And if that comparison keeps failing, stop pushing rather than + re-dispatching.** @sprint-review (57014) caught this branch taking four heads + in twelve minutes against a `tests.yml` that runs 5–6 — a cadence under the + suite's runtime means no run can ever land on the tip, and every fix looks + like one more dispatch away. The rule above detects the mismatch; only + noticing the *rate* tells you why it will keep recurring. Green-on-head needs + a quiet period. - **Check `mergeable` before reading `statusCheckRollup`.** On `CONFLICTING` the rollup is either a smaller set than you think (mechanism 1) or a full set measured against a tree that no longer exists (mechanism 2). Neither is From ef626a5e5f273df88735ca6981ae3ba7a9804c57 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Sat, 22 Aug 2026 17:21:24 -0700 Subject: [PATCH 10/12] docs(ax): a workflow fix reaches a stacked PR through its BASE, not its head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fourth mechanism, found by chasing @sprint-review's #1132 measurement (57463): its entire check list is one skipped Release Branch Guard, and it reads CLEAN because nothing is failing because nothing ran. They measured the counts and declined to claim the cause. This is the cause. For a pull_request event GitHub reads the workflow definition from the MERGE REF — base + head — so the base branch's copy of the file decides whether the event matches at all. #1123 dropped `branches: [main]` from tests.yml on main at 15:20:27Z. #1132's head was pushed at 18:11:30Z, three hours later, and still got nothing, because its base `docs/ax-two-call-sites` was last touched at 06:52 and still carries the old filter. Read the file on that branch to confirm rather than inferring it. The general form is worth more than the instance: a workflow fix on main reaches a stacked PR only when that PR's BASE absorbs it. Not the head. So "we fixed CI for stacked PRs" is true of the repo and false of every PR already stacked on a stale branch, and nothing in the UI separates the two. Recorded both wrong guesses I made first, because the sequence is the point. I proposed "the head predated the fix" — refuted by timestamps, it postdates it by three hours — then "a paths filter" — refuted by reading tests.yml, which has none. Rule 16 killed both: make the mechanism predict something and go check. Co-Authored-By: Claude Opus 5 --- docs/development/agent-experience-audit.md | 24 ++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/development/agent-experience-audit.md b/docs/development/agent-experience-audit.md index 40cc7524e..3685d8b08 100644 --- a/docs/development/agent-experience-audit.md +++ b/docs/development/agent-experience-audit.md @@ -2392,6 +2392,30 @@ is a bad trade on a busy repo, so the mitigation stays where Sam put it: force a head event deliberately. Worth writing down that the alternative was considered and declined, or the next reader re-derives it. +4. *The base branch predates the workflow fix* → the fix never applies. For a + `pull_request` event GitHub reads the workflow definition from the **merge + ref**, which is base + head — so the BASE branch's copy of the file decides + whether the event matches. `#1123` dropped `branches: [main]` from + `tests.yml` on main at 15:20:27Z, and `#1132` still got zero test runs from + a head pushed at 18:11:30Z, three hours later, because its base + (`docs/ax-two-call-sites`, last touched 06:52) still carries the old + filter. Verified by reading `tests.yml` on that branch. + + Its whole check list is one skipped `Release Branch Guard`, and its + `mergeStateStatus` is `CLEAN` — nothing failing, because nothing ran. + @sprint-review measured that and explicitly declined to claim the cause; + this is the cause. + + **A workflow fix on `main` reaches a stacked PR only when that PR's BASE + absorbs it.** Not the head — the base. So "we fixed CI for stacked PRs" is + true of the repo and false of every PR already stacked on a stale branch, + and there is no signal distinguishing the two. + + I got this wrong twice before reading the file. First guess: the head + predated the fix (refuted by timestamps — it postdates it by three hours). + Second: a paths filter (refuted — `tests.yml` has none). Rule 16's "make + the mechanism predict something" is what killed both. + Mechanism 3 is the worst of the three for a stacked PR, because nothing about it looks wrong. There is no conflict, no thin check list, no red. The PR is green and mergeable — and it now means something different from what was From 4e90d3e0c81c24fb750cfb28a4e59ae6ca240b8e Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Sat, 22 Aug 2026 17:37:51 -0700 Subject: [PATCH 11/12] docs(ax): mechanism 4 is sprint-review's finding, not mine MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit They established it at 15:25:50Z — "for pull_request, the trigger is read from the PR's own merge ref" — measured on #1120, whose head was pushed 15:22:21Z, two minutes after #1123 merged, and got zero runs because its base still carried the old filter. They also corrected their own earlier "merge #1123 first and the ordering stops mattering" in the same message. I derived the same thing nine hours later from #1132 and wrote it up as "@sprint-review measured the counts and explicitly declined to claim the cause; this is the cause." Both halves wrong: they did claim it, and they were first. Their message was unread in my redelivery queue while I re-derived it from a different PR. Corrected in place rather than quietly reworded, because the entry is about evidence and a provenance error inside it is the same defect. Two instances, two PRs, one mechanism — theirs is the finding, mine is the confirmation, and the independent second instance is worth keeping for exactly that reason. Co-Authored-By: Claude Opus 5 --- docs/development/agent-experience-audit.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/docs/development/agent-experience-audit.md b/docs/development/agent-experience-audit.md index 3685d8b08..9adbf401c 100644 --- a/docs/development/agent-experience-audit.md +++ b/docs/development/agent-experience-audit.md @@ -2403,8 +2403,20 @@ considered and declined, or the next reader re-derives it. Its whole check list is one skipped `Release Branch Guard`, and its `mergeStateStatus` is `CLEAN` — nothing failing, because nothing ran. - @sprint-review measured that and explicitly declined to claim the cause; - this is the cause. + + **ATTRIBUTION, corrected.** @sprint-review established this mechanism at + 15:25:50Z — including "the trigger is read from the PR's own merge ref" — + measured on `#1120`, whose head was pushed 15:22:21Z, two minutes after + `#1123` merged, and produced zero runs for the same reason. They also + corrected their own earlier "merge #1123 first and the ordering stops + mattering" in the same message. + + I derived it independently nine hours later from `#1132` and posted it as + "they measured the counts and declined to claim the cause; this is the + cause." That was wrong. They claimed it, first, and correctly; their + message was sitting unread in my redelivery queue while I re-derived it. + Two instances, two PRs, one mechanism — theirs is the finding and mine is + the confirmation. **A workflow fix on `main` reaches a stacked PR only when that PR's BASE absorbs it.** Not the head — the base. So "we fixed CI for stacked PRs" is From 2a32ae1131bbd3f07a63fa0492d2727560806d13 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Sat, 22 Aug 2026 17:51:33 -0700 Subject: [PATCH 12/12] docs(ax): a hand-dispatch is the artifact that hides the bug it works around MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit @sprint-review (57058) nearly retracted a correct finding. They measured zero runs on #1120's 0147fa24, then one run six minutes later — which reads as "my earlier claim was wrong, #1123 did reach it." The second run was my hand-dispatch landing between their two checks. The `event` field is the only discriminator. `workflow_dispatch` proves someone pushed a button; `pull_request` proves the trigger matched. Counting runs cannot tell those apart, and the count is what a reader reaches for. The general form is why it belongs in the entry rather than in the pod: a manual dispatch is the STANDARD response to noticing a PR has no checks. So the act of working around the bug manufactures the exact artifact that makes the bug look absent — and the person most likely to measure afterwards is the one who dispatched, comparing against their own memory of "it had nothing before." I am that person here. Every hand-dispatch I ran on this stack today is noise in anyone else's measurement of the same shas, which is a cost of the workaround I had not counted. The table earlier in this entry already splits its counts by event; this adds the rule that made the split necessary. Co-Authored-By: Claude Opus 5 --- docs/development/agent-experience-audit.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/development/agent-experience-audit.md b/docs/development/agent-experience-audit.md index 9adbf401c..4d304fb55 100644 --- a/docs/development/agent-experience-audit.md +++ b/docs/development/agent-experience-audit.md @@ -2418,6 +2418,21 @@ considered and declined, or the next reader re-derives it. Two instances, two PRs, one mechanism — theirs is the finding and mine is the confirmation. + **Read `event`, not the count, when asking whether a fix reached a PR.** + @sprint-review (57058) measured zero runs on `#1120`'s `0147fa24`, then one + run six minutes later — a flip that reads exactly like "my earlier claim + was wrong, #1123 did reach it after all." It was my hand-dispatch landing + between their two checks. The discriminator is the `event` field: + `workflow_dispatch` proves someone pushed a button, `pull_request` proves + the trigger matched. Only the second is evidence about the fix. + + This is worth more than the instance because a manual dispatch is the + standard response to noticing a PR has no checks — so the act of working + around the bug produces exactly the artifact that makes the bug look + absent, and the person most likely to measure afterwards is the one who + dispatched. Both counts in the table above are split by event for this + reason. + **A workflow fix on `main` reaches a stacked PR only when that PR's BASE absorbs it.** Not the head — the base. So "we fixed CI for stacked PRs" is true of the repo and false of every PR already stacked on a stale branch,