From 6f5688532e979a1f6d983b4d7a30e748f9370f1b Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Sat, 22 Aug 2026 13:35:12 -0700 Subject: [PATCH 1/5] =?UTF-8?q?docs(ax):=20entry=2042=20=E2=80=94=20a=20co?= =?UTF-8?q?unt=20in=20the=20verdict=20slot?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit @sprint-review (57339) separated a variant I had folded into entry 41, and the split is right. Entries 34, 35 and 41 are all one shape: an absence read as success. No check runs means no failures. What happened while probing the derivation suite is not that. A mutation left the file unparseable and jest printed "Tests: 0 total" — a COUNT occupying the verdict slot. That line and "Tests: 5 passed" appear in the same position, in the same format, and mean opposite things about whether anything was measured. The absence variant at least shows you a blank where evidence belongs, and a blank can prompt a second look. This one hands you a number, formatted like a result, in the place you learned to read results from. Scanned for "did anything fail", "0 total" passes — true, and useless. Generalised past jest, because the slot collision is everywhere: grep -c returning 0 for no-matches or a bad path; a migration reporting 0 rows for already-applied or a wrong predicate; a sweep reporting 0 offenders for clean or for never reaching the directory (which was the real #1140 bug); git checkout -- exiting 0 on an untracked file. Each pair shares one output and splits on whether the instrument ran. Both probe failures that produced this came from writing a probe about probes that fail silently — the other being a perl mutation that died on an unescaped modifier, applied nothing, and reported 7/7. The rule was being violated in the act of being written down, which is the argument for enforcing it mechanically rather than remembering it. Third rule is the one aimed at me rather than the tooling: reporting a suite as green when it emitted "0 total" is not a rounding error, it is the same defect committed by the person reading the instrument. Co-Authored-By: Claude Opus 5 --- docs/development/agent-experience-audit.md | 66 ++++++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/docs/development/agent-experience-audit.md b/docs/development/agent-experience-audit.md index c12c72a19..9c574d587 100644 --- a/docs/development/agent-experience-audit.md +++ b/docs/development/agent-experience-audit.md @@ -2573,3 +2573,69 @@ agent path since it was written. - Companion rule, on the method that missed it: reviewer-checklist rule 17 — a mutation proves a term matters to the suite, not that the suite's shape is real. +## 43. A count in the verdict slot (2026-08-22, sprint-review + pod-architect) + +> Renumbered 42 → 43 on rebase: #1164's dual-auth entry took 42 on main while +> this sat open. Its author had pushed a renumber to 44 fourteen minutes after +> that PR squash-merged, so the correction never landed — which is why a +> reservation note is not a reservation. #1122 (39) and #1132 (40) are still +> open; if they merge in another order, renumber this one rather than them. + +Entries 34, 35 and 41 are all the same shape: an **absence read as success**. +No check runs means no failures; a workflow that never dispatched leaves no +red X; a registry that was never consulted reports nothing wrong. + +@sprint-review (57339) pointed out that a fourth thing happened today which is +*not* that shape, and the difference is worth separating. While probing a test +suite, a mutation left the file unparseable. Jest printed: + +``` +Tests: 0 total +``` + +That is not an absence. It is a **count occupying the verdict slot**. The line +`Tests: 5 passed` and the line `Tests: 0 total` appear in the same position, in +the same format, in the same colour-free summary — and they mean opposite +things about whether anything was measured at all. One says the suite ran and +agreed with you. The other says the suite declined to run and has no opinion. + +**Why this is the harder one to catch.** With an absence you are at least +looking at a blank where evidence should be, and a blank can prompt a second +look. Here there IS a number, formatted exactly like a result, in the place you +learned to read results from. Scanning for "did anything fail", `0 total` +passes — nothing failed. It is a true statement and a useless one. + +The same slot-collision recurs well outside jest: + +- `grep -c` returning `0` — no matches, or a path that does not exist. +- A migration reporting `0 rows updated` — already applied, or matched nothing + because the predicate was wrong. +- A sweep reporting `0 offenders` — clean, or the scan never reached the + directory (entry 41's sibling bug, #1140: a non-recursive walk whose controls + passed while a subtree went unvisited). +- `git checkout -- ` exiting 0 on an untracked file — restored, or did + nothing at all. + +Each pair shares one output and splits on whether the instrument ran. + +**What made it land.** Both probe failures that day came from writing a probe +about probes that fail silently — a perl mutation that died on an unescaped +modifier and applied nothing, reporting `7/7`, and this one. The rule was being +violated in the act of being written down, which is the strongest argument +available for enforcing it mechanically rather than remembering it. + +**Rules earned.** + +- **Read the denominator, not the verdict.** `0 total` and `N passed` are + different claims. Before believing a green run, confirm the instrument + processed a non-zero population — and that the population is the one you + meant. +- **A probe must assert its own anchor before mutating.** Not after, and not + by reading the outcome: if the edit did not apply, the outcome is the same + shape as a successful edit that the tests could not detect. The fixed probes + in `threadRootDerivation.pgmem.test.js` and + `unguardedScriptImporters.test.js` both assert the anchor exists and fail + loudly when it does not. +- **Distinguish "did not fail" from "ran and passed" in your own reports too.** + Saying a suite is green when it emitted `0 total` is not a rounding error; it + is the same defect as the tooling, committed by a human reading the tooling. From 24fc64d77ff63502bde3da3dd89e5794c37ada9a Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Sat, 22 Aug 2026 13:38:24 -0700 Subject: [PATCH 2/5] docs(ax): entry 42 invented two of its own examples MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit @sprint-review (57346) tested the examples and two were false. Re-derived both rather than taking it: git checkout -- exit=1, "pathspec did not match any file(s)" git checkout -- exit=1 grep -c pat file (no match) prints 0, exit=1 grep -c pat nofile prints nothing, exit=2 + warning I had written that git exits 0 having silently done nothing, and that grep returns 0 for both no-match and bad-path. Neither is true. Worse, my own session printed that exact git error an hour earlier and I read it and said so at the time, then wrote the opposite into an entry about instruments whose output does not mean what it appears to. But the correction is more useful than the deletion, because the incidents were real. The tools are not silent — they signal on stderr and in $?. What discarded the signal was the INVOCATION: `n=$(grep -c ...)` keeps the count for the caller and strips the status, `grep -c ... | sed` makes $? sed's, and `2>/dev/null` removes the warning that separated the two cases. The number survives into the next command and the verdict does not. That is what actually happened to me, several times, in one session. So the entry now names three sub-mechanisms instead of one list: an instrument that declines to run and says so in the results slot; a signal that exists and is thrown away by the call; and a count that is genuinely ambiguous because nothing else was ever emitted (0 rows updated, 0 offenders — the latter being the real #1140 bug). Two rules added. Do not pipe or interpolate away the status of a step you are about to trust. And when you write down how a tool fails, run the tool — a remembered failure mode is a hypothesis, and it costs one command to make it an observation. Co-Authored-By: Claude Opus 5 --- docs/development/agent-experience-audit.md | 57 ++++++++++++++++++---- 1 file changed, 48 insertions(+), 9 deletions(-) diff --git a/docs/development/agent-experience-audit.md b/docs/development/agent-experience-audit.md index 9c574d587..318846eda 100644 --- a/docs/development/agent-experience-audit.md +++ b/docs/development/agent-experience-audit.md @@ -2605,18 +2605,47 @@ look. Here there IS a number, formatted exactly like a result, in the place you learned to read results from. Scanning for "did anything fail", `0 total` passes — nothing failed. It is a true statement and a useless one. -The same slot-collision recurs well outside jest: +**Two of the four examples I first wrote here were false, and @sprint-review +tested them (57346).** Keeping the correction visible, because how they were +false is more useful than the list was. -- `grep -c` returning `0` — no matches, or a path that does not exist. -- A migration reporting `0 rows updated` — already applied, or matched nothing - because the predicate was wrong. +I claimed `git checkout -- ` exits 0 having done nothing. It does +not — exit 1, `error: pathspec ... did not match any file(s) known to git`, +verified on git 2.50.1. **My own session had printed that exact error an hour +earlier**, and I had read it and said so at the time, before writing the +opposite into this entry. I claimed `grep -c` returns `0` for both no-match +and bad-path; it returns `0` with exit 1 for no-match, and *empty* with exit 2 +and a warning for a bad path. + +So the tools are not silent. They signal correctly, on stderr and in `$?`. + +**What discards the signal is the invocation.** Every one of these throws away +the channel carrying the distinction while keeping the one carrying the count: + +```sh +n=$(grep -c pat file) # $? is grep's... but the caller reads $n +grep -c pat file | sed '...' # $? is now sed's — always 0 +cmd 2>/dev/null # the warning that distinguished them is gone +``` + +That is how it actually happened to me, repeatedly, in one session: not a tool +that failed quietly, but a pipeline that preserved the number and dropped the +verdict. The count is what you interpolate into your next command or your next +sentence; the status is what you never see again. + +The genuinely ambiguous-at-exit-0 cases are the domain ones, where no channel +was lost because none exists: + +- A migration reporting `0 rows updated` — already applied, or the predicate + matched nothing. - A sweep reporting `0 offenders` — clean, or the scan never reached the - directory (entry 41's sibling bug, #1140: a non-recursive walk whose controls - passed while a subtree went unvisited). -- `git checkout -- ` exiting 0 on an untracked file — restored, or did - nothing at all. + directory. Not hypothetical: that was #1140, where a non-recursive walk left + a subtree unvisited while both controls passed. -Each pair shares one output and splits on whether the instrument ran. +Three sub-mechanisms, then, not one list: an instrument that declines to run +and says so in the results slot (`0 total`); a signal that exists and is +discarded by the call; and a count that is genuinely ambiguous because nothing +else was ever emitted. **What made it land.** Both probe failures that day came from writing a probe about probes that fail silently — a perl mutation that died on an unescaped @@ -2630,6 +2659,16 @@ available for enforcing it mechanically rather than remembering it. different claims. Before believing a green run, confirm the instrument processed a non-zero population — and that the population is the one you meant. +- **Do not pipe or interpolate away the status of a step you are about to + trust.** `$(cmd)` and `cmd | sed` both keep the number and drop the verdict, + and the number is the part that flows into your next sentence. If a command's + success is load-bearing, run it where `$?` is visible, or check it explicitly. +- **When you write down how a tool fails, run the tool.** Two of this entry's + four original examples were invented mechanisms for real incidents — plausible + stories about `git` and `grep` that neither program performs. One of them was + contradicted by output printed in my own session an hour before I wrote it. + A remembered failure mode is a hypothesis; it costs one command to make it an + observation. - **A probe must assert its own anchor before mutating.** Not after, and not by reading the outcome: if the edit did not apply, the outcome is the same shape as a successful edit that the tests could not detect. The fixed probes From db48339e1b686056189369ac4c12bd40cc36df0d Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Sat, 22 Aug 2026 13:40:12 -0700 Subject: [PATCH 3/5] docs(ax): the three suppressions differ, and one of them is recoverable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit @sprint-review (57347), two things. First, they were right that deleting the git example was the wrong fix. The hazard is the suppression the author chose, not the command — and the suppression is the part a reader controls, so it is the actionable half. Re-pointed rather than removed. Measured while re-pointing it, and the three are not interchangeable: cmd message shown, $?=1 both channels intact cmd 2>/dev/null message gone, $?=1 recoverable if you check $? cmd | head message shown, $?=0 status is head's cmd || true message shown, $?=0 deliberately discarded So `2>/dev/null` is the one that looks worst and is the mildest: it hides the explanation and keeps the verdict. The PIPE is the dangerous one, because nothing about `grep -c pat file | sed ...` announces that $? has stopped referring to grep. That distinction was not in the entry and is the whole difference between a mistake you can still catch and one you cannot. Second, and better evidence than anything I could construct: sprint-review's own first test reported exit 0 for a nonexistent path, and it was `head -2` eating git's status. Two people, separately, lost an exit status to a pipeline inside the same hour, while writing and verifying the entry about losing exit statuses to pipelines. Recorded, because it establishes the construct is invisible at the point of use rather than that either of us was inattentive. Co-Authored-By: Claude Opus 5 --- docs/development/agent-experience-audit.md | 25 ++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/docs/development/agent-experience-audit.md b/docs/development/agent-experience-audit.md index 318846eda..315ba9337 100644 --- a/docs/development/agent-experience-audit.md +++ b/docs/development/agent-experience-audit.md @@ -2633,6 +2633,31 @@ that failed quietly, but a pipeline that preserved the number and dropped the verdict. The count is what you interpolate into your next command or your next sentence; the status is what you never see again. +**The three common suppressions are not equivalent, and the difference decides +whether you can still catch it.** Measured on `git checkout -- `, +which exits 1 with a message: + +| Written as | Message | `$?` | Recoverable? | +|---|---|---|---| +| `cmd` | shown | 1 | yes — both channels intact | +| `cmd 2>/dev/null` | gone | **1** | yes, if you check `$?` | +| `cmd \| head` | shown | **0** | no — status is `head`'s | +| `cmd \|\| true` | shown | **0** | no — deliberately discarded | + +`2>/dev/null` is the one that looks worst and is actually the mildest: it hides +the explanation and keeps the verdict. The pipe is the dangerous one, because +nothing about `grep -c pat file | sed …` announces that `$?` no longer refers +to `grep`. @sprint-review's re-pointing (57347) is the version worth keeping — +the hazard is the suppression the author chose, not the command, because the +suppression is the part the reader controls. + +**Independently reproduced while checking this entry.** @sprint-review's first +test of the git behaviour reported exit 0 for a nonexistent path, which was +`head -2` consuming git's status — not git. So two people, working separately, +lost an exit status to a pipeline *within the same hour*, while writing and +verifying the entry about losing exit statuses to pipelines. Neither of us was +being careless; the construct is simply invisible at the point of use. + The genuinely ambiguous-at-exit-0 cases are the domain ones, where no channel was lost because none exists: From 3f01f2b44eb8c3f91fee2e490b02cd847b015fe0 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Sat, 22 Aug 2026 13:43:44 -0700 Subject: [PATCH 4/5] docs(ax): the table's own numbers were being carried across commands MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit @sprint-review (57351) ran the four-way table and found the transfer error. The numbers were correct for git, which exits 1 — and the paragraph directly below them reasons about grep, which exits 1 for no-match and 2 for a missing file. A reader carries the 1 across. In an entry about numbers meaning different things in the same slot, a table whose literals silently belong to a different command than the surrounding prose is the defect itself. Fixed by removing the literal rather than by adding a caveat: the status column is now `E`, whatever non-zero the command returns, so nothing can be carried anywhere. The table's actual claim was never about the value — it is about WHICH CHANNEL SURVIVES, and the literal was decoration that happened to be load-bearing in the wrong direction. Concrete values are listed separately and verified, because E is not one number even within one tool: git checkout -- E = 1 grep -c pat , no match E = 1 (prints "0") grep -c pat E = 2 (prints nothing, warns) And the observation that falls out of writing them down together: grep -c's two failure modes print DIFFERENT things, "0" versus nothing. So `n=$(grep -c ...)` yields n=0 for no-match and n="" for a missing file, and the empty string is the only hint that something other than "no matches" happened — a distinction that survives the very interpolation this entry says destroys the verdict. Worth knowing which half of the signal actually survives, rather than assuming all of it dies. Co-Authored-By: Claude Opus 5 --- docs/development/agent-experience-audit.md | 24 ++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/docs/development/agent-experience-audit.md b/docs/development/agent-experience-audit.md index 315ba9337..807934fe5 100644 --- a/docs/development/agent-experience-audit.md +++ b/docs/development/agent-experience-audit.md @@ -2634,16 +2634,32 @@ verdict. The count is what you interpolate into your next command or your next sentence; the status is what you never see again. **The three common suppressions are not equivalent, and the difference decides -whether you can still catch it.** Measured on `git checkout -- `, -which exits 1 with a message: +whether you can still catch it.** `E` below is whatever non-zero status the +command itself returns — deliberately not a literal, because the whole entry +is about numbers being carried between contexts where they mean different +things, and an earlier draft of this table printed git's `1` directly above a +paragraph reasoning about grep. @sprint-review caught the transfer (57351). | Written as | Message | `$?` | Recoverable? | |---|---|---|---| -| `cmd` | shown | 1 | yes — both channels intact | -| `cmd 2>/dev/null` | gone | **1** | yes, if you check `$?` | +| `cmd` | shown | `E` | yes — both channels intact | +| `cmd 2>/dev/null` | gone | **`E`** | yes, if you check `$?` | | `cmd \| head` | shown | **0** | no — status is `head`'s | | `cmd \|\| true` | shown | **0** | no — deliberately discarded | +Concrete values, verified, since `E` is not one number even within one tool: + +``` +git checkout -- E = 1 +grep -c pat , no match E = 1 (prints "0") +grep -c pat E = 2 (prints nothing, warns) +``` + +Note that `grep -c` alone spans two of them, and the two failures print +*different things* — `0` versus nothing. A caller doing `n=$(grep -c …)` gets +`n=0` in the first case and `n=""` in the second, and only the empty string +hints that something other than "no matches" occurred. + `2>/dev/null` is the one that looks worst and is actually the mildest: it hides the explanation and keeps the verdict. The pipe is the dangerous one, because nothing about `grep -c pat file | sed …` announces that `$?` no longer refers From eabfd4596e69b102ccfeb4868e222208dd155fa8 Mon Sep 17 00:00:00 2001 From: Lily Shen <115414357+lilyshen0722@users.noreply.github.com> Date: Sat, 22 Aug 2026 13:49:41 -0700 Subject: [PATCH 5/5] docs(ax): the surviving hint dies at the comparison, and shells disagree MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit @sprint-review (57354) extended the n=0 vs n="" finding: the distinction that survives interpolation is destroyed by the first numeric comparison. Verified, and it splits three ways rather than one: with n="" zsh bash [ "$n" -eq 0 ] TRUE, silently error, exit 2 (( n == 0 )) TRUE TRUE So `(( n == 0 ))` coerces empty to zero in both shells and always destroys the distinction, while `[ "$n" -eq 0 ]` is loud in bash and silent in zsh. The consequence is the part worth having: THE SAME LINE BEHAVES DIFFERENTLY LOCALLY AND IN CI. GitHub Actions `run:` steps are bash; an interactive macOS shell is zsh. That is the reverse of the usual direction — the check is noisy where nobody is watching and quiet where the author is writing it, which is precisely how a bad guard gets committed feeling fine. And one more layer, measured rather than assumed: the bash error does NOT abort a `set -e` script when written as `[ "$n" -eq 0 ] && ...`, because a command in condition position is exempt. Even the loud shell is loud only in its output, not in its exit path. Net window in which the distinction is recoverable: after assignment, before the first numeric use, and only if `[ -z "$n" ]` is tested first. Co-Authored-By: Claude Opus 5 --- docs/development/agent-experience-audit.md | 25 ++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/docs/development/agent-experience-audit.md b/docs/development/agent-experience-audit.md index 807934fe5..cb4932dd7 100644 --- a/docs/development/agent-experience-audit.md +++ b/docs/development/agent-experience-audit.md @@ -2660,6 +2660,31 @@ Note that `grep -c` alone spans two of them, and the two failures print `n=0` in the first case and `n=""` in the second, and only the empty string hints that something other than "no matches" occurred. +**That hint survives the interpolation and dies at the first comparison** — +@sprint-review (57354). Measured, and it is worse than they framed it, because +the two constructs and the two shells do not agree: + +| with `n=""` | zsh | bash | +|---|---|---| +| `[ "$n" -eq 0 ]` | **TRUE, silently** | error: `integer expression expected`, exit 2 | +| `(( n == 0 ))` | **TRUE** | **TRUE** | + +`(( n == 0 ))` coerces empty to zero in both, so it always destroys the +distinction. `[ "$n" -eq 0 ]` is loud in bash and silent in zsh — which means +**the same line behaves differently in a local terminal than in CI.** GitHub +Actions `run:` steps are bash; an interactive macOS shell is zsh. That is the +reverse of the usual failure direction: the check is noisy where nobody is +watching and quiet where the author is developing it. + +One more layer, measured: the bash error did **not** abort a `set -e` script +when written as `[ "$n" -eq 0 ] && …`, because a command in a condition +position is exempt from `set -e`. So even the loud shell stays loud only in +its output, not in its exit path. + +The window in which the distinction exists is therefore: after assignment, +before the first numeric use — and only if you test `[ -z "$n" ]` before +treating `$n` as a number. + `2>/dev/null` is the one that looks worst and is actually the mildest: it hides the explanation and keeps the verdict. The pipe is the dangerous one, because nothing about `grep -c pat file | sed …` announces that `$?` no longer refers