diff --git a/CHANGELOG.md b/CHANGELOG.md index d57dfad3..bcc71ecd 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2021,6 +2021,62 @@ true until the next version shipped. unanchored `grep -F` finds 3 occurrences because both renames EXTEND the name rather than replace it, so each renamed line still matches its own old form. Anchoring on the closing quote gives 1, which is the one that matters. +- `pgc_ledger.py orphan-scan` reports a ledger row that no record in its own part + matches, and `--prune` removes it only when no history would be lost (#983). + + The comparison already existed and already printed the answer. `rename-scan` pairs an + appearance with a disappearance, so an UNPAIRED disappearance -- a check deleted, or + renamed in a run where nothing appeared -- printed `vanished=2` and returned 0. Two rows + in the committed ledger named checks that no longer existed; the census counted both, and + every run for days said so in a line nobody acted on. **A guard that compels one list and + ignores the second manufactures the confidence that the thing is handled.** + + Driven on the real ledger, which is the only instance that matters: + + orphan: harness_selftest 330-... premise: all three runner functions were extracted + orphan: harness_selftest 330-... premise: and all three are callable + not checked: 44 row(s) in 1 part(s) this run does not contain + orphan scan: parts in the run=43, rows in those parts=861, orphans=2 + (0 carrying history), not checked=44 + orphan prune: removed 2 row(s), the ledger now holds 925 + + **A row carrying history is never pruned**, and one such row refuses the WHOLE prune. + No run can recreate the catalogue of what has been seen red, and removing the safe rows + while naming the unsafe ones would leave a partial job for whoever reads the output. + + **Rows in parts the run does not contain are counted out loud as `not checked`**, never + as present. Otherwise a single-suite log would certify the whole ledger, which is the + same defect one level up. + + **`--prune` refuses a part that skipped, which was this change's own worst bug.** + `not checked` protects a part the run does not contain. A part *contained but skipped + wholesale* fell in the gap: one SKIP record put the part in the run's `parts`, every other + row of that suite became an orphan, and `--prune` deleted the suite while reporting + `not checked=0` and `rc=0` -- the most confident output the tool can produce. Measured on + a three-row fixture for `analyze_differential`, whose PG17 run is a single SKIP; found by + @pgcolumnar-9b in review. Nine suites skip wholesale on PG17 and `suites_not_covered` is + 250, so seeding any one of them would have armed it. + + The rule is broader than that case deliberately: a SKIP **anywhere** in the part means + some arm did not run, so the run cannot tell "this row's check was deleted" from "this + row's check was skipped under a name that does not match it" -- #994's defect at suite + granularity rather than branch granularity. One skipped timing check therefore blocks + pruning that whole part, and that is the direction a deleting command should err in. The + control holds the other half: the same rows are still pruned when the part's record is a + PASS, so this is not a tool that refuses to prune anything. + + The four categories -- matched, orphan, unprunable, not checked -- are asserted to account + for every ledger row, because a classification that silently loses one is the failure this + tool exists to report. + + **It reports and is NOT wired into the gate, for a measured reason.** Scanned against a + real run, `340-the-binary-must-be-built-from.sh` records ONE skip under a name neither + of its two arms has (`the unreadable-source refusal`) whenever the box has no non-root + user to read as. On such a box two live ledger rows have no matching record, so an absent + record does not yet mean a removed check and a gate refusing on absence would redden a + correct run. Arming it needs those branches to record a SKIP under the names they stand + in for -- the conversion #965 made for the eleven timeout paths -- and an arm now pins + that shape so the day it changes, the arm says so. - Nine git bundles are out of the tree, `*.bundle` is ignored, and `310-a-compiled-artifact-must-not-be.sh` now covers transfer artifacts as well as diff --git a/test/check_ledger.tsv b/test/check_ledger.tsv index 8511e774..d2e29c8a 100644 --- a/test/check_ledger.tsv +++ b/test/check_ledger.tsv @@ -322,9 +322,7 @@ harness_selftest 330-the-incomplete-path-must-run-whole control: and the major r harness_selftest 330-the-incomplete-path-must-run-whole control: the same loop leaves a passing suite passing never - harness_selftest 330-the-incomplete-path-must-run-whole every function the collect loop calls is defined here never - harness_selftest 330-the-incomplete-path-must-run-whole premise: all five runner functions were extracted, not empty ranges never - -harness_selftest 330-the-incomplete-path-must-run-whole premise: all three runner functions were extracted, not empty ranges never - harness_selftest 330-the-incomplete-path-must-run-whole premise: and all five are callable never - -harness_selftest 330-the-incomplete-path-must-run-whole premise: and all three are callable never - harness_selftest 330-the-incomplete-path-must-run-whole premise: and each extraction ends at its own closing brace never - harness_selftest 330-the-incomplete-path-must-run-whole premise: the major-verdict branch was extracted, not an empty range never - harness_selftest 330-the-incomplete-path-must-run-whole premise: the runner's collect loop was extracted, not an empty range never - @@ -655,6 +653,10 @@ harness_selftest 400-a-check-result-must-be-machine the same ratio check, ENABLE harness_selftest 400-a-check-result-must-be-machine the same timing check, ENABLED, emits one record and passes never - harness_selftest 400-a-check-result-must-be-machine the sweep finds a check inside a PIPED loop never - harness_selftest 410-a-check-must-have-been-red --mutation across two failing checks in one run is refused never - +harness_selftest 410-a-check-must-have-been-red --prune REFUSES the whole prune when any orphan carries history never - +harness_selftest 410-a-check-must-have-been-red --prune removes NOTHING from a part that skipped never - +harness_selftest 410-a-check-must-have-been-red --prune removes a historyless orphan and says which never - +harness_selftest 410-a-check-must-have-been-red a before-log and an after-log together are refused, as rename-scan refuses them never - harness_selftest 410-a-check-must-have-been-red a before-log and an after-log together are refused, not silently empty never - harness_selftest 410-a-check-must-have-been-red a budget stating no census is not refused, because absence is not a contradiction never - harness_selftest 410-a-check-must-have-been-red a budget that does not exist at the prior is a note, not a refusal never - @@ -670,6 +672,7 @@ harness_selftest 410-a-check-must-have-been-red a date that is not a date is ref harness_selftest 410-a-check-must-have-been-red a deliberate break says so with --mutation never - harness_selftest 410-a-check-must-have-been-red a gate over a nonexistent log is an integrity failure, not a pass never - harness_selftest 410-a-check-must-have-been-red a later green run does not erase an observation never - +harness_selftest 410-a-check-must-have-been-red a ledger row no record in its own part matches is named an orphan never - harness_selftest 410-a-check-must-have-been-red a log carrying a FAIL is refused when no reason is given never - harness_selftest 410-a-check-must-have-been-red a name that appeared while another disappeared is reported as a rename never - harness_selftest 410-a-check-must-have-been-red a named mutation is recorded against the check that reddened never - @@ -679,6 +682,9 @@ harness_selftest 410-a-check-must-have-been-red a ref that exists with the budge harness_selftest 410-a-check-must-have-been-red a ref that exists without the budget is still the bootstrap case never - harness_selftest 410-a-check-must-have-been-red a refusal keeps the regenerate-the-ledger wording never - harness_selftest 410-a-check-must-have-been-red a rename in one part survives an addition in another never - +harness_selftest 410-a-check-must-have-been-red a row in a part that SKIPPED is reported as unprunable, not as an orphan never - +harness_selftest 410-a-check-must-have-been-red a row in a part the run does not contain is not called an orphan never - +harness_selftest 410-a-check-must-have-been-red a run that emits every row in its parts is clean never - harness_selftest 410-a-check-must-have-been-red a run whose checks are all ledgered passes the gate never - harness_selftest 410-a-check-must-have-been-red a second mutation ACCUMULATES rather than replacing the first never - harness_selftest 410-a-check-must-have-been-red a stale prior is named WITH its distance from HEAD never - @@ -688,23 +694,28 @@ harness_selftest 410-a-check-must-have-been-red an empty check name is one too, harness_selftest 410-a-check-must-have-been-red an empty log is one too, because there is nothing to reconcile never - harness_selftest 410-a-check-must-have-been-red an integrity failure says regenerating will not help never - harness_selftest 410-a-check-must-have-been-red an older observation does not overwrite a newer one never - +harness_selftest 410-a-check-must-have-been-red an orphan carrying history is reported as carrying it never - +harness_selftest 410-a-check-must-have-been-red and --prune does NOT turn that 1 into a 0, because it pruned nothing never - harness_selftest 410-a-check-must-have-been-red and CI collects from the retained path rather than the deleted one never - harness_selftest 410-a-check-must-have-been-red and a genuine observation says so with --reds-are-real never - harness_selftest 410-a-check-must-have-been-red and a level prior carries no distance, so zero is silent never - harness_selftest 410-a-check-must-have-been-red and a log that does not reconcile with its own checks run: never - harness_selftest 410-a-check-must-have-been-red and a newer one does never - +harness_selftest 410-a-check-must-have-been-red and a prune with nothing left to remove is clean, not an error never - harness_selftest 410-a-check-must-have-been-red and a raise against that base ref is refused never - harness_selftest 410-a-check-must-have-been-red and a record missing its verdict never - harness_selftest 410-a-check-must-have-been-red and an empty mutation is a placeholder, not an empty last field never - harness_selftest 410-a-check-must-have-been-red and an undated merge does not erase a known date never - harness_selftest 410-a-check-must-have-been-red and it copies them BEFORE removing the build directory, which is the only order that works never - harness_selftest 410-a-check-must-have-been-red and it entered as debt, not as an observation nothing made never - +harness_selftest 410-a-check-must-have-been-red and it is REFUSED, not merely printed never - harness_selftest 410-a-check-must-have-been-red and it is a different sentence from the refusal, not the same one twice never - harness_selftest 410-a-check-must-have-been-red and it is named, so the author knows which one never - harness_selftest 410-a-check-must-have-been-red and it is that case that says the change introduces the file never - harness_selftest 410-a-check-must-have-been-red and it is the new one that is named, not the one already ledgered never - harness_selftest 410-a-check-must-have-been-red and it names the verdict, so the author knows which record never - harness_selftest 410-a-check-must-have-been-red and it runs before the build directory is removed, which is the only place it can never - +harness_selftest 410-a-check-must-have-been-red and it says so rather than declining silently never - harness_selftest 410-a-check-must-have-been-red and it says the change introduces the file rather than raising anything never - harness_selftest 410-a-check-must-have-been-red and it says the ref does not resolve, rather than claiming the file is new never - harness_selftest 410-a-check-must-have-been-red and it says why, rather than falling back to something weaker never - @@ -720,6 +731,8 @@ harness_selftest 410-a-check-must-have-been-red and the census line it always pr harness_selftest 410-a-check-must-have-been-red and the comparison passes when the ceiling did not rise never - harness_selftest 410-a-check-must-have-been-red and the gate prints that it compared them never - harness_selftest 410-a-check-must-have-been-red and the history it is about to lose travels with it never - +harness_selftest 410-a-check-must-have-been-red and the history it would lose is printed with it never - +harness_selftest 410-a-check-must-have-been-red and the ledger is one row shorter afterwards never - harness_selftest 410-a-check-must-have-been-red and the message says how to fix it, because regenerating is the intended action never - harness_selftest 410-a-check-must-have-been-red and the refusal names both values never - harness_selftest 410-a-check-must-have-been-red and the refusal names how many failed, so the author can narrow the run never - @@ -727,26 +740,41 @@ harness_selftest 410-a-check-must-have-been-red and the refusal names the check harness_selftest 410-a-check-must-have-been-red and the refusal names the raise never - harness_selftest 410-a-check-must-have-been-red and the refusal quotes the claim beside the measurement never - harness_selftest 410-a-check-must-have-been-red and the runner names no remote at that call site never - +harness_selftest 410-a-check-must-have-been-red and the scan states how many rows it could not speak about never - +harness_selftest 410-a-check-must-have-been-red and the summary keeps the two apart, so a zero orphan count is not a clean bill never - harness_selftest 410-a-check-must-have-been-red auto refuses when GITHUB_BASE_REF names a ref that is not here never - harness_selftest 410-a-check-must-have-been-red auto uses the base ref when it resolves, and names it never - harness_selftest 410-a-check-must-have-been-red auto with no base ref and no upstream is an integrity failure never - harness_selftest 410-a-check-must-have-been-red both failure arms fail the major never - harness_selftest 410-a-check-must-have-been-red but that suite is counted as not covered, which is the debt never - harness_selftest 410-a-check-must-have-been-red but the gate says so, so the skip is visible rather than silent never - +harness_selftest 410-a-check-must-have-been-red control: a prune that leaves nothing outstanding returns 0 never - harness_selftest 410-a-check-must-have-been-red control: a well-formed log still merges never - harness_selftest 410-a-check-must-have-been-red control: an all-PASS log still merges with no flag at all never - +harness_selftest 410-a-check-must-have-been-red control: and that one did prune, so 0 is not a refusal in disguise never - +harness_selftest 410-a-check-must-have-been-red control: and the ledger really is shorter afterwards never - +harness_selftest 410-a-check-must-have-been-red control: the row in the part the run never mentioned SURVIVES the prune never - harness_selftest 410-a-check-must-have-been-red control: the same log merges with a different reason, so the refusal above is --mutation-across-two-checks and not the log never - +harness_selftest 410-a-check-must-have-been-red control: the same two rows ARE pruned when that part's record is a PASS never - harness_selftest 410-a-check-must-have-been-red each says what was wrong with the input never - harness_selftest 410-a-check-must-have-been-red every committed row has five fields never - harness_selftest 410-a-check-must-have-been-red every row has five fields and no trailing tab never - +harness_selftest 410-a-check-must-have-been-red it is still a finding, so the scan does not return success never - harness_selftest 410-a-check-must-have-been-red lowering it is allowed, which is the direction the burn-down goes never - harness_selftest 410-a-check-must-have-been-red merging a green run records both checks never - harness_selftest 410-a-check-must-have-been-red nor is one merely removed never - harness_selftest 410-a-check-must-have-been-red once the suite is covered, a new check in it IS refused never - harness_selftest 410-a-check-must-have-been-red one --mutation cannot be attributed across several runs at once never - harness_selftest 410-a-check-must-have-been-red premise: and it is a file git has never seen, which is the case that used to fail open never - +harness_selftest 410-a-check-must-have-been-red premise: and it really pruned nothing -- the rows are all still there never - +harness_selftest 410-a-check-must-have-been-red premise: and it reported all four of its categories, so a zero is a measurement never - harness_selftest 410-a-check-must-have-been-red premise: and the nobudget branch does not, which is the bootstrap shape never - +harness_selftest 410-a-check-must-have-been-red premise: and the refusal removed NOTHING -- the row is still there never - +harness_selftest 410-a-check-must-have-been-red premise: and the run the scan is given holds only one of them never - +harness_selftest 410-a-check-must-have-been-red premise: and the skipped run emits exactly one of them, as a SKIP never - +harness_selftest 410-a-check-must-have-been-red premise: and the two arms it stands in for are still named in the ledger never - harness_selftest 410-a-check-must-have-been-red premise: both are never, so this ledger's census is two never - +harness_selftest 410-a-check-must-have-been-red premise: that part has rows the skipped run cannot speak for never - harness_selftest 410-a-check-must-have-been-red premise: the budget is a tracked file too never - harness_selftest 410-a-check-must-have-been-red premise: the budget was restored byte-exact never - harness_selftest 410-a-check-must-have-been-red premise: the check has history before the rename never - @@ -754,24 +782,33 @@ harness_selftest 410-a-check-must-have-been-red premise: the fixture ledger hold harness_selftest 410-a-check-must-have-been-red premise: the fixture log names checks the real ledger already knows never - harness_selftest 410-a-check-must-have-been-red premise: the gate's status block was found in the runner never - harness_selftest 410-a-check-must-have-been-red premise: the hasbudget branch carries the budget never - +harness_selftest 410-a-check-must-have-been-red premise: the ledger holds all three of that part's rows never - +harness_selftest 410-a-check-must-have-been-red premise: the ledger holds all three rows before the scan never - harness_selftest 410-a-check-must-have-been-red premise: the ledger is not empty, so the partition means something never - harness_selftest 410-a-check-must-have-been-red premise: the ledger itself is a tracked file, not a variable never - harness_selftest 410-a-check-must-have-been-red premise: the ledger tool exists never - +harness_selftest 410-a-check-must-have-been-red premise: the orphan about to be pruned carries no history never - +harness_selftest 410-a-check-must-have-been-red premise: the orphan now carries a date and a mutation never - harness_selftest 410-a-check-must-have-been-red premise: the raised copy really does carry a higher ceiling never - harness_selftest 410-a-check-must-have-been-red premise: the real budget is inside a git repository never - harness_selftest 410-a-check-must-have-been-red premise: the scratch prior really is three commits behind never - harness_selftest 410-a-check-must-have-been-red premise: the scratch repo has a committed ceiling and no upstream never - harness_selftest 410-a-check-must-have-been-red premise: the scratch repo has a prior ceiling committed never - +harness_selftest 410-a-check-must-have-been-red premise: the skip-loop sweep is present, so its counts can be read never - harness_selftest 410-a-check-must-have-been-red premise: the workflow file is where this part thinks it is never - harness_selftest 410-a-check-must-have-been-red raising the ceiling above its committed value is refused never - harness_selftest 410-a-check-must-have-been-red raising the ceiling in the tracked file is refused never - harness_selftest 410-a-check-must-have-been-red regenerating the ledger lets the new check through never - harness_selftest 410-a-check-must-have-been-red the budget names a ceiling and a census, and says which is which never - harness_selftest 410-a-check-must-have-been-red the ceiling refuses being exceeded never - +harness_selftest 410-a-check-must-have-been-red the check the run still emits is not called an orphan never - harness_selftest 410-a-check-must-have-been-red the committed census matches the committed ledger never - harness_selftest 410-a-check-must-have-been-red the distance travels with a refusal too, not only with a pass never - +harness_selftest 410-a-check-must-have-been-red the four categories account for every ledger row never - harness_selftest 410-a-check-must-have-been-red the gate refuses to run without the registered suite list never - harness_selftest 410-a-check-must-have-been-red the ledger partitions into observed and never never - +harness_selftest 410-a-check-must-have-been-red the orphan scan stays a REPORT while any skip loop cannot be compared to its arms never - +harness_selftest 410-a-check-must-have-been-red the refusal says why, rather than only that it refused never - harness_selftest 410-a-check-must-have-been-red the runner asks the tool to resolve the prior rather than naming one never - harness_selftest 410-a-check-must-have-been-red the runner captures the gate's status rather than only its success never - harness_selftest 410-a-check-must-have-been-red the runner invokes the ledger gate never - @@ -779,6 +816,7 @@ harness_selftest 410-a-check-must-have-been-red the runner keeps the logs somewh harness_selftest 410-a-check-must-have-been-red the runner passes --against to the gate never - harness_selftest 410-a-check-must-have-been-red the same check in two logs is two runs, not a duplicate never - harness_selftest 410-a-check-must-have-been-red the same name twice in ONE log is a duplicate, and is named never - +harness_selftest 410-a-check-must-have-been-red the scan reports a finding on a skipped part never - harness_selftest 410-a-check-must-have-been-red the stable check is not reported never - harness_selftest 410-a-check-must-have-been-red the suites job fetches the PR base for the ceiling comparison never - harness_selftest 420-a-deleted-part-must-be-visible and every name in the manifest is a part on disk, so a deletion reddens never - diff --git a/test/check_ledger_budget.txt b/test/check_ledger_budget.txt index 762e8b5d..fb09efcd 100644 --- a/test/check_ledger_budget.txt +++ b/test/check_ledger_budget.txt @@ -34,4 +34,4 @@ suites_not_covered 250 # Without that it is a hand-maintained count that drifts, which is the failure # this repository has spent a day proving. It is not a ceiling; it is a # measurement that must be true. -checks_never_observed_red 913 +checks_never_observed_red 951 diff --git a/test/pgc_ledger.py b/test/pgc_ledger.py index a717c354..791170f7 100755 --- a/test/pgc_ledger.py +++ b/test/pgc_ledger.py @@ -28,6 +28,14 @@ refuses an increase, so widening the debt is an edit a reviewer sees AND a gate refuses, rather than either alone. +THE COMPARISON RUNS BOTH WAYS +----------------------------- +`gate` refuses a check the ledger has never seen. `orphan-scan` reports a ledger +row no record in its own part matches -- the other direction, which for a long +time printed `vanished=N` and refused nothing while two rows named checks that no +longer existed. It reports rather than gates, for a measured reason given in its +own docstring: an absent record does not yet mean a removed check. + WHAT THE GATE REFUSES --------------------- A check the committed ledger has never seen. That is the allowlist the issue asks @@ -46,6 +54,30 @@ It is not only mutation runs. Every real CI red fills it, every flake, every bisect. A mutation run is the deliberate accelerator. +REGENERATING ACROSS A REBASE, in this order, and the order is the point +----------------------------------------------------------------------- +Paid four times on one PR before it was written down. A rebase moves the LEDGER +without moving the BUDGET: git merges both sides' rows into the tsv and keeps one +side's number in the budget, so the committed pair contradicts itself before +anything is run. Four arms then fail and all four trace to that one cause -- one +asserts the pair agrees, three run the real gate, which correctly refuses a +contradiction. Diagnosing it from those four failures costs an hour. + + 1. rebase onto the new base FIRST + 2. DERIVE to reconcile the budget from the merged tsv, before running + 3. run the suite on the REBASED tree, and guard the log + 4. merge the guarded run + 5. prune orphans, if any + 6. DERIVE again the final census, read back from the file + +Two derives, not one: step 2 makes the tree self-consistent so the suite can pass at +all, step 6 records the result of steps 4 and 5. Both are READ BACK from the tsv -- +`old + n` is right once and wrong every time after. + +And re-run whatever your evidence names whose FILES moved in the rebase. A gate +statement is a claim about a tree, and a rebase silently changes which tree; saying +which suites you re-ran and which you did not is part of the claim. + FAIL CLOSED ----------- An unreadable file, an empty one, or a record with too few fields is an ERROR. @@ -357,6 +389,159 @@ def cmd_rename_scan(args): return rc +def cmd_orphan_scan(args): + """A ledger row that no record in its OWN PART matches: the unpaired half. + + `rename-scan` pairs an appearance with a disappearance. An unpaired + disappearance -- a check deleted, or renamed in a run where nothing appeared -- + was printed as `vanished=N` and refused nothing. Two such rows sat in the + committed ledger naming checks that no longer existed; the census counted both, + and every run returned 0 while the note scrolled past. A guard that compels one + list and ignores the second manufactures the confidence that the thing is + handled. + + SCOPED TO THE PARTS THE RUN CONTAINS, and the scope is REPORTED, not assumed. + A one-suite log has nothing to say about another suite's rows. Counting those + as present would make a single-suite run certify the whole ledger, so they are + counted OUT LOUD as `not checked` instead. + + WHY THIS REPORTS AND IS NOT WIRED INTO THE GATE. Measured, not assumed: part + 340 records ONE skip under a DIFFERENT name ("the unreadable-source refusal") + when the box has no non-root user to read as, rather than skipping its two + named arms. On such a box two committed rows have no matching record and are + not removed checks, so a gate refusing on absence would redden a correct run. + Arming this needs those branches to record a SKIP under the names they stand + in for -- the same conversion #965 made for the eleven timeout paths. + + A ROW CARRYING HISTORY IS NEVER PRUNED. The catalogue of what has been seen + red is the thing this ledger exists to be, and no run can recreate it. Dropping + an entry because a name moved is the precise loss `rename-scan` was written to + prevent, so `--prune` refuses the WHOLE prune when any orphan carries history + rather than removing the safe ones and leaving a partial job to be finished by + whoever reads the output. + + AND A PART HOLDING ANY SKIP IS UNPRUNABLE, which is the first version's worst + bug rather than a refinement of it. `not checked` protects a part the run does + not contain at all. A part the run CONTAINS BUT SKIPPED WHOLESALE fell between + the two: one SKIP record put the part in `parts`, every other row of that suite + became an orphan, and `--prune` deleted the suite while reporting + `not checked=0` and rc=0 -- the most confident output the tool can produce. + Measured on a three-row fixture for `analyze_differential`, whose run is one + SKIP on PG17; reported by @pgcolumnar-9b reviewing this change. + + THE RULE IS BROADER THAN THAT CASE ON PURPOSE. A SKIP anywhere in the part + means some arm did not run, so the run cannot distinguish "this row's check was + deleted" from "this row's check was skipped under a name that does not match + it" -- which is #994's defect, at suite granularity instead of branch + granularity. That is the same sentence this docstring already uses about part + 340, so writing the hole one level up was not an oversight I get to call + subtle. + + It is deliberately conservative: one skipped timing check blocks pruning that + whole part. Prune is a rare, deliberate act; a refusal costs a sentence and a + deletion costs history no run can recreate. + + THE EXIT CODES, stated because a caller only ever sees the code: + + 0 nothing left to report: no orphan and nothing unprunable + 1 something is still there -- an orphan, or a row this run cannot speak for + 2 an integrity failure, or a prune refused because history would be lost + + `--prune` returning 0 when it had pruned NOTHING was the first version's subtler + bug, reported by @jdatcmd in review. A caller that scans, sees 1, re-runs with + `--prune` and sees 0 reads "it pruned them" -- when nothing was pruned and nothing + could be. Prose covers a human; a script sees only the code. So 0 now means the + ledger and the run agree, and anything outstanding keeps the 1 the scan gave. + """ + runs = _by_run(args.logs) + if len(runs) > 1: + raise LedgerError( + f"orphan-scan compares ONE run against the ledger, but got {len(runs)} logs: " + "the union of a before-log and an after-log hides the disappearance") + rows = read_ledger(args.ledger) + verdicts = runs[0][1] + now = set(verdicts) + + parts = {(s, p) for s, p, _ in now} + # A part holding ANY SKIP cannot speak about absence: see the docstring. + skipped_parts = {(s, p) for (s, p, _), v in verdicts.items() if "SKIP" in v} + checkable = {k for k in rows if (k[0], k[1]) in parts} + unchecked = sorted(set(rows) - checkable) + absent = sorted(checkable - now) + orphans = [k for k in absent if (k[0], k[1]) not in skipped_parts] + unprunable = [k for k in absent if (k[0], k[1]) in skipped_parts] + + with_history = [k for k in orphans + if rows[k][0] != NEVER or rows[k][1]] + historyless = [k for k in orphans if k not in with_history] + + for k in orphans: + last, muts = rows[k] + if k in with_history: + print(f" ORPHAN CARRYING HISTORY: {k[0]}\t{k[1]}\t{k[2]} " + f"(last red {last}, mutations: {';'.join(sorted(muts)) or NONE})") + else: + print(f" orphan: {k[0]}\t{k[1]}\t{k[2]} (no history)") + + # The parts the run never mentioned, named rather than counted alone: a number + # with no names is a number nobody can act on. + if unchecked: + silent = sorted({(k[0], k[1]) for k in unchecked}) + print(f" not checked: {len(unchecked)} row(s) in {len(silent)} part(s) this run " + f"does not contain, so it cannot speak about them: " + + ", ".join(f"{a}/{b}" for a, b in silent[:5]) + + (" ..." if len(silent) > 5 else "")) + + # NAMED, not just counted: a number with no names is a number nobody can act on. + if unprunable: + parts_named = sorted({(k[0], k[1]) for k in unprunable}) + print(f" unprunable: {len(unprunable)} row(s) in {len(parts_named)} part(s) that " + f"SKIPPED at least one check, so absence there is not removal: " + + ", ".join(f"{a}/{b}" for a, b in parts_named[:5]) + + (" ..." if len(parts_named) > 5 else "")) + for k in unprunable[:5]: + print(f" {k[0]}\t{k[1]}\t{k[2]}") + + print(f" orphan scan: parts in the run={len(parts)}, rows in those parts={len(checkable)}, " + f"orphans={len(orphans)} ({len(with_history)} carrying history), " + f"unprunable={len(unprunable)}, not checked={len(unchecked)}") + # The four categories must account for every row, or a row went missing in the + # classification itself -- which is the failure this tool exists to report. + matched = len(set(rows) & now) + if matched + len(orphans) + len(unprunable) + len(unchecked) != len(rows): + raise LedgerError( + f"classification lost rows: matched {matched} + orphans {len(orphans)} + " + f"unprunable {len(unprunable)} + not checked {len(unchecked)} != {len(rows)} " + f"ledger rows -- every row must land in exactly one of the four") + + if not args.prune: + return 1 if (orphans or unprunable) else 0 + + if unprunable: + print(f" not pruning {len(unprunable)} row(s) in a part that skipped: the run did " + "not exercise those checks, so their absence is not removal") + + if with_history: + print(f" refusing to prune: {len(with_history)} orphan row(s) carry history, and " + "the catalogue is what this ledger is for -- no run can recreate it") + print(" reconcile them instead: rename the ledger row to the check's new name, " + "or say in the commit why the history may go") + return 2 + + if not historyless: + # Nothing WAS pruned. If anything is still outstanding the caller must not read + # that as success, so the scan's own verdict stands. + return 1 if unprunable else 0 + + for k in historyless: + print(f" pruned: {k[0]}\t{k[1]}\t{k[2]}") + del rows[k] + write_ledger(args.ledger, rows) + print(f" orphan prune: removed {len(historyless)} row(s), the ledger now holds {len(rows)}") + # Pruning some of it is not finishing it. + return 1 if unprunable else 0 + + def read_budget(path): out = {} text = pathlib.Path(path).read_text() @@ -539,7 +724,7 @@ def cmd_gate(args): # The suite restriction is not a softening, it is the meaning of # suites_not_covered: the gate cannot refuse a new check in a suite it has # never seen, because it has no idea which of that suite's checks are new. - # Without it the gate refuses every check of all 250 uncovered suites and + # Without it the gate refuses every check of every uncovered suite and # reddens the whole matrix on the first run -- which is a gate somebody turns # off, the failure mode this issue family exists to prevent. # @@ -669,6 +854,15 @@ def main(argv=None): r.add_argument("logs", nargs="+") r.set_defaults(fn=cmd_rename_scan) + o = sub.add_parser("orphan-scan", + help="refuse a ledger row no record in its own part matches") + o.add_argument("--ledger", required=True) + o.add_argument("--prune", action="store_true", + help="remove orphan rows that carry no history; refuse the whole " + "prune if any of them does") + o.add_argument("logs", nargs="+") + o.set_defaults(fn=cmd_orphan_scan) + g = sub.add_parser("gate", help="refuse a check the ledger has never seen") g.add_argument("--ledger", required=True) g.add_argument("--budget", required=True) diff --git a/test/pytest/TESTS.md b/test/pytest/TESTS.md index 5e21c01f..8cea5118 100644 --- a/test/pytest/TESTS.md +++ b/test/pytest/TESTS.md @@ -2237,10 +2237,46 @@ A global positional pairing misses a real rename whenever unrelated movement in part shifts the ordering. Given a before-log and an after-log together the vanished name is present in the union, so the scan **refuses** rather than silently finding nothing. +### `test_an_orphan_row_is_named_and_the_unscanned_rows_are_counted` + +`rename-scan` pairs an appearance with a disappearance, so an **unpaired** disappearance +— a check deleted, or renamed in a run where nothing appeared — printed `vanished=N` and +refused nothing. Two rows in the committed ledger named checks that no longer existed; +the census counted both and every run returned 0. + +The assertion that matters is the **scope**. A row in a part the run does not contain is +not an orphan, because the run cannot speak about it — counting those as present would +let a one-suite log certify the whole ledger. So the scan states how many rows it could +not speak about, and this test pins that number as well as the orphan it found. + +### `test_a_part_that_skipped_is_unprunable_because_absence_is_not_removal` + +The first version of `--prune` **deleted a suite**. One SKIP record put the part in the +run's `parts`, so every other row of that suite became an orphan, and the prune removed them +while reporting `not checked=0` and `rc=0` — the most confident output the tool can produce. +`not checked` protects a part the run does not contain; a part *contained but skipped +wholesale* fell in the gap between the two. + +The rule is deliberately broader than that case: a SKIP **anywhere** in the part means some +arm did not run, so the run cannot tell a deleted check from one skipped under a name that +does not match it. One skipped timing check blocks pruning that whole part, which is the +direction a deleting command should err in. + +The control is the half that matters — the same two rows must still be pruned when the +part's record is a `PASS`, or this is simply a tool that refuses to prune anything. + +### `test_prune_drops_a_historyless_orphan_and_refuses_one_carrying_history` + +The catalogue of what has been seen red is what the ledger exists to be, and no run can +recreate it. `--prune` therefore refuses the **whole** prune when any orphan carries +history, rather than removing the safe ones and leaving a partial job for whoever reads +the output. A historyless orphan is removed and named as it goes; the row in the part the +run never mentioned survives, which is the control that the scope holds under a write. + ### `test_the_gate_refuses_a_new_check_only_in_a_suite_it_covers` The suite restriction is the *meaning* of `suites_not_covered`, not a softening: without -it the gate refuses every check of all 250 uncovered suites and reddens the whole matrix +it the gate refuses every check of every uncovered suite and reddens the whole matrix on its first run. It tightens on its own as suites are seeded, and the deadlock that shipped is pinned as its own arm — regenerating the ledger lets a new check through. diff --git a/test/pytest/test_mutation_ledger.py b/test/pytest/test_mutation_ledger.py index 0248af4b..b3eb023f 100644 --- a/test/pytest/test_mutation_ledger.py +++ b/test/pytest/test_mutation_ledger.py @@ -173,10 +173,146 @@ def test_renames_are_grouped_by_part_and_scanned_against_one_run(tmp_path, expec "a before-log and an after-log together are refused, not silently empty") +def test_an_orphan_row_is_named_and_the_unscanned_rows_are_counted(tmp_path, expect): + """The other direction of the set-compare: a ledger row no record matches. + + `rename-scan` pairs an appearance with a disappearance, so an UNPAIRED + disappearance printed `vanished=N` and refused nothing -- two rows naming checks + that no longer existed sat in the committed ledger while the census counted both. + + THE SCOPE IS THE ASSERTION THAT MATTERS. A row in a part the run does not contain + is not an orphan, because the run cannot speak about it; counting those as present + would let a one-suite log certify the whole ledger. So the scan says how many rows + it could not speak about, and this test pins that number rather than only the + orphan it found. + """ + ledger = _w(tmp_path, "l.tsv", "") + before = _w(tmp_path, "b.log", + "RESULT\tdemo\tpart1\tstill here\tPASS\t\n" + "RESULT\tdemo\tpart1\tgone tomorrow\tPASS\t\n" + "RESULT\tdemo\tpartZ\telsewhere\tPASS\t\nchecks run: 3\n") + after = _w(tmp_path, "a.log", + "RESULT\tdemo\tpart1\tstill here\tPASS\t\nchecks run: 1\n") + _run("merge", "--ledger", ledger, "--date", "2026-09-01", before) + expect.num(len(_rows(ledger)), 3, "premise: the ledger holds all three rows") + + out, rc = _run("orphan-scan", "--ledger", ledger, after) + expect.num(out.count("orphan: demo\tpart1\tgone tomorrow"), 1, + "a row no record in its own part matches is named an orphan") + expect.num(rc, 1, "and it is refused, not merely printed") + expect.num(out.count("still here"), 0, "the check the run still emits is not an orphan") + expect.num(out.count("elsewhere"), 0, + "nor is a row in a part the run does not contain") + expect.num(out.count("not checked=1"), 1, + "and the scan states how many rows it could not speak about") + + expect.num(_run("orphan-scan", "--ledger", ledger, before)[1], 0, + "a run that emits every row in its parts is clean") + expect.num(_run("orphan-scan", "--ledger", ledger, before, after)[1], 2, + "a before-log and an after-log together are refused, as rename-scan refuses them") + + +def test_prune_drops_a_historyless_orphan_and_refuses_one_carrying_history(tmp_path, expect): + """The catalogue of what has been seen red is what this ledger exists to be. + + No run can recreate it, so dropping an entry because a name moved is the precise + loss `rename-scan` was written to prevent. `--prune` refuses the WHOLE prune when + any orphan carries history, rather than removing the safe ones and leaving a + partial job for whoever reads the output. + """ + after = _w(tmp_path, "a.log", + "RESULT\tdemo\tpart1\tstill here\tPASS\t\nchecks run: 1\n") + + plain = _w(tmp_path, "plain.tsv", "") + _run("merge", "--ledger", plain, "--date", "2026-09-01", + _w(tmp_path, "p.log", + "RESULT\tdemo\tpart1\tstill here\tPASS\t\n" + "RESULT\tdemo\tpart1\tgone tomorrow\tPASS\t\n" + "RESULT\tdemo\tpartZ\telsewhere\tPASS\t\nchecks run: 3\n")) + out, rc = _run("orphan-scan", "--prune", "--ledger", plain, after) + expect.num(out.count("pruned: demo\tpart1\tgone tomorrow"), 1, + "a historyless orphan is pruned, and named as it goes") + expect.num(rc, 0, "and a prune that did its job is not an error") + names = {r[2] for r in _rows(plain)} + expect.num(len(names), 2, "the ledger is one row shorter") + expect.num(1 if "elsewhere" in names else 0, 1, + "control: the row in the part the run never mentioned survives the prune") + expect.num(_run("orphan-scan", "--prune", "--ledger", plain, after)[1], 0, + "and a prune with nothing left to remove is clean, not an error") + + hist = _w(tmp_path, "hist.tsv", "") + _run("merge", "--reds-are-real", "--mutation", "drop the guard", "--ledger", hist, + "--date", "2026-09-01", + _w(tmp_path, "h.log", + "RESULT\tdemo\tpart1\tstill here\tPASS\t\n" + "RESULT\tdemo\tpart1\tgone tomorrow\tFAIL\t\nchecks run: 2\n")) + expect.text({r[2]: r[3] for r in _rows(hist)}["gone tomorrow"], "2026-09-01", + "premise: the orphan now carries a date") + + out, rc = _run("orphan-scan", "--prune", "--ledger", hist, after) + expect.num(out.count("ORPHAN CARRYING HISTORY"), 1, + "an orphan carrying history is reported as carrying it") + expect.num(out.count("last red 2026-09-01"), 1, + "and the history it would lose is printed with it") + expect.num(rc, 2, "the prune is refused") + expect.num(out.count("the catalogue is what this ledger is for"), 1, + "and it says why, rather than only that it refused") + expect.num(len([r for r in _rows(hist) if r[2] == "gone tomorrow"]), 1, + "premise: and the refusal removed NOTHING") + + +def test_a_part_that_skipped_is_unprunable_because_absence_is_not_removal(tmp_path, expect): + """The first version of `--prune` deleted a suite. + + One SKIP record put the part in `parts`, so every other row of that suite became an + orphan, and `--prune` removed them while reporting `not checked=0` and rc=0 -- the + most confident output the tool can produce. `not checked` protects a part the run does + not contain; a part CONTAINED BUT SKIPPED WHOLESALE fell in the gap between the two. + + The rule is deliberately broader than that case: a SKIP anywhere in the part means + some arm did not run, so the run cannot tell a deleted check from one skipped under a + name that does not match it. One skipped timing check blocks pruning that whole part, + which is the direction a deleting command should err in. + + The CONTROL is the half that matters: the same two rows must still be pruned when the + part's record is a PASS, or this is a tool that refuses to prune anything. + """ + ledger = _w(tmp_path, "l.tsv", "") + _run("merge", "--ledger", ledger, "--date", "2026-09-01", + _w(tmp_path, "full.log", + "RESULT\tdemo\tpart1\tarm one\tPASS\t\n" + "RESULT\tdemo\tpart1\tarm two\tPASS\t\n" + "RESULT\tdemo\tpart1\tthe whole thing\tPASS\t\nchecks run: 3\n")) + expect.num(len(_rows(ledger)), 3, "premise: the ledger holds all three rows") + skipped = _w(tmp_path, "skipped.log", + "RESULT\tdemo\tpart1\tthe whole thing\tSKIP\tno fixture on this box\n" + "checks run: 1\n") + + out, rc = _run("orphan-scan", "--ledger", ledger, skipped) + expect.num(out.count("unprunable: 2 row(s)"), 1, + "a row in a part that skipped is unprunable, not an orphan") + expect.num(out.count("orphans=0 (0 carrying history), unprunable=2"), 1, + "and the summary keeps the two apart") + expect.num(rc, 1, "it is still a finding, so the scan does not return success") + + _run("orphan-scan", "--prune", "--ledger", ledger, skipped) + expect.num(len(_rows(ledger)), 3, "--prune removes nothing from a part that skipped") + expect.num(_run("orphan-scan", "--prune", "--ledger", ledger, skipped)[0] + .count("the run did not exercise those checks"), 1, + "and it says so rather than declining silently") + + passed = _w(tmp_path, "pass.log", + "RESULT\tdemo\tpart1\tthe whole thing\tPASS\t\nchecks run: 1\n") + out, _ = _run("orphan-scan", "--prune", "--ledger", ledger, passed) + expect.num(out.count("removed 2 row(s)"), 1, + "control: the same rows ARE pruned when that part's record is a PASS") + expect.num(len(_rows(ledger)), 1, "control: and the ledger really is shorter") + + def test_the_gate_refuses_a_new_check_only_in_a_suite_it_covers(tmp_path, expect): """The suite restriction is the MEANING of `suites_not_covered`, not a softening. - Without it the gate refuses every check of all 250 uncovered suites and reddens the + Without it the gate refuses every check of every uncovered suite and reddens the whole matrix on its first run -- a gate somebody turns off within the week, which is the failure this issue family exists to prevent. It tightens on its own as suites are seeded. diff --git a/test/selftest/410-a-check-must-have-been-red.sh b/test/selftest/410-a-check-must-have-been-red.sh index f537f724..5a241310 100644 --- a/test/selftest/410-a-check-must-have-been-red.sh +++ b/test/selftest/410-a-check-must-have-been-red.sh @@ -228,6 +228,211 @@ printf 'RESULT\tdemo\tpart1\ta stable check\tPASS\t\nchecks run: 1\n' > "$_lw/re check "nor is one merely removed" \ "$(_led_run rename-scan --ledger "$_lw/ren.tsv" "$_lw/removed.log" | grep -c 'possible rename')" "0" +# ---- and what it IS, which nothing asked until #983 ------------------------- +# +# `rename-scan` pairs an appearance with a disappearance. An UNPAIRED +# disappearance -- a check deleted, or renamed in a run where nothing appeared -- +# printed as `vanished=N` and refused nothing. Measured on the committed ledger: +# two rows named checks that no longer existed, the census counted both, and the +# gate returned 0 on every run while the note scrolled past. +# +# SCOPED TO THE PARTS THE RUN CONTAINS, and the scope is REPORTED rather than +# assumed. A one-suite log has nothing to say about another suite's rows, and +# counting those as present is how a guard manufactures the confidence that the +# ledger was checked -- the shape of the whole issue family. + +: > "$_lw/orph.tsv" +printf 'RESULT\tdemo\tpart1\tstill here\tPASS\t\nRESULT\tdemo\tpart1\tgone tomorrow\tPASS\t\nRESULT\tdemo\tpartZ\telsewhere\tPASS\t\nchecks run: 3\n' > "$_lw/o_before.log" +printf 'RESULT\tdemo\tpart1\tstill here\tPASS\t\nchecks run: 1\n' > "$_lw/o_after.log" +_led_run merge --ledger "$_lw/orph.tsv" --date 2026-09-01 "$_lw/o_before.log" >/dev/null +check "premise: the ledger holds all three rows before the scan" \ + "$(wc -l < "$_lw/orph.tsv" | tr -d ' ')" "3" +check "premise: and the run the scan is given holds only one of them" \ + "$(grep -c '^RESULT' "$_lw/o_after.log")" "1" + +check "a ledger row no record in its own part matches is named an orphan" \ + "$(_led_run orphan-scan --ledger "$_lw/orph.tsv" "$_lw/o_after.log" \ + | grep -c 'orphan: demo part1 gone tomorrow')" "1" +check "and it is REFUSED, not merely printed" \ + "$(_led_rc orphan-scan --ledger "$_lw/orph.tsv" "$_lw/o_after.log")" "1" +check "the check the run still emits is not called an orphan" \ + "$(_led_run orphan-scan --ledger "$_lw/orph.tsv" "$_lw/o_after.log" | grep -c 'still here')" "0" + +# The half that decides whether this guard is honest. A row in a part the run does +# not contain is NOT an orphan -- the run cannot speak about it -- and saying so +# out loud is the difference between a scope and a blind spot. +check "a row in a part the run does not contain is not called an orphan" \ + "$(_led_run orphan-scan --ledger "$_lw/orph.tsv" "$_lw/o_after.log" | grep -c 'elsewhere')" "0" +check "and the scan states how many rows it could not speak about" \ + "$(_led_run orphan-scan --ledger "$_lw/orph.tsv" "$_lw/o_after.log" | grep -c 'not checked=1')" "1" +check "a run that emits every row in its parts is clean" \ + "$(_led_rc orphan-scan --ledger "$_lw/orph.tsv" "$_lw/o_before.log")" "0" +check "a before-log and an after-log together are refused, as rename-scan refuses them" \ + "$(_led_rc orphan-scan --ledger "$_lw/orph.tsv" "$_lw/o_before.log" "$_lw/o_after.log")" "2" + +# ---- pruning: historyless rows may go, a catalogue entry may not ------------ +# +# The catalogue of what has been seen red is the thing this ledger exists to be. +# Dropping an entry because the name moved is the exact loss `rename-scan` was +# written to prevent, so prune REFUSES on history rather than asking nicely. + +cp "$_lw/orph.tsv" "$_lw/prune.tsv" +check "premise: the orphan about to be pruned carries no history" \ + "$(awk -F'\t' '$3=="gone tomorrow"{print $4}' "$_lw/prune.tsv")" "never" +check "--prune removes a historyless orphan and says which" \ + "$(_led_run orphan-scan --prune --ledger "$_lw/prune.tsv" "$_lw/o_after.log" \ + | grep -c 'pruned: demo part1 gone tomorrow')" "1" +check "and the ledger is one row shorter afterwards" \ + "$(wc -l < "$_lw/prune.tsv" | tr -d ' ')" "2" +check "control: the row in the part the run never mentioned SURVIVES the prune" \ + "$(awk -F'\t' '$3=="elsewhere"' "$_lw/prune.tsv" | wc -l | tr -d ' ')" "1" +check "and a prune with nothing left to remove is clean, not an error" \ + "$(_led_rc orphan-scan --prune --ledger "$_lw/prune.tsv" "$_lw/o_after.log")" "0" + +# An orphan that has been seen red is the one case where deleting the row loses +# something no run can recreate. +: > "$_lw/hist.tsv" +printf 'RESULT\tdemo\tpart1\tstill here\tPASS\t\nRESULT\tdemo\tpart1\tgone tomorrow\tFAIL\t\nchecks run: 2\n' > "$_lw/h_before.log" +_led_run merge --reds-are-real --mutation "drop the guard" --ledger "$_lw/hist.tsv" \ + --date 2026-09-01 "$_lw/h_before.log" >/dev/null +check "premise: the orphan now carries a date and a mutation" \ + "$(awk -F'\t' '$3=="gone tomorrow"{print $4"/"$5}' "$_lw/hist.tsv")" "2026-09-01/drop the guard" +check "an orphan carrying history is reported as carrying it" \ + "$(_led_run orphan-scan --ledger "$_lw/hist.tsv" "$_lw/o_after.log" \ + | grep -c 'ORPHAN CARRYING HISTORY')" "1" +check "and the history it would lose is printed with it" \ + "$(_led_run orphan-scan --ledger "$_lw/hist.tsv" "$_lw/o_after.log" \ + | grep -c 'last red 2026-09-01')" "1" +check "--prune REFUSES the whole prune when any orphan carries history" \ + "$(_led_rc orphan-scan --prune --ledger "$_lw/hist.tsv" "$_lw/o_after.log")" "2" +check "premise: and the refusal removed NOTHING -- the row is still there" \ + "$(awk -F'\t' '$3=="gone tomorrow"' "$_lw/hist.tsv" | wc -l | tr -d ' ')" "1" +check "the refusal says why, rather than only that it refused" \ + "$(_led_run orphan-scan --prune --ledger "$_lw/hist.tsv" "$_lw/o_after.log" \ + | grep -c 'the catalogue is what this ledger is for')" "1" + +# ---- a part the run SKIPPED is not a part the run can speak about ------------ +# +# The first version of this deleted a suite. One SKIP record put the part in +# `parts`, so every other row of that suite became an orphan, and `--prune` removed +# them while reporting `not checked=0` and rc=0 -- the most confident output the +# tool can produce. `not checked` protects a part the run does not contain at all; +# a part CONTAINED BUT SKIPPED WHOLESALE fell in the gap between the two. +# +# Reported by @pgcolumnar-9b reviewing this change, on a three-row fixture for +# `analyze_differential`, whose run on PG17 is a single SKIP. Nine suites skip +# wholesale on PG17 and `suites_not_covered` is 250, so seeding any one of them +# would have armed it. +# +# THE RULE IS BROADER THAN THAT CASE DELIBERATELY: a SKIP anywhere in the part +# means some arm did not run, so the run cannot tell "this row's check was deleted" +# from "this row's check was skipped under a name that does not match it" -- #994's +# defect at suite granularity. One skipped timing check therefore blocks pruning +# that whole part, and that is the direction a deleting command should err in. + +: > "$_lw/skp.tsv" +printf 'RESULT\tdemo\tpart1\tarm one\tPASS\t\nRESULT\tdemo\tpart1\tarm two\tPASS\t\nRESULT\tdemo\tpart1\tthe whole thing\tPASS\t\nchecks run: 3\n' > "$_lw/sk_full.log" +printf 'RESULT\tdemo\tpart1\tthe whole thing\tSKIP\tno fixture on this box\nchecks run: 1\n' > "$_lw/sk_skipped.log" +_led_run merge --ledger "$_lw/skp.tsv" --date 2026-09-01 "$_lw/sk_full.log" >/dev/null +check "premise: the ledger holds all three of that part's rows" \ + "$(wc -l < "$_lw/skp.tsv" | tr -d ' ')" "3" +check "premise: and the skipped run emits exactly one of them, as a SKIP" \ + "$(awk -F'\t' '$5=="SKIP"' "$_lw/sk_skipped.log" | wc -l | tr -d ' ')" "1" + +check "a row in a part that SKIPPED is reported as unprunable, not as an orphan" \ + "$(_led_run orphan-scan --ledger "$_lw/skp.tsv" "$_lw/sk_skipped.log" \ + | grep -c 'unprunable: 2 row(s)')" "1" +check "and the summary keeps the two apart, so a zero orphan count is not a clean bill" \ + "$(_led_run orphan-scan --ledger "$_lw/skp.tsv" "$_lw/sk_skipped.log" \ + | grep -c 'orphans=0 (0 carrying history), unprunable=2')" "1" +check "it is still a finding, so the scan does not return success" \ + "$(_led_rc orphan-scan --ledger "$_lw/skp.tsv" "$_lw/sk_skipped.log")" "1" + +_led_run orphan-scan --prune --ledger "$_lw/skp.tsv" "$_lw/sk_skipped.log" >/dev/null +check "--prune removes NOTHING from a part that skipped" \ + "$(wc -l < "$_lw/skp.tsv" | tr -d ' ')" "3" +check "and it says so rather than declining silently" \ + "$(_led_run orphan-scan --prune --ledger "$_lw/skp.tsv" "$_lw/sk_skipped.log" \ + | grep -c 'the run did not exercise those checks')" "1" + +# CONTROL. Without this the arms above are satisfied by a tool that refuses to prune +# anything at all, which is the failure mode of every over-broad guard. +printf 'RESULT\tdemo\tpart1\tthe whole thing\tPASS\t\nchecks run: 1\n' > "$_lw/sk_pass.log" +check "control: the same two rows ARE pruned when that part's record is a PASS" \ + "$(_led_run orphan-scan --prune --ledger "$_lw/skp.tsv" "$_lw/sk_pass.log" \ + | grep -c 'removed 2 row(s)')" "1" +check "control: and the ledger really is shorter afterwards" \ + "$(wc -l < "$_lw/skp.tsv" | tr -d ' ')" "1" + +# Every row must land in exactly one of the four categories, or the classification +# itself lost one -- which is the failure this whole tool exists to report. +check "the four categories account for every ledger row" \ + "$(_led_run orphan-scan --ledger "$_lw/orph.tsv" "$_lw/o_after.log" \ + | grep -c 'classification lost rows')" "0" + +# ---- and the exit code cannot say "done" when nothing was done -------------- +# +# The first version returned 0 from `--prune` whenever it pruned nothing, including +# when everything it found was unprunable. A caller that scans, sees 1, re-runs with +# `--prune` and sees 0 reads "it pruned them" -- when nothing was pruned and nothing +# could be. Prose covers a human; a script sees only the code. Reported by @jdatcmd +# in review. + +: > "$_lw/rc.tsv" +_led_run merge --ledger "$_lw/rc.tsv" --date 2026-09-01 "$_lw/sk_full.log" >/dev/null +check "premise: that part has rows the skipped run cannot speak for" \ + "$(wc -l < "$_lw/rc.tsv" | tr -d ' ')" "3" +check "the scan reports a finding on a skipped part" \ + "$(_led_rc orphan-scan --ledger "$_lw/rc.tsv" "$_lw/sk_skipped.log")" "1" +check "and --prune does NOT turn that 1 into a 0, because it pruned nothing" \ + "$(_led_rc orphan-scan --prune --ledger "$_lw/rc.tsv" "$_lw/sk_skipped.log")" "1" +check "premise: and it really pruned nothing -- the rows are all still there" \ + "$(wc -l < "$_lw/rc.tsv" | tr -d ' ')" "3" +# CONTROL: 0 still means 0. A prune with nothing outstanding must report success, or +# the code says "work remains" forever and nobody can use it in a script either. +check "control: a prune that leaves nothing outstanding returns 0" \ + "$(_led_rc orphan-scan --prune --ledger "$_lw/rc.tsv" "$_lw/sk_pass.log")" "0" +check "control: and that one did prune, so 0 is not a refusal in disguise" \ + "$(wc -l < "$_lw/rc.tsv" | tr -d ' ')" "1" + +# WHY THIS REPORTS AND DOES NOT GATE -- pinned to the PRECONDITION, not to one +# instance of it. +# +# This arm used to grep `340` for `check_skip "the unreadable-source refusal"`: one +# skip standing in for three named arms, so on a box with no non-root user three +# committed rows had no record and were not removed checks. #998 removed that line, +# which SATISFIED the premise rather than breaking it -- and the arm as written would +# then have failed. Worse, it would have failed in `main`: the two PRs compose with a +# conflict only in the budget file, so nothing would have presented a marker to read. +# Found by @pgcolumnar-9b, by composing the merge rather than reasoning about it. +# +# THE PRECONDITION IS NOT "340 HAS THAT LINE". It is that somewhere in the corpus a +# skip still cannot be matched to the arms it stands in for -- while that holds, an +# absent record does not reliably mean a removed check, and a gate refusing on absence +# would redden a correct run. #998's sweep is the authority on that number, so this +# reads it instead of re-deriving it: a second implementation of one count is how two +# numbers come to disagree, which is the defect this file exists to catch. +# +# AND IT RETIRES ITSELF. The day `interpolated` and `armless` both reach zero this arm +# fails, and the fix for that failure is to arm the gate -- which is the direction the +# whole issue wants to go, stated as a check rather than as a comment somebody has to +# remember to re-read. +_orph_tool="$PGC_TESTDIR/../.github/scripts/skip-loop-arms.py" +check "premise: the skip-loop sweep is present, so its counts can be read" \ + "$([ -r "$_orph_tool" ] && echo yes || echo no)" "yes" +_orph_sweep="$(python3 "$_orph_tool" "$PGC_TESTDIR" 2>&1)" || _orph_sweep="TOOL FAILED" +check "premise: and it reported all four of its categories, so a zero is a measurement" \ + "$(printf '%s\n' "$_orph_sweep" | grep -cE '^(loops|compared|interpolated|armless) [0-9]+$')" "4" +_orph_i="$(printf '%s\n' "$_orph_sweep" | sed -n 's/^interpolated \([0-9]*\)$/\1/p')" +_orph_a="$(printf '%s\n' "$_orph_sweep" | sed -n 's/^armless \([0-9]*\)$/\1/p')" +check "the orphan scan stays a REPORT while any skip loop cannot be compared to its arms" \ + "$({ [ "${_orph_i:-0}" -gt 0 ] || [ "${_orph_a:-0}" -gt 0 ]; } && echo "not yet armable" || echo "armable: arm the gate")" \ + "not yet armable" +unset _orph_tool _orph_sweep _orph_i _orph_a +check "premise: and the two arms it stands in for are still named in the ledger" \ + "$(grep -cP '^harness_selftest\t340-the-binary-must-be-built-from\tpremise: the unprivileged' \ + "$_ledger")" "2" + # ---- the gate refuses a check the ledger has never seen --------------------- : > "$_lw/g.tsv" @@ -384,7 +589,7 @@ check "but the gate says so, so the skip is visible rather than silent" \ # ---- the gate cannot refuse a check in a suite it has never seen ------------- # # The suite restriction is the MEANING of suites_not_covered, not a softening of -# the refusal. Without it the gate refuses every check of all 250 uncovered +# the refusal. Without it the gate refuses every check of every uncovered # suites and reddens the whole matrix on its first run -- a gate somebody turns # off within the week, which is the failure this issue family exists to prevent. #