test: a log can now say which tree it came from (#1073)

[OffgridwithJD]

Closes #1073.

The hole

orphan-scan refuses a ledger row that no record in its own part matches. A row whose check was ADDED after the log was written produces exactly that signal, and nothing in a RESULT record dates it against a tree. A stale log and a genuinely deleted check are indistinguishable, and the tool cannot close that from inside.

The issue measured it: replaying a log from one tree against the ledger one commit later reported two orphans, both of which were checks that tree had just gained — one step from a defect report against a tool merged an hour earlier.

Nothing needed inventing

test/lib.sh already writes -- source: <fingerprint> into every log, from the one implementation in test/pgc_fingerprint.py. It needed reading.

--expect-source FINGERPRINT refuses a log that does not carry the one the caller names — on orphan-scan and on merge. On merge it matters more: a misread orphan is recoverable by looking again, and a stale log stamped into the ledger persists.

Reproduced end to end, with the control that makes the refusal mean something

without the flag        orphan: demo part1 one added since       rc=1
                        (reported DELETED; it was ADDED)

--expect-source <new>   came from a different tree: it names source aaaa1111bbbb,
                        and you expected c9e65b1b35ba             rc=2

a log FROM that tree    orphan: demo part1 one added since       rc=1
                        (tree confirmed, so the orphan is a REAL finding)

a log naming none       names no source fingerprint, so it cannot be shown to come
                        from the tree you named                   rc=2

The third line is the one that stops this being a flag that just suppresses everything. The fourth is the vacuity: "does not disagree" is how an opt-in check reports success having asked nothing — the shape #1032 and #965 both turned out to be.

The two decisions the issue said were not mechanical

Where the expectation comes from. Opt-in. A hand caller may not know the build its log came from, and a flag that refused every hand invocation is a flag nobody passes. Today's runner is safe by construction — it passes the logs from the run it has just finished — and that is an argument for stating the guarantee, not for assuming the next caller inherits it.

Whether other subcommands want it. merge yes, for the reason above. gate no: it asks whether the run has a check the ledger has never seen, and it is the runner's own logs by construction there too — but gate never writes, so a stale log costs a wrong verdict rather than a wrong row. Worth revisiting if a second caller appears; not worth a flag nobody passes today.

The runner now passes it, from the same stamp it wrote rather than recomputed — two implementations of one fingerprint drift, which is the defect pgc_fingerprint.py exists to have ended. Verified they agree on this tree:

lib.sh  : c9e65b1b35ba
module  : c9e65b1b35ba

So the flag is exercised in production, not only in its own arms — which is the state orphan-scan itself was in until #983: written, tested, and unable to fire on anybody's change.

Both harnesses, independently

Thirteen arms in test/selftest/410, nine in test/pytest/test_mutation_ledger.py, each with its own fixture and its own names.

harness_selftest   985 checks, 0 failed
pytest guard half  347 passed, 907 checks, 0 failed

Every refusal arm greps its message, not just its status. --expect-source did not exist before this change, so argparse exited 2 for an unknown flag and the status-only arms passed against the absent feature — measured, before implementing: 5 red of 13, and the three rc-only refusals green.

A fixture error worth recording

The first end-to-end run used 0ld7reefaaaa as a fingerprint. That is not hexadecimal, so the tool reported "names no source fingerprint" where I expected a mismatch, and for a moment it looked like the comparison was broken. The format check was right and the fixture was wrong — the second run used real 12-hex-digit values and both paths behaved.

🤖 Generated with Claude Code

https://claude.ai/code/session_012RSw4qMHS7ByE7PY8Ns4cs

[jdatcmd]

Approving 8653f43. The four-state table is the right shape and the "names none is refused, not skipped" arm is the one that makes it worth having. One finding, not blocking, and it is your own thesis pointed at the caller you added.

Verified at source, not from the body

run_all_versions.sh:1602   _orph_fp="$( . "$builddir/test/lib.sh"; pgc_source_fingerprint "$builddir" )"
run_all_versions.sh:1617   --expect-source "$_orph_fp"
pgc_ledger.py:491          if got is None:  raise  ... "names no source fingerprint"
pgc_ledger.py:498          if got != want:  raise  ... "came from a different tree"

I also checked something that could have been a real defect and is not: the log's stamp is computed over PGC_SRCDIR, the runner's expectation over $builddir. lib.sh:238 sets PGC_SRCDIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" and lib.sh lives at $builddir/test/lib.sh, so they are the same directory. Worth stating because two different variables naming one path is how that stops being true later.

The finding: an empty expectation disables the check silently

want = (getattr(args, "expect_source", None) or "").strip()
if not want:
    return

Correct for a hand caller who omits the flag. But the runner always passes the flag, and the value can be empty:

pgc_source_fingerprint() {
    py="$(_pgc_fp_python)" || true
    [ -n "$py" ] || { _pgc_fp_warn_once; printf ''; return 0; }
    out="$("$py" "$(_pgc_fp_module)" fingerprint "${1:-.}" 2>/dev/null)"
    [ "$rc" -eq 0 ] || { printf ''; return 0; }
}

Both failure paths return empty with status 0. So if the module errors — missing from the build dir, or a transient failure — the runner passes --expect-source "", the tool returns on the first line, and the guard does nothing, while the comment three lines above says the guarantee is now "stated rather than assumed".

That is the sentence your own PR body uses about the opposite case:

"does not disagree" is how an opt-in check reports success having asked nothing

It is the same failure, moved from the log to the expectation.

Why it is not blocking. In that degraded state lib.sh cannot stamp the log either, so _log_source_fingerprint returns None and the log would be refused — if the check ran at all. The two halves degrade together, so the outcome is no protection rather than a wrong answer. And it needs python3 present but the module broken, which is narrow.

What I would do, one line in the runner rather than in the tool, since the tool's opt-in semantics are right:

[ -n "$_orph_fp" ] || { echo "  PG$major: could not compute a source fingerprint, so the"
                        echo "  orphan scan cannot be shown to be reading this tree's logs"; verfail=1; }

Loud rather than silent is the whole argument of the PR. Your call whether it belongs here or in a follow-up — it is a pre-existing property of pgc_source_fingerprint that this change is the first caller to depend on.

The composition with #1115

Already checked and reported: the pgc_ledger.py conflict is positional only — both sides add a new top-level definition above cmd_merge, neither edits the other. Your resolution keeping both is what I would have done, and your composed run covers the shell side I could not:

harness_selftest  1033 checks, 0 failed, part 410 206 arms + part 520 16 arms
gate              rc=0, census 1334 agrees, new this run=0
both features on one merge: --expect-source accepts the matching log AND the
                            subset warning fires on it; a foreign log is refused

Your probe error

my first attempt passed --target, which is #1114's and not in #1117, so argparse refused the whole merge and my summary read rows written: 0 because I had captured grep's status rather than merge's.

Two stacked, and the second is the one that hid the first: a 0 that came from the wrong $? looks exactly like a 0 that came from the right one. Same family as your awk-quoting zero and my grep -E '\t' that matched a literal t. The defence that works for all of them is to make the probe produce a NON-zero on a case where the thing is present, before believing any zero.

[OffgridwithJD]

9c72c5e. Your finding is right and I took it here rather than as a follow-up, because it is my own thesis pointed at the caller I added.

Verified before fixing

pgc_source_fingerprint /nonexistent/tree   -> value=[] rc=0
PATH=/nonexistent pgc_source_fingerprint   -> value=[] rc=0
pgc_source_fingerprint <healthy tree>      -> value=[c9e65b1b35ba] rc=0

And the consequence, with the runner's would-be argument:

orphan-scan --expect-source "" <a log from another tree>
  orphan: demo  part1  gone (claims 18, no history)
  rc=1

The guard did nothing, and said nothing about doing nothing. "Does not disagree" satisfying an opt-in check — moved from the log to the expectation, three lines under a comment claiming the guarantee is "stated rather than assumed".

The fix is in the runner, not the tool

The tool's opt-in rule is right for a hand caller and I did not touch it. The runner now refuses:

if [ -z "$_orph_fp" ]; then
    echo "  PG$major: the source fingerprint could not be computed, so the"
    echo "  orphan scan cannot be told which tree these logs came from. That"
    echo "  is the freshness machinery being unavailable, not a clean scan."
    _orph_broken=1
fi
...
for _orph_log in ${_orph_fp:+$_led_logs}; do

Two halves: the flag is set and the loop does not run, so it cannot report clean either way.

I kept your "they degrade together" observation in the comment, as the reason this is belt-and-braces rather than the only thing standing — but "it happens to be covered elsewhere" is how a guard stops being one.

Three arms, and the premise is the part I would not have written without your message:

PASS  the runner refuses the scan when it cannot name the tree
PASS  and the scan does not run at all without an expectation, so it cannot report clean
PASS  premise: pgc_source_fingerprint really does return empty with status 0

Removing the guard reddens the first two.

Re-seeded, because three arms in a covered part is three rows

PG15..PG19   rc=0   records=1036   FAILs=0
namehash     e7dc4d1a1861   verdicthash  193222e22fa1   identical across all five
ledger       1345 rows, all 15;16;17;18;19, 0 duplicate keys
census       re-derived 1337
gate         rc=0

On PGC_SRCDIR vs $builddir

Thank you for checking that and saying it was not a defect. lib.sh:238 derives PGC_SRCDIR from BASH_SOURCE/.. and the file lives at $builddir/test/lib.sh, so they are the same directory — and your closing line is the useful half: two variables naming one directory is how that stops being true later. If this ever grows a third caller I would rather it took the path explicitly than inherited the coincidence.

[jdatcmd]

Re-approving at 25722a0. The fail-open is closed properly and the union holds.

The fix is belt and braces, correctly

if [ -z "$_orph_fp" ]; then ... _orph_broken=1; fi
for _orph_log in ${_orph_fp:+$_led_logs}; do

${_orph_fp:+...} expands to nothing when the fingerprint is empty, so the loop cannot run and the major is already failed. It cannot report clean by either route, which is more than I asked for — I only suggested the flag.

And you kept the right reason in the comment:

"it happens to be covered elsewhere" is how a guard stops being one

That is the better statement of my own point. I had framed the degrade-together property as the reason it was not blocking; you framed it as the reason it is not sufficient.

Union re-read against the new base

main bcf1030      1338 rows, census 1330
#1117             1350 rows
  main keys lost  0
  keys added      12    all harness_selftest/410, all 15;16;17;18;19
  duplicate keys  0     non-6-field rows 0
  census recipe   1342  stated 1342

12 is your 9 plus the 3 arms for the fix, which reconciles.

One nit, not blocking, and it predates the fix

The comment above the loop still reads:

Read from the stamp this runner wrote above, not recomputed: two implementations of one fingerprint drift

but line 1602 is pgc_source_fingerprint "$builddir", which recomputes — through the same implementation, so the anti-drift property you are claiming does hold, but not by the mechanism the sentence describes.

It matters a little more than wording. Recomputing means that if the tree changed between the log being stamped and the scan running, the two disagree and the scan refuses its own logs. That is arguably the better behaviour — and it is not hypothetical, since I had a tree change under a five-major loop today — but a reader reasoning from "read from the stamp" would predict the opposite. Either drop "not recomputed", or read the stamp; the second is a behaviour change and I would not make it here.

[jdatcmd]

Retracting my approval — CI at 25722a0 is red for a reason I should have predicted and did not, and it is the feature refusing the runner's own logs.

What CI says

ledger integrity failure: .../audit.log names no source fingerprint, so it cannot be
shown to come from the tree you named (c9e65b1b35ba)

and the same for thirteen more. Fourteen distinct suites:

audit  concurrency  decode_interrupts  hilbert_curve  objstore_stash_recovery
phase2  phase3  phase4  phase5  phase6  smoke  unique_conc  update_conc  wal_envelope

Fourteen of 258. The orphan scan then fails for the whole major.

Why, and it is not a bug in your refusal

The stamp is written inside pgc_setup — lib.sh:302 sets _pgc_fresh_current and the three -- source: spellings are all in that function. A suite that does not call pgc_setup produces a log with no fingerprint, and some of these reference lib.sh without calling it:

audit                    lib.sh=4  pgc_setup=0
phase2                   lib.sh=4  pgc_setup=0
objstore_stash_recovery  lib.sh=1  pgc_setup=0
concurrency              lib.sh=6  pgc_setup=1

So the premise in the comment is the thing that is wrong:

This caller is safe by construction -- these are the logs from the run it has just finished

They are the runner's own logs, and fourteen of them cannot satisfy the flag. "Safe by construction" was true of provenance and not of stamping, and those are different properties.

This is the same population that bit #1109 — suites carrying their own harness — and I should have connected them when I approved. That one is on me.

Three ways out, and I would take the third

1. Skip logs with no stamp. This is the fail-open you just closed, reintroduced for a different reason. No.
2. Make the fourteen call pgc_setup. Large, touches suites with their own harness for reasons unrelated to this, and #1109 exists because that population is deliberate.
3. Stamp in the runner. It already owns every log — it redirects each suite into $builddir/${s}.log — so writing the -- source: line there covers all 258 at one site, makes "safe by construction" literally true, and needs nothing from the suites. The suites that already stamp would carry it twice; _log_source_fingerprint returns the first match, so a runner-written line placed first is the one read.

3 also removes the coupling that made this surprising: today a log's provenance depends on which harness the suite chose.

What still stands

The union, re-read against bcf1030, is sound: 1350 rows, 0 main keys lost, 12 added all in part 410 at 15;16;17;18;19, 0 duplicates, census 1342 matching the recipe. The empty-fingerprint fix is right and I would keep it exactly as written. Only the precondition needs closing.

Sorry for the churn — you asked me to check the unions and I checked those carefully while missing the thing CI then found in one run.

[OffgridwithJD]

9803ddb. Option 3 taken, and a full one-major matrix run alone says it works.

suite FAILs                               0
command not found                         0
not the one this tree built               0
names no source fingerprint               0
came from a different tree                0
source fingerprint could not be computed  0
suites that ran: 256 of 258 (skipped: 2, incomplete: 0)

The third line is the claim you asked me to hold for, and the fifth is what makes it mean something: the expectation was non-empty, so the flag did work rather than being skipped by its own opt-in rule.

What changed

The runner stamps: one helper, three sites, nothing asked of any suite. Written first, suites append — _log_source_fingerprint returns the first match, so a suite that also stamps yields one answer, and a > where a >> belongs would erase the line just written. The helper writes nothing at all when it has no fingerprint, so a -- source: that names no tree cannot exist.

Three defects found implementing it, all mine

Definition order. I defined pgc_stamp_log at 1128 and called it at 762. bash reads top to bottom, so the matrix printed command not found once per suite and the loop ran anyway — every log unstamped while the run looked normal. No behavioural arm over the helper can see that (the helper is fine; the file is wrong), so an arm compares the definition's line number to the first call's.

"Twenty" was counted on mentions. grep -l pgc_setup says 20; six of the fourteen name it only in a comment saying they skip it deliberately. The real count is 28, and counting the mention would have put six suites on the wrong side of the claim the comment itself makes. The counting method is now recorded beside the number.

selftest/390 re-anchored, not weakened. It pinned the skip branch on the literal >"$builddir/${s}.log", which had to become >>. Its subject — that the branch declining to dispatch is the thing that records it — is unchanged.

And one in my rebase tooling, which is the one I would flag hardest

My resolver unioned check_ledger.tsv with sort -u. That deduplicates whole lines, so the two rows I back-filled in #1114 came through twice — once as main's 2026-09-18 / <mutation> and once as my branch's stale 2026-09-12 / -:

dupkeys=2
  native_join_runtime_filter  default plan has runtime coordinator  2026-09-12  -
  native_join_runtime_filter  default plan has runtime coordinator  2026-09-18  <mutation>

This is your MODIFIED-versus-LOST warning pointed the other way: in a diff it hides a change, in a union it duplicates a key. A ledger row is not a line — last-red moves forward only, mutations accumulate, majors are a set — so the union is now by key with those semantics, and the resolver refuses outright if it ever produces a duplicate. History preserved:

default plan has runtime coordinator  last-red=2026-09-18  mutation=enable_join_runtime_filter boot value true -> false
join runtime filter defaults on       last-red=2026-09-18  mutation=enable_join_runtime_filter boot value true -> false

Verification

ledger            1376 rows, 0 duplicate keys, all at 15;16;17;18;19
census            1368 == budget 1368
guard_tests       374 by collection
harness_selftest  1067 checks, 0 failed
pytest guard half 374 passed, 1003 checks, 0 failed
gate              rc=0 against a real PG18 log

All seven new stamping arms carry rows on five majors, seeded from one frozen snapshot: records 1048, namehash 85d97e787e2a and verdicthash 4a2a406245b5 identical across all five legs.

One observation I could not chase

The matrix gate printed this run observed 18, unknown — some log in the batch carries major=unknown. It is not harness_selftest (0 such records on PG15 and PG18), and the gate itself says it is benign. The builddir is removed on success, so I lost the evidence before I could name the suite. Filing nothing on a claim I cannot support; noting it here in case it turns up on your side.

[jdatcmd]

Approving 9803ddb. Ledger re-read by key, the definition-order arm's premise checked against the pushed file, and I chased the one lead you could not.

The union, after the sort -u fix

main 6eead59   1357 rows
#1117          1376 rows
  main keys lost     0
  keys added        19     (your 12 plus the 7 new arms)
  DUPLICATE KEYS     0
  non-6-field rows   0     C-sorted yes
  census recipe   1368     stated 1368

And the back-fill survived the re-resolution:

default plan has runtime coordinator | 2026-09-18 | ...boot value true -> false
join runtime filter defaults on      | 2026-09-18 | ...boot value true -> false

Your sort -u finding is the better half of this exchange and I want to state why. I gave you "whole-line comparison cannot tell MODIFIED from LOST" as a diff problem. You found the same root cause inverted: in a union, whole-line dedup keeps both versions of a modified row and corrupts the key space. A line is not a row — and the failure looked clean from the outside. No conflict, no marker, a plausible row count. Only a dupkeys print after every resolve caught it.

That generalises past git: any set operation over a table keyed on a subset of the line is wrong if it operates on the line.

The definition-order arm

Premise holds on the pushed file:

pgc_stamp_log defined at 762
call sites     794, 828, 838

All three after. And the arm is right that no behavioural arm can reach it — the helper is correct in isolation, the file is wrong — so comparing line numbers is the only instrument that sees it. command not found not being fatal, with the loop running anyway, is the part worth having in the comment.

Your unchased lead: I can name it

the gate printed this run observed 18, unknown, so some log in the batch carries major=unknown ... the builddir is removed on success, so the evidence was gone

It is not one suite, it is eight, and every record they emit is affected. Found statically — sources lib.sh (so pgc_record exists), never calls pgc_setup (which is what sets PGC_MAJOR), never sets PGC_MAJOR itself — then confirmed by running each:

smoke                     9 records,   9 unknown
audit                    31 records,  31 unknown
objstore_stash_recovery  17 records,  17 unknown
phase2                   42 records,  42 unknown
phase3                   32 records,  32 unknown
phase4                   38 records,  38 unknown
phase5                   36 records,  36 unknown
phase6                   43 records,  43 unknown
----
TOTAL                   248 records, 248 unknown

(pg_upgrade is a ninth by the same static test but produced 0 records here — it exits 2 without a second major installed.)

This is exactly what #1109 fixed for concurrency, unique_conc and update_conc, one line each. The same fix applies. Filing it separately with the measurement, since it is not yours and predates this branch — your change only made it visible, which is the change working.

Nothing outstanding on this PR.

[OffgridwithJD]

Adversarial pass over my own branch at 9803ddb, since it is approved and about to merge. Six attacks, five clean, and one real gap in my verification that I want on the record before it lands.

The gap: the not-dispatched branch was never exercised

pgc_stamp_log is called at three sites. The matrix I used as proof exercised two of them. The third — the branch that forges a log for a suite it declines to dispatch — did not run:

not dispatched=0

So "verified on a full matrix" covers two of three call sites, and I should not have written it without saying which. Exercised directly instead, with the runner's own classifiers extracted rather than reimplemented:

                   classify(66)  shows_accounting  shows_any
STAMPED    SKIP          no                no
UNSTAMPED  SKIP          no                no

Identical, so prepending the stamp does not change how a not-dispatched suite reads. That is evidence, not coverage — nothing will notice if it regresses. I am not adding an arm for it now because each new check in a covered part costs a five-major reseed and this one is static and already pinned by and stamps at every site that writes a suite log → 3. Flagging it so the decision is visible rather than absent.

My first attempt at this probe printed empty for both rows, because those three classifiers live in run_all_versions.sh, not lib.sh, so sourcing lib.sh defined none of them. Empty == empty would have read as "no difference" and been wrong for the reason the arm exists. The numbers above come from extracting the function bodies and asserting the extraction found three.

Five attacks that came back clean

Does anything read a suite log's first line? No — zero hits for head -1/sed -n 1p over test/*.sh and test/selftest/*.sh. A stamp prepended to every log would shift what such a reader sees, and there is no such reader.

Does anything count a log's lines as a measure? No.

Does ${_orph_fp:+$_led_logs} word-split correctly with several logs? Yes:

fp set   -> 3 iterations (want 3)
fp empty -> 0 iterations (want 0)

Does _orph_broken=1 actually fail the major? Yes — run_all_versions.sh:1688 tests it and sets verfail=1 at 1691.

A fingerprint that is not hexadecimal? Fails closed:

'c9e65b1b35ba' -> c9e65b1b35ba
'abc.def'      -> None
'a*b'          -> None
'../etc'       -> None
''             -> None

None means "names no source fingerprint", which is refused rather than accepted. So a malformed stamp cannot vouch for a tree.

Can two subshells race on the stamp? No — each writes its own $builddir/<suite>.log, and distinct paths cannot collide.

What the matrix does prove

suite FAILs                               0
command not found                         0
not the one this tree built               0
names no source fingerprint               0
came from a different tree                0
source fingerprint could not be computed  0
suites that ran: 256 of 258

The third line is the property; the fifth is what stops the third being vacuous, since an empty expectation would have made the check skip itself.

Standing corrections to my own earlier claims on this PR

• "safe by construction" was true of provenance, not stamping — jdatcmd's finding, and the reason this PR changed shape entirely.
• "twenty suites never call pgc_setup" was counted on mentions; six of the fourteen name it only in a comment saying they skip it. The real number is 28.
• my rebase resolver unioned the ledger with sort -u, which deduplicates lines, so two rows modified on main came through twice. Union is by key now, and the resolver refuses rather than writing a duplicate.

I cannot approve my own work and am not asking anyone to re-approve on the strength of this. It is here so the one untested path is stated by me rather than discovered later.