The pytest harness counts assertions but records nothing, so no count can be reconciled

[jdatcmd]

The shell harness records every check outcome as a machine-readable line, and counting a check is the same operation as recording it — pgc_record does both, and there is no code path that does either alone. pgc_reconcile_records then validates the schema and reconciles the record count against checks run:.

The pytest harness has no equivalent, and after #923 the gap is explicit rather than hidden.

What exists today

Expect._counted() increments a per-test counter. report.user_properties carries exactly one structured value, pgc_unrunnable, and pytest_terminal_summary prints the unrunnable count. Nothing emits a per-assertion record, so there is nothing to reconcile a count against.

Concretely, the pytest layer cannot answer any of these:

• which named assertion produced which outcome,
• what its verdict and reason were,
• whether a name carrying a tab or newline would corrupt the stream,
• whether a conditional SKIP was counted,
• whether the session's totals reconcile against the records.

Every one of those is answered on the shell side, and each was found to be answerable only because the shell has records to check.

Why this is filed rather than done in #923

#923 originally carried test_check_results_are_machine_readable.py, which reached the property by sourcing the real test/lib.sh and extracting pgc_reconcile_records from run_all_versions.sh. @linuxhikerpm blocked it under the harness-independence rule in CONTEXT.md, correctly: the rule is already on main, and a file that has not landed is a new violation rather than pre-existing debt. The owner's ruling on the same question was "neither rule wins. each harness is independent and should only parallel test functionality."

A twin that drives the other harness agrees with it by construction and can never report it wrong, so the file was deleted rather than repaired — the precedent being #927, where the shell part whose subject was a python module's source text was deleted for the same reason.

The remedy the review asks for is a feature, not a port. There is no per-assertion record stream in pytest to write a native test against; it has to be built first.

What "parallel in functionality" means here

Not a copy of the shell's format. The pytest harness should reach the same property through Python's own reporting interface — pytest_runtest_logreport, user_properties, and the junitxml those feed — so that:

1. every counted assertion produces exactly one structured record, in one operation, with no path that does either alone;
2. each record carries enough to identify the assertion, its verdict and its reason;
3. field separators cannot be smuggled in through a name or a reason;
4. an assertion that declined to run is counted like any other outcome;
5. the session total reconciles against the records, and the reconciliation can fail.

Point 1 is the one that matters: it is the property #917 exists to establish, and on the shell side it was reached by making counting and recording literally the same call.

Acceptance

• No file under test/pytest/ reads, sources, executes or derives from any file in test/*.sh or test/selftest/*.sh. The arm for this should be derived, not a filename list.
• Each of the five properties above has an arm with a removal proof: name the mutation, apply it, show which arm reddens.
• A reconciliation that cannot fail is not acceptance. The shell side shipped one twice — inputs == sum(buckets) that could not go red — and both were caught only by mutating them.

Related

• Phase 3 of #858: check results are prose, so a named check cannot be cited mechanically #917, test: count a check and record it in one operation (#917) #923 — the shell half, and the review that produced this.
• docs: the two test harnesses are parallel, not coupled #932 — the harness-independence rule and its debt inventory.
• test/pytest: a broad raises must name a SQLSTATE (#432) #927 — the same call made in the other direction.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Unuuvh3fRR67SceiGpfeeK