Skip to content

The matrix prints two different "accounted" totals in one report (237 and 235) #928

Description

@OffgridwithJD

One matrix report prints two different answers to "how many suites accounted for their checks", three lines apart, and a reader cannot tell which is the claim.

Measured on a full test/run_all_versions.sh for PG 17 on /usr/local/pg17a, tree clean, under the lock, with ci.yml's environment (PGC_SKIP_TIMING=1 PGC_JOBS=4 PGC_REQUIRE_ISOLATION=1):

  accounting reconciliation: inputs=239 | both=239, declared only=0, accounted only=0 | sum=239
  population reconciliation: registered=251 | accounted=237, not dispatched=4, known debt=10, unaccounted=0 | sum=251
  suites that ran: 242 of 251 (skipped: 9, incomplete: 0)
  of those, 235 accounted for their checks and 7 did not

237 and 235. Both describe suites that accounted for their checks, in the same report, and they differ by exactly 2.

Why

The two lines derive from different readers:

  • the population reconciliation counts accounted with the wide reader, pgc_log_shows_any_accounting;
  • the breakdown line derives _acc_ran from _acc_observed, which is built with the narrow reader, pgc_log_shows_accounting.

The narrow reader matches lib.sh's accounting: line. The wide one also recognises a suite that accounts by its own mechanism. The gap is therefore the suites that keep a private tally instead of lib.sh's — bench_guards and docs_style, which is exactly 2, and which I had measured independently from the other direction earlier today: of the twelve registered suites that never call pgc_summary, exactly those two maintain a tally of their own.

So of those, 235 accounted ... and 7 did not overstates the debt by 2. The figure a reader acts on is the second one, because it is the one phrased as a problem.

Why it matters more than a cosmetic mismatch

The breakdown line exists to stop an overcount — it was added because counting a suite as having "run" when it never accounted for its checks was hiding exactly this class of gap. Deriving it from the narrower reader reintroduces a smaller version of the same overcount, one level down, in the line added to close it.

Related, in the same comment

test/run_all_versions.sh, in the comment immediately above _acc_ran, says:

Ten registered suites exit 0 having never called pgc_summary

That conflates two different populations. Measured on this tree: twelve registered suites never call pgc_summary; ten of those keep no tally at all. The two the sentence drops are bench_guards and docs_style — precisely the two that make the numbers above disagree. Both halves of the sentence are true of something; neither is true of what it says.

Suggested shape of a fix

Derive both lines from one reader, or say in the line itself which question it answers. If the breakdown is meant to be "accounted via lib.sh's accounting", the remaining suites are not a debt, they are a different mechanism — and naming them as such would make the 2 visible instead of counted against the total.

Found while running the matrix on #925's branch (2289f10d); it is not that branch's defect. The code came in with #922 and is on main at f0f1f40.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions