The nightly's "per file, least covered first" table names the best-covered files as the worst. columnar_parquet_codec.c is 100% line-covered in the run's own tracefile and the table prints it at 2.0%, sorted to the top of a list whose heading says least covered first. Anyone using that table to choose what to test next is being pointed at the wrong files.
Found while diagnosing the two-night nightly red (#973, fixed in #972); unrelated to that failure.
Measured, three ways, all from nightly deep gate #66
1. What the job printed (test/run_coverage.sh:289):
-- per file, least covered first
|Lines |Functions |Branches
columnar_compat.h | 0.0% 5| - 0| - 0
columnar_parquet_codec.c | 2.0% 100|3200% 2| - 0
columnar_arrow.c | 2.7% 1145|2487% 31| - 0
columnar_write_state.c | 2.8% 1487|1655% 40| - 0
3200% of 2 functions is impossible on its face, and every branch column is - 0.
2. What the same step printed four lines earlier, from the same tracefile:
Summary coverage rate:
lines......: 93.7% (21320 of 22765 lines)
functions..: 96.8% (927 of 958 functions)
branches...: 70.4% (9791 of 13900 branches)
3. The run's own uploaded coverage.info, with the job's identical pipeline, locally on lcov 2.0-1:
columnar_autovacuum.c |72.3% 173|71.4% 7|51.2% 84
columnar_objstore.c |80.4% 97| 100% 6|56.2% 64
columnar_sink.c |82.5% 63| 100% 5|48.0% 50
columnar_parallel_copy.c |83.1% 657|88.2% 17|53.8% 396
and for the two files above:
columnar_parquet_codec.c | 100% 100| 100% 2|73.4% 64
columnar_arrow.c |94.9% 1206| 100% 31|74.7% 771
So the tracefile is right, the summary is right, and the per-file table is the only thing that is wrong. Its line rates bear no relation to the data, its function rates exceed 100%, and its branch column is empty where the tracefile carries 13,900 branches.
I also checked the raw records rather than trusting any tool: for columnar_parquet_codec.c the tracefile holds FNF:2 FNH:2 — 100% of functions hit, not 3200%.
What I could not isolate
The two lcov builds differ: the runner installs 2.0-4ubuntu2, I ran 2.0-1. Same upstream 2.0, different distro patch level, and the only thing that behaves differently is --list. I have not bisected which patch does it, and I am not going to guess — the point of the report is that the output is wrong, and the evidence is that the identical command on the identical input gives different answers on the two builds.
A second, smaller defect, certain
lcov --list ... | awk '/\|/ && !/Total:/ && !/^Filename/ {print}' | sort -t'|' -k2 -n | head -20!/^Filename/ never matches. lcov's header line begins with spaces, not with Filename, so it survives the filter, goes into the sort, and consumes one of the twenty slots. Visible in the job's own output — the header is the first row of the table — and in my local run, where sort is handed field2=[Lines ] as data.
That one costs a row rather than correctness, but it is the tell that the pipeline was never checked against its own output.
Suggested fix
Compute the table from the tracefile, not from a tool's human-readable table. coverage.info carries LF/LH, FNF/FNH and BRF/BRH per SF: record; a dozen lines of awk or Python over those is version-proof, needs no column guessing, and is immune to whatever distinguishes the two lcov builds. The summary line can keep using lcov --summary, which is correct on both.
Acceptance test, since this is a reporting path and reporting paths fail silently: pick one file whose true rate is known from the tracefile records and assert the table's row for it matches. The current table would fail that test at columnar_parquet_codec.c — 100% in the records, 2.0% in the table — which is exactly the assertion that would have caught this on the day it started.
Why it matters beyond the numbers
A coverage table is read to decide where to spend effort, and this one inverts the ranking: four of the files it lists as least covered are between 94% and 100%. #282 and #282-like issues have been opened off this table before. While it is wrong, "the nightly says X is untested" is not evidence.
The nightly's "per file, least covered first" table names the best-covered files as the worst.
columnar_parquet_codec.cis 100% line-covered in the run's own tracefile and the table prints it at 2.0%, sorted to the top of a list whose heading says least covered first. Anyone using that table to choose what to test next is being pointed at the wrong files.Found while diagnosing the two-night nightly red (#973, fixed in #972); unrelated to that failure.
Measured, three ways, all from nightly deep gate #66
1. What the job printed (
test/run_coverage.sh:289):3200%of 2 functions is impossible on its face, and every branch column is- 0.2. What the same step printed four lines earlier, from the same tracefile:
3. The run's own uploaded
coverage.info, with the job's identical pipeline, locally on lcov 2.0-1:and for the two files above:
So the tracefile is right, the summary is right, and the per-file table is the only thing that is wrong. Its line rates bear no relation to the data, its function rates exceed 100%, and its branch column is empty where the tracefile carries 13,900 branches.
I also checked the raw records rather than trusting any tool: for
columnar_parquet_codec.cthe tracefile holdsFNF:2 FNH:2— 100% of functions hit, not 3200%.What I could not isolate
The two lcov builds differ: the runner installs 2.0-4ubuntu2, I ran 2.0-1. Same upstream 2.0, different distro patch level, and the only thing that behaves differently is
--list. I have not bisected which patch does it, and I am not going to guess — the point of the report is that the output is wrong, and the evidence is that the identical command on the identical input gives different answers on the two builds.A second, smaller defect, certain
!/^Filename/never matches. lcov's header line begins with spaces, not withFilename, so it survives the filter, goes into the sort, and consumes one of the twenty slots. Visible in the job's own output — the header is the first row of the table — and in my local run, wheresortis handedfield2=[Lines ]as data.That one costs a row rather than correctness, but it is the tell that the pipeline was never checked against its own output.
Suggested fix
Compute the table from the tracefile, not from a tool's human-readable table.
coverage.infocarriesLF/LH,FNF/FNHandBRF/BRHperSF:record; a dozen lines of awk or Python over those is version-proof, needs no column guessing, and is immune to whatever distinguishes the two lcov builds. The summary line can keep usinglcov --summary, which is correct on both.Acceptance test, since this is a reporting path and reporting paths fail silently: pick one file whose true rate is known from the tracefile records and assert the table's row for it matches. The current table would fail that test at
columnar_parquet_codec.c— 100% in the records, 2.0% in the table — which is exactly the assertion that would have caught this on the day it started.Why it matters beyond the numbers
A coverage table is read to decide where to spend effort, and this one inverts the ranking: four of the files it lists as least covered are between 94% and 100%. #282 and #282-like issues have been opened off this table before. While it is wrong, "the nightly says X is untested" is not evidence.