test_a_query_that_cannot_use_the_order_does_not_pay_to_decide has failed on two different trees, on two different arms, in opposite directions. The measurement is noisy at the scale of the effect it is trying to detect, so neither bound can be adjusted to fix it.
The two readings
main @ 133c3fbd control: a query that CAN use the ordering does read to decide
got 'margin -9 (on=41 off=50)' want 'reads more'
#1201 @ bc44360 a query with no ORDER BY does not read the group list to decide
got 'differs by 39 (on=255 off=216 over 1000 groups)' want 'within 5'
Both are pytest (cluster tests, PG 17). I read each from its own failing job rather than from the other's description.
Why this is not "the tolerance is too tight"
The obvious fix — raise 5 — silences the second reading and leaves the first untouched, because the first is the control failing. That arm exists to prove the instrument can tell the two cases apart: the query that can use the ordering must read more. It read nine fewer.
A control that fails is the instrument reporting that it cannot discriminate on that run. Once the control is unreliable, a green main arm no longer means "no unnecessary read" — it can equally mean "today the two queries were indistinguishable". So the pass is as suspect as the failure.
The two failures also point opposite ways: one says a difference appeared where none should be, the other says a difference vanished where one should be. A single bound cannot be wrong in both directions at once; a noisy measurement can produce both.
The file already says why
From the test's own comment:
A TOLERANCE, not equality: two backends differ by a couple of buffers whatever this code does. Set far below the effect it must detect -- without the guard this read +44 on this fixture, and it grows with the group count.
and
THE GROUP COUNT IS IN THE FAILURE MESSAGE ... at a hundred groups the unguarded read was +6 -- close enough to the noise floor that the arm could not discriminate.
So the design already knows the effect and the noise are the same order at small group counts. The 39 observed at 1,000 groups says the noise is not confined to small ones.
Why it matters beyond one red
This is the #1164 shape: two sessions spent an afternoon unable to say whether the oracle or the control had failed, because the arm reduced four measured buffer counts to a boolean. That was fixed by putting the numbers in the message — which is the only reason this report can name the arms at all. The remaining half is that an unattributable red on a shared branch costs whoever is holding it, and this one has now landed on a PR that had nothing to do with it.
What would settle it
Not a number. Some way for the arm to know whether the measurement is usable on this run — the control is nearly that already, so making a failed control refuse the run rather than fail the suite would turn an unattributable red into "the instrument could not discriminate today", which is a different and honest outcome.
Occurrences here are the two I verified. @jdatcmd has a third from their own run plus the control-arm reading, to be added as a comment.
test_a_query_that_cannot_use_the_order_does_not_pay_to_decidehas failed on two different trees, on two different arms, in opposite directions. The measurement is noisy at the scale of the effect it is trying to detect, so neither bound can be adjusted to fix it.The two readings
Both are
pytest (cluster tests, PG 17). I read each from its own failing job rather than from the other's description.Why this is not "the tolerance is too tight"
The obvious fix — raise 5 — silences the second reading and leaves the first untouched, because the first is the control failing. That arm exists to prove the instrument can tell the two cases apart: the query that can use the ordering must read more. It read nine fewer.
A control that fails is the instrument reporting that it cannot discriminate on that run. Once the control is unreliable, a green main arm no longer means "no unnecessary read" — it can equally mean "today the two queries were indistinguishable". So the pass is as suspect as the failure.
The two failures also point opposite ways: one says a difference appeared where none should be, the other says a difference vanished where one should be. A single bound cannot be wrong in both directions at once; a noisy measurement can produce both.
The file already says why
From the test's own comment:
and
So the design already knows the effect and the noise are the same order at small group counts. The 39 observed at 1,000 groups says the noise is not confined to small ones.
Why it matters beyond one red
This is the #1164 shape: two sessions spent an afternoon unable to say whether the oracle or the control had failed, because the arm reduced four measured buffer counts to a boolean. That was fixed by putting the numbers in the message — which is the only reason this report can name the arms at all. The remaining half is that an unattributable red on a shared branch costs whoever is holding it, and this one has now landed on a PR that had nothing to do with it.
What would settle it
Not a number. Some way for the arm to know whether the measurement is usable on this run — the control is nearly that already, so making a failed control refuse the run rather than fail the suite would turn an unattributable red into "the instrument could not discriminate today", which is a different and honest outcome.
Occurrences here are the two I verified. @jdatcmd has a third from their own run plus the control-arm reading, to be added as a comment.