Found while resolving #96 and #107 during the 2026-08-27 queue drain.
The problem
check_generated() is the authority for the synthetic fixture contract: it regenerates all five artifacts and compares every byte. It early-returns on non-3.12 lanes, on the assumption that other Python versions cannot reproduce the bytes.
That assumption is no longer true. CI's Python 3.10.21 and 3.11.16 both now ship SQLite 3.53.1 and resolve SQLAlchemy 2.0.52 — the same versions as the 3.12 reference builder. So on those lanes actual == expected, the early return fires, the inner assertion never runs, and the function returns having verified nothing.
Two of the three test lanes assert nothing about the fixtures. They report green either way.
Confirmed from the CI logs, not inferred from reading the code.
Why it matters
This is a check that cannot fail on two thirds of its lanes. It is the same shape as the failure that nearly shipped a real break elsewhere in the org today: a green result that is green because nothing was tested, not because the thing was correct.
It is not currently hiding a defect — the 3.12 lane does run the real comparison, and #96 and #107 both passed it. The risk is that if the 3.12 lane is ever skipped, reordered, or made non-required, the contract silently stops existing while CI stays green.
Why it was not fixed here
The correct fix is to let 3.10 and 3.11 run the real comparison, which requires verifying empirically that those lanes reproduce the bytes. That needs a machine with the right SQLite; the machine this was found on has the wrong SQLite for local 3.10, so the check could not be validated before changing it.
Changing the guard without that verification would risk turning a silent pass into a flaky failure, which is worse.
Suggested fix
- Empirically confirm 3.10 and 3.11 reproduce the fixture bytes in CI.
- If they do, drop the version guard and let all three lanes assert.
- If they do not, keep the guard but make it explicit and loud: skip with a stated reason, so a lane that verifies nothing reports as skipped rather than as passed.
Option 3 is worth doing even if option 2 turns out to be possible, because "passed" and "did not check" should never look identical.
Found while resolving #96 and #107 during the 2026-08-27 queue drain.
The problem
check_generated()is the authority for the synthetic fixture contract: it regenerates all five artifacts and compares every byte. It early-returns on non-3.12 lanes, on the assumption that other Python versions cannot reproduce the bytes.That assumption is no longer true. CI's Python 3.10.21 and 3.11.16 both now ship SQLite 3.53.1 and resolve SQLAlchemy 2.0.52 — the same versions as the 3.12 reference builder. So on those lanes
actual == expected, the early return fires, the inner assertion never runs, and the function returns having verified nothing.Two of the three test lanes assert nothing about the fixtures. They report green either way.
Confirmed from the CI logs, not inferred from reading the code.
Why it matters
This is a check that cannot fail on two thirds of its lanes. It is the same shape as the failure that nearly shipped a real break elsewhere in the org today: a green result that is green because nothing was tested, not because the thing was correct.
It is not currently hiding a defect — the 3.12 lane does run the real comparison, and #96 and #107 both passed it. The risk is that if the 3.12 lane is ever skipped, reordered, or made non-required, the contract silently stops existing while CI stays green.
Why it was not fixed here
The correct fix is to let 3.10 and 3.11 run the real comparison, which requires verifying empirically that those lanes reproduce the bytes. That needs a machine with the right SQLite; the machine this was found on has the wrong SQLite for local 3.10, so the check could not be validated before changing it.
Changing the guard without that verification would risk turning a silent pass into a flaky failure, which is worse.
Suggested fix
Option 3 is worth doing even if option 2 turns out to be possible, because "passed" and "did not check" should never look identical.