Repository navigation
TLP violation: a cached scan returns four NULL rows in two partitions at once #19
Description
A WHERE p / WHERE NOT p / WHERE p IS NULL triple over a cached scan returns
more rows than the table holds. Four NULL rows land in two partitions at once.
Ternary-logic partitioning must hold for any correct SQL engine whatever p is:
each row satisfies exactly one of the three. It does not need an oracle to check,
which is why the failure is unambiguous.
Found by runtimedb's query fuzzer. The same query, same data, same seed passes
without liquid cache and fails with it.
The failure
[iter 137 | tlp-violation]
base: SELECT t1.s AS c0 FROM fz.fuzz.t1 AS t1
partitioned: SELECT t1.s AS c0 FROM fz.fuzz.t1 AS t1
WHERE (((t1.f <= 333.0 OR t1.s = t1.s) OR t1.id BETWEEN 1 AND 7))
unfiltered = 60 rows, union of the three partitions = 64 rows
0 only in left, 4 only in right — all four NULL
t1.s = t1.s is NULL rather than TRUE where s is NULL, so p is NULL exactly
where s IS NULL and f is NULL-or-above-333 and id is outside 1..7. Those
rows belong in the p IS NULL bucket and in no other. Four of them appear twice.
Note the shape: the query projects s while filtering on f, s and id —
a projection that does not match the filter columns.
What is ruled out
- Not batch sizing. Reproduces identically with
query_batch_target_bytes
set and with it at0(narrowing disabled): same iteration, same 60-vs-64,
same four rows. This was found while adding batch-size coverage, and that is
not the cause. - Not a reference disagreement. TLP runs entirely inside the engine under
test. Nothing is being compared against vanilla DataFusion here. - Not the plan. The DuckLake arm of the same suite — same generator, same
seed, same data, liquid cache off — passes this iteration.
What is NOT yet isolated
I tried to reduce it to a minimal case in runtimedb's null_semantics_tests
(own table, (a <= 5 OR a = a) over a nullable a, twelve rows, four NULL) and
it passed under both plain and liquid-cache modes. So the minimal shape is
wrong and one of these differences is load-bearing:
| Failing case | My passing reduction |
|---|---|
| DuckLake managed table | register_parquet on the filesystem |
Projects s, filters on f/s/id |
Projects and filters the same column |
| Three disjuncts, three columns, mixed types | Two disjuncts, one column |
| 60 rows, 4 columns | 12 rows, 1 column |
My guess, unverified: the table provider or the projection/filter mismatch. Those
are the two that touch the reader's selection and fallback paths.
Platform caveat — possibly the whole story
Observed only on macOS/aarch64, at 1fc1049 (#18), which means the store was
mounted with buffered I/O rather than io-uring. I have not reproduced it on
Linux, and runtimedb's CI has not run this query there.
So there are two possibilities and they matter differently:
- A general reader bug that happens to have been found on macOS.
- A correctness divergence specific to the buffered mount — which would
contradict feat: build and test natively on macOS #18's stated caveat that off-Linux builds are
correctness-equivalent and differ only in memory residency and compression.
Confirming or excluding (2) is the first thing worth doing, and it is cheap:
run the same case on Linux.
Reproducing
The automated repro needs runtimedb's unmerged feat/fuzz-batch-size-coverage
branch — that branch changes the fuzzer's table picker, which shifts the RNG
stream, and iteration 137 is a query the suite had never generated before.
QUERY_FUZZ_SEED=15785383 QUERY_FUZZ_ITERS=160 ./scripts/test.sh query_fuzz
The data does not depend on that branch: t1 is generated identically on main
for the same seed (60 rows; nullable id/a/f/s). So the four SQL
statements above, run over that table, should be reproducible anywhere.
Activity
zfarrell commentedon Aug 27, 2026
Not the buffered mount — reproduces on Linux/io-uring (2026-08-27)
The issue flagged two possibilities and said excluding the second was the first thing worth doing. It is now excluded.
runtimedb CI ran this on
ubuntu-2404-arm, where the store mounts through io-uring, and hit it identically:Same seed, same iteration, same row counts, same four NULL rows as the macOS run. So this is a general reader bug, not a correctness divergence introduced by the platform-aware mount in #18 — that PR's claim that off-Linux builds differ only in memory residency and compression ratios still stands.
Two further notes from that run, both narrowing where to look:
The remaining unknowns are unchanged: the minimal shape is still not isolated, and the two candidate axes are still the table provider (DuckLake vs
register_parquet) and the projection/filter mismatch — the failing query projectsswhile filtering onf,sandid.Reproduced from runtimedb PR #1204, which is where the fuzz change that generates this query now lives.