[epic] Correctness testing plan for the reader: differential + invariance harness #22

Tracking issue. Detail lives in the children — this one holds the motivation and the ordering.

Seven correctness bugs in the last few weeks, six in src/datafusion/src/reader. Every one was found by a human writing a specific test, by a downstream fuzzer, or by reviewing another fix — none by this repo's own test suite.

The only bugs found systematically came from a fuzzer in another repo. And #21 came from reviewing the fix for #19 — the class was wider than the first fix assumed.

The suite isn't weak in general (381 tests, shuttle, ASAN, TPC-H/DS and ClickBench with answer checks). It's weak in four specific places: the oracles only check queries someone wrote down, the cache config space is barely swept, parquet file shape isn't varied systematically, and predicate shape isn't varied at all.

This research also turned up a live wrong-results bug, tracked separately as #23 — the row filter silently drops conjuncts it cannot evaluate. It needs none of the work below and should land first. #26 is what would catch the next instance of that class automatically.

Work

Do first — independent of everything else

• Nightly fuzz job discards its corpus and uploads artifacts from the wrong path #24 — Nightly fuzz job discards its corpus, uploads from the wrong path

Harness — #25 blocks the rest

• Fuzz harness: seeded generator, vanilla DataFusion reference, differential oracle #25 — Skeleton: seeded generator, vanilla DataFusion reference, differential oracle
• Fuzz harness: query-shape generation #26 — Query-shape generation (fix(reader): don't apply limit before the row filter #11, TLP violation: a cached scan returns four NULL rows in two partitions at once #19, Nested-column conjuncts are silently dropped from the row filter #21 are all query-shape bugs)
• Fuzz harness: parquet writer-shape generation #27 — Parquet writer-shape generation
• Fuzz harness: cache-config sweep with config-invariance and cache-state oracles #28 — Cache-config sweep + invariance oracles
• Fuzz harness: TLP self-check oracle #29 — TLP self-check (the oracle that found TLP violation: a cached scan returns four NULL rows in two partitions at once #19)
• Fuzz harness: budget and stats accounting invariants #30 — Budget/stats invariants (the only item covering the feat(admission): count the on-disk liquid tier toward the gate budget #9 class)
• Fuzz harness: wire PR-gating and nightly arms into CI #31 — CI wiring (PR + nightly arms)

Encoding layer

• Property tests for the remaining liquid array types #32 — Property tests for remaining array types
• Fuzz target for read_from_bytes over arbitrary bytes #33 — read_from_bytes fuzz target

Later

• Fuzz harness: concurrent-scan arm #34 — Concurrent-scan arm
• Parquet corpus from third-party writers #35 — Third-party parquet corpus (the real answer to the fix(reader): treat the parquet page index as optional #10 class)
• Confirm stale-file semantics: the cache is keyed by path with no invalidation #36 — Stale-file semantics question

Caveats

• Doesn't cover the distributed path. datafusion-client and the server's flight service get only smoke examples today. Accepted residual risk.
• A mismatch isn't automatically our bug. LiquidCacheLocalBuilder::build forces pushdown_filters = true and schema_force_view_types = false (src/datafusion-local/src/lib.rs:178-183), both non-default upstream. Budget for triage.
• Determinism is partial. A seed reproduces generated inputs, not tokio interleavings or eviction order. Main arms run single-partition; concurrency is Fuzz harness: concurrent-scan arm #34.
• Platform is a real dimension. TLP violation: a cached scan returns four NULL rows in two partitions at once #19 was seen only on macOS/aarch64, where the store mounts buffered rather than io-uring. Still unresolved and cheap to settle.