Repository navigation
fix(query): stop trusting unverified Parquet scan ordering - #48
Merged
Merged
Conversation
Manifest and hot-tier Parquet scans declared `time DESC NULLS LAST` output ordering whenever the footer-derived `sort_order_id` said so (one file needed no further proof; multi-file groups were chained by min/max statistics). The staging writer records that footer claim without sorting: concurrent requests are appended in writer-lock order and time-partitioned rows keep client order. With the claim, DataFusion drops the sort under `SortPreservingMergeExec` and `ORDER BY p_timestamp DESC NULLS LAST LIMIT n` over a historical window returns wrong rows. Remove the ordering declaration and its helpers. The scan body moves to a private `parquet_scan_plan` so tests exercise the real plan code without the CLI global. `pub fn partitioned_files` keeps its signature; its flag is now always `false`. Add regression tests over files from the real staging writer, checked against an independent tie-aware sort of the decoded rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
VibeSWE
commented
Oct 19, 2026
VibeSWE
marked this pull request as ready for review
October 21, 2026 13:40
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix incorrect timestamp ordering when persisted Parquet rows do not match their footer sorting declaration. Concurrent requests can acquire timestamps before acquiring the writer lock, so append order is not proof of timestamp order. The reader must retain the sort even when metadata advertises an ordering.
The public no-LIMIT reproduction returns all 120,000 correct rows but eight strict timestamp ascents on the original reader. This change returns the same population correctly ordered. Equal-timestamp row order is intentionally unconstrained.
Changes
The optional serializer is isolated in #54. No serializer, typed TopK, dependency fork or ordering-certificate feature is included here.
Performance tradeoff
A completed application campaign compared original main A, this reader fix B at
35988060, and B plus a separate selective serializer C. Three counterbalanced blocks completed 18 fresh server lifetimes and 84,600 timed requests. An independent audit recomputed every cell and ratio from raw records.LIMIT 10000over 20,000 rows: B used 49.3–49.7% more CPU with the default response and 43.4–44.7% more CPU with null filling. The added TopK sort is visible in the plan.See the checked-in evidence summary for the matrix, source/ELF provenance limits, raw-artifact hashes and interpretation. The incomplete earlier R181 campaign is not pooled. Historical optimization failures remain failures.
Validation and disposition
Full-crate CI now passes on
4282f781e81404e02423ebdfe6485ae24f5b2d95(CI run): Rust tests, Clippy, image build, and standalone/distributed smoke suites are green. Both new production-path regression tests passed:listing_table_ignores_footer_sort_orderandlisting_table_plans_footer_with_invalid_sort_column. Formatting and independent Opus/Sol source reviews also passed. The source-level negative-control expectation (tests fail without the wrapper) has not been executed.The measured campaign predates the legacy listing repair. Ordered legacy queries may now pay for a required sort; that path’s performance has not been measured.
Merge authorized by the user on 2026-10-21 after review of the correctness-first tradeoff and completed CI. The known dense-query cost is accepted for this correctness repair; it has not been eliminated and no CPU/RAM parity claim is made. The optional serializer is reviewed separately. A safe future ordered-scan optimization would require verified physical ordering and a trustworthy certificate; existing files cannot acquire that guarantee merely from footer flags or min/max statistics.