fix(vortex): compute per-file column statistics in-stream for vortex writes (mixed parquet+vortex corruption)

[moshap-firebolt]

Problem

Vortex data files were registered without per-file column statistics (the vortex COPY reports only a file list), so ducklake_file_column_stats got no rows and ducklake_table_column_stats was never updated past the parquet-only bounds. Readers trust those stats. On a table with a parquet file (ids 0–249) and a vortex file (ids 250–499):

• SELECT id, s FROM m WHERE id = 300 → no rows (filter folded to empty against stale max=249)
• SELECT count(*) FROM m WHERE id >= 250 → 0
• SELECT id, s FROM m ORDER BY id DESC LIMIT 2 → 243, 242 — DuckDB's sort-key compression packs the sort column into the stale range's width, truncating values mod 256

Silent wrong results, not errors.

Fix — compute the statistics while writing, like parquet

New DuckLakeStatsCopy wraps any sink-based COPY function that lacks a copy_to_get_written_statistics hook. The wrapper accumulates per-column min/max (arg-min/max raw comparison scan per chunk; one Value materialization per chunk per bound), null/value counts, and NaN detection for FLOAT/DOUBLE (NaN excluded from bounds and flagged, parquet-style) as the chunks stream through the sink — the same point where the parquet writer computes its statistics — and reports them through WRITTEN_FILE_STATISTICS. No extra pass over the written file. No extension change needed.

The insert path now consumes one unified WRITTEN_FILE_STATISTICS chunk shape for every format (the non-parquet file-list special case is gone; footer_size tolerates NULL since only parquet reports one). Consequences:

• vortex files get real ducklake_file_column_stats rows → file pruning now works on vortex data
• global table bounds stay correct → the filter-folding and sort-compression corruption is gone
• NOT NULL is now supported for vortex tables, enforced from the written null counts exactly like parquet (the previous rejection is lifted; vortex_write.test updated)

Bounds are recorded for types whose physical order matches logical order and whose values round-trip through the stats VARCHAR form; other columns (nested, blob, uuid, interval, oversized strings >1KB) record counts only — counts are what NOT NULL and AnyValid need.

Relation to feat/rtdl-vortex-written-stats (PR #4)

That branch flipped the return type to WRITTEN_FILE_STATISTICS assuming the vortex extension implements the hook — it doesn't (the pinned duckdb-vortex has no copy_to_get_written_statistics), so duckdb calls a null function pointer at file open and segfaults. This PR adopts that branch's insert-side unification and supplies the missing statistics via the wrapper. If the extension later implements the hook natively, DuckLakeStatsCopy::NeedsWrapping makes the wrapper step aside automatically.

Validation

• Regression test test/sql/vortex/vortex_mixed_format_stats.test: real per-file stats for the vortex file (bounds, null/value counts), correct global bounds spanning both files, all three previously-wrong query shapes, NOT NULL enforcement. Manually verified NaN handling (bounds exclude NaN, contains_nan recorded on file and global stats).
• test/sql/*: 396/398 — the 2 failures (settings/max_retry_count.test, metadata/ducklake_settings_sqlite.test) fail identically on unpatched main in this environment (env-dependent, unrelated).
• Cross-engine: ClickHouse, DataFusion, and Firebolt already read the same catalog correctly (a 3-readers-vs-writer divergence in the RTDL cross-check matrix is how this was found); with this fix the duckdb reader agrees with all of them on the full matrix.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes core insert/statistics registration for non-parquet writes; incorrect min/max or merge logic could reintroduce filter pruning or constraint bugs, though scope is limited to single-file vortex paths with extensive test coverage.

Overview
Fixes silent wrong results on mixed parquet+vortex tables where vortex files were registered without per-file column statistics, leaving stale parquet-only bounds in catalog metadata (broken filters, ORDER BY ... LIMIT truncation).

Adds DuckLakeStatsCopy, a wrapper around sink-based COPY functions that lack copy_to_get_written_statistics (e.g. vortex). It accumulates min/max, null/value counts, and NaN flags while chunks stream through the sink, then reports WRITTEN_FILE_STATISTICS (file size from the filesystem; footer_size left unset).

The DuckLake insert path drops the vortex file-list special case and always consumes the parquet-shaped statistics chunk (nullable footer_size). Non-parquet writers get the wrapper in GetCopyOptions when NeedsWrapping is true. NOT NULL enforcement for vortex is enabled from computed null counts (the previous blanket rejection is removed).

New regression tests cover mixed-format stats, type edge cases, and updated vortex_write.test expectations.

^{Reviewed by Cursor Bugbot for commit 8c47c26. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 0faa88c. Configure here.}

[cursor]

Variant columns not skipped in stats

Medium Severity

StatsCopyFinalize writes null_count/num_values for every sunk column, including VARIANT. AddWrittenFiles then rejects top-level variant stats, so a vortex insert into a table that has a VARIANT column fails after the file is written. The intended skip of variant columns is not implemented.

Additional Locations (1)

• src/storage/ducklake_insert.cpp#L183-L186

 

^{Reviewed by Cursor Bugbot for commit 0faa88c. Configure here.}

[moshap-firebolt]

Added adversarial coverage in test/sql/vortex/vortex_written_stats_types.test (195 assertions): every stats-capable type with extremes and negatives, ±inf as valid bounds vs NaN excluded+flagged, all-NaN and all-NULL columns, empty writes, the 1024-byte string-bound cap (1024 keeps bounds, 1025 drops them), counts-only types (BLOB/LIST/STRUCT), cross-chunk extremes under a parallel 500k-row insert, multi-insert global bound merges, mid-write NOT NULL abort atomicity, and vortex-then-parquet reverse mixed order.

Also pins three write-time type limitations discovered while testing (clean bind errors, so a silent behaviour change gets noticed): vortex rejects HUGEINT/UUID/INTERVAL; ducklake rejects ENUM.

Full suite: 397/399, same two pre-existing env-dependent failures as unpatched main.