feat(vortex): read DuckLake data files in the vortex format (per-file dispatch) - #1
Merged
Merged
Conversation
…rtex) Make the DuckLake scan path honor the per-file file_format column of ducklake_data_file instead of assuming parquet for every file. Files recorded as "vortex" are now read via a dedicated reader that drives the vortex extension's read_vortex table function; parquet files are unchanged. Formats can be mixed within a single table (per-file dispatch), reusing the existing delete-filter, row-id and column-mapping machinery. - thread file_format from ducklake_data_file into DuckLakeFileData and on into OpenFileInfo.extended_info (GetFileSelectList/ReadDataFile gain an opt-in flag, enabled only on the data-file scan paths) - DuckLakeMultiFileReader::CreateReader dispatches "vortex" files to the new DuckLakeVortexReader (BaseFileReader), mirroring the existing inlined-data reader dispatch - vendor vortex-data/duckdb-vortex as the ./vortex submodule and load it alongside parquet via extension_config.cmake (opt-out with DISABLE_VORTEX) Write support (recording file_format="vortex" on insert / COPY FORMAT VORTEX) is intentionally deferred to a follow-up, so end-to-end vortex round-trip tests land with the write path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tex dispatch Follow-up to the file_format read dispatch: generalize the reader and add the write side so a table's data files can be produced and read in a non-parquet format, verified end-to-end. - rename DuckLakeVortexReader -> DuckLakeFormatReader: drives read_<format> for any non-parquet format (vortex, json, ...), mapping columns by name and applying deletes/row-ids/filters via the existing MultiFileReader contract - write path: data_file_format catalog option (parquet default, or vortex); GetCopyOptions routes to the format's COPY, records file_format in metadata, and PlanInsert/AddWrittenFiles stamp it. A guard rejects copy functions without copy_to_bind (e.g. json/csv plan-based COPY) instead of crashing. - tests: data_file_format_option (recording + validation), json_read_dispatch (proves per-file dispatch end-to-end via read_json, incl. projection/filter pushdown), and vortex_format (round-trip; runs where vortex's DuckDB pin is aligned with DuckLake's). Verified on an icecc-farm release build: option 9/9, json_read_dispatch 24/24, full existing DuckLake suite 503/503 (no regressions). Vortex round-trip is currently unbuildable here because the latest vortex extension pins DuckDB d8cdaa33 (Jul-21) while DuckLake main requires c97bd8b9 (Jul-22, adds table_index.hpp, removes Vector::GetBuffer) - an upstream version-alignment gap, not a code defect. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
moshap-firebolt
force-pushed
the
firebolt/vortex-support
branch
from
August 4, 2026 21:20
5cda6b9 to
e7d05b9
Compare
moshap-firebolt
force-pushed
the
firebolt/vortex-support
branch
2 times, most recently
from
August 4, 2026 21:46
d896754 to
9b7110e
Compare
… dispatch) Adds vortex (and general non-parquet) read support to DuckLake keyed off the per-file file_format column, on an aligned DuckDB base where DuckLake + vortex build together. - generic DuckLakeFormatReader drives read_<format> and adapts chunks to the MultiFileReader contract (column mapping, deletes, filters, file_row_number) - thread file_format through all scan paths (forward, insertions, delete-scan, compaction) and into transaction-local files + DuckLakeDataFile copy/assign - resolve the written format once on DuckLakeInsertGlobalState so INSERT, CTAS, flush and compaction all record it consistently - data_file_format catalog option; vendor vortex-data/duckdb-vortex submodule - pin DuckDB to d8cdaa33 (vortex's fork: old Vector layout + GEOMETRY/VARIANT C-API); two unique_ptr covariance fixes to build DuckLake against it - tests: vortex_read_dispatch (read + projection/filter pushdown + delete), json_read_dispatch, data_file_format_option Write path (INSERT -> vortex) is a follow-up: vortex's C-API COPY cannot report per-file statistics, so DuckLake's WRITTEN_FILE_STATISTICS writer would need to route through CHANGED_ROWS_AND_FILE_LIST + fstat. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
moshap-firebolt
force-pushed
the
firebolt/vortex-support
branch
from
August 4, 2026 22:20
9b7110e to
4e9776f
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 4e9776f. Configure here.
| for (idx_t r = 0; r < scan_count; r++) { | ||
| ordinal_data[r] = file_row_number + NumericCast<int64_t>(r); | ||
| } | ||
| break; |
There was a problem hiding this comment.
Ordinal column left non-flat
High Severity
DuckLakeFormatReader::Scan writes file_row_number ordinals with FlatVector::GetData but never forces that column back to a flat vector. After a batch is Sliced by deletes or filters, the ordinal column can remain a dictionary vector, so later batches overwrite the child data while the old selection stays active. That corrupts row-ids used by deletes, updates, and any rowid projection on multi-batch vortex/json scans.
Reviewed by Cursor Bugbot for commit 4e9776f. Configure here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds vortex (and general non-parquet) read support to DuckLake, keyed off the per-file
file_formatcolumn, and vendors the vortex extension so DuckLake + vortex build together.What this does
file_format='vortex'through a genericDuckLakeFormatReaderthat drivesread_vortex(parquet files unchanged). Formats can mix within one table (per-file dispatch), reusing the existing delete-filter / row-id / column-mapping machinery.file_formatfromducklake_data_filemetadata into the scan; adds adata_file_formatcatalog option + write-side recording.vortex-data/duckdb-vortexas the./vortexsubmodule and wires it viaextension_config.cmake.Why the base is April-1 DuckLake + DuckDB
d8cdaa33Vortex's C++ glue needs DuckDB's pre-March-24
Vectorlayout (data/validitymembers,StringVector::AddBuffer) and its Rust needs theGEOMETRY/VARIANTC-API types.d8cdaa33(vortex's DuckDB fork) is the one commit that has both, and DuckLake @ Apr-1 is the newest DuckLake that predates the DuckDB Vector refactor /table_index.hppdependency. This is the only combination where DuckLake + vortex compile together. Two small covariance fixes (unique_ptr<Derived>→Base) let DuckLake build against that DuckDB.Verified
test/sql/vortex/vortex_read_dispatch.test(24 assertions): writes a real.vortexfile with vortex's own COPY, tags it in metadata, reads it back through DuckLake with projection + filter pushdown + aggregation.test/sql/format/json_read_dispatch.test(24) anddata_file_format_option.test(9): the same mechanism viaread_json+ option validation.Known gap (follow-up)
INSERT ... (data_file_format='vortex')is not wired yet: vortex's C-API COPY cannot report per-file statistics (the DuckDB C API has no written-stats setter) and DuckLake's writer requestsWRITTEN_FILE_STATISTICS. Next step: route vortex writes throughCHANGED_ROWS_AND_FILE_LIST+fstatfor size.🤖 Generated with Claude Code
Note
Medium Risk
Touches core scan/insert and metadata I/O for all tables; behavior change is gated on per-file format, but bugs could mis-route reads or persist wrong format metadata.
Overview
Adds pluggable data-file formats to DuckLake: each data file’s physical format is stored in metadata and used at scan time, while inserts can target parquet or vortex via a new catalog option.
Read path: Metadata now loads
ducklake_data_file.file_formatinto scan file options. Non-parquet files use a newDuckLakeFormatReaderthat binds and runs the matchingread_<format>table function (e.g.read_vortex,read_json), maps columns by name (with rename maps viamapping_id), synthesizes row ordinals, and applies the same deletion and filter pushdown as the parquet path. Parquet still uses the existing reader.Write / config:
ducklake.set_option('data_file_format', …)validatesparquetorvortex;GetCopyOptionspicks the corresponding COPY function and recordsfile_formaton written files instead of hard-coding parquet. Parquet-only options (field IDs, encryption, compression) are skipped for other formats.Build: Optional
vortexsubmodule andENABLE_VORTEX=1inextension_config.cmake;.gitignoreadds.cache/.SQL tests cover option validation, JSON read dispatch, vortex read dispatch (including deletes), and name-map reads after column rename.
Reviewed by Cursor Bugbot for commit 4e9776f. Bugbot is set up for automated code reviews on this repo. Configure here.