Repository navigation
fix: CSV empty cells become null; CLI filter on nullable columns - #593
Merged
Merged
Conversation
filter and select --where compared a row only against unmasked arrays, so any nullable column (a CSV import with an empty cell, a nullable Parquet column) failed with "filter not supported for column type: MaskedArray". A comparison now skips a null row, as SQL does (comparing with null is not true, for != as well), and compares the others through the masked column's values. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
An empty CSV cell in a number or boolean column was stored as 0, 0.0 or false, indistinguishable from a real value, and a column empty in every row became non-nullable I64 zeros. Inferred number and boolean columns are now nullable and store an empty cell as null; an inferred text column keeps it as "", as Arrow's CSV reader does by default. With an explicit schema a nullable column gets null (text too) and a non-nullable number or boolean column fails on an empty cell instead of storing a value. Every inferred column is nullable, not only those with an empty cell among the first chunk's rows, as Arrow and DuckDB infer them: an empty cell later in the file is then a null instead of a failed import. On a 2.7 GB CSV without empty cells that costs +0.008% file size and a few percent of write CPU. CsvEmptyCellsInteropIntegrationTest reads an imported file with vortex-jni, full and filtered, and finds nulls exactly where the cells were empty. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dfa1
force-pushed
the
fix/csv-empty-cells-null
branch
from
October 23, 2026 18:04
1402c59 to
1cc4b0f
Compare
A CSV import schema could only use I64, F64, Bool and Utf8: any other type, U16 included, failed with UnsupportedOperationException on the first chunk, after the output file was created. Now every integer type from I8 to U64 and F32 is accepted. Integers parse through the same byte fast path as I64, are range-checked against the declared type (a cast would wrap 65536 to 0 in a U16), and are narrowed into the array the writer takes. A U64 above Long.MAX_VALUE goes through Long.parseUnsignedLong. F32 uses Float.parseFloat: rounding a parsed double to float rounds twice and can differ. withSchema rejects F16 and non-primitive types up front. CsvEmptyCellsInteropIntegrationTest becomes CsvImportInteropIntegrationTest and gains a case where vortex-jni reads every narrow type back with Arrow's unsigned types and the same values, full and filtered on an I16 column. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #592. Two bugs, two commits (rebase-merge).
fix(csv): an empty CSV cell in a number or boolean column was stored as
0/0.0/false. Inferred number and boolean columns are now nullable and store an empty cell as null; inferred text keeps""(Arrow's default). With an explicit schema: nullable → null (text too), non-nullable number/boolean → the import fails on an empty cell instead of storing a value.Every inferred number/boolean column is nullable, not only those with an empty cell in the first 65 536 rows, as Arrow and DuckDB infer them, so a late empty cell is a null instead of a failed import. Measured on the 2.7 GB CSV (no empty cells): +0.008% file size (+67 KB on 875 MB), a few % write CPU, same wall time.
fix(cli): found while measuring the read side:
filter/select --wherethrewfilter not supported for column type: MaskedArrayon any nullable column. A comparison now skips null rows (SQL semantics,!=included) and compares the rest through the masked values.Tests:
CsvEmptyCellsInteropIntegrationTest: CSV with empty cells →CsvImporter→ read with vortex-jni, full and filtered: nulls exactly where cells were empty. Failed before the fix.CsvImporterTest: inference (nullable types, export round-trip of nulls), late empty cell → null, non-nullable schema → fails, nullable text schema → null.FilterCommandTest: nullable column with>,!=,=; failed before the fix../mvnw verifygreen locally.🤖 Generated with Claude Code