What happened?
The CSV and JSONL scan source operators infer each column's type from only the first INFER_READ_LIMIT (= 100) rows. When a later row holds a value that does not parse as the inferred type, the operator silently discards the entire row — no warning, no skipped-row count, no console message, no log entry. The user has no way to learn that data was lost, so every downstream count, aggregate, join, and model is computed on a silently truncated dataset.
Cells that are simply empty are kept as null (the row survives), while cells that fail to parse cause the whole row to vanish. The operator treats "missing" and "malformed" differently, and the more destructive path is the silent one. Rejecting malformed input is defensible but doing it silently would be a bug
What I expected to happen: the loss should not be silent. At minimum the number of skipped rows should be surfaced (execution statistic / console warning), or the scan should fail loudly and name the offending row and column. Losing data with zero signal is the defect.
How to reproduce?
Using the diabetes dataset (400 data rows). Type inference only samples the first 100 rows, so inject a bad value after row 100 into a column that is all-integer in the first 100 rows (e.g. age), which makes inference stably INTEGER:
- Create a corrupted copy that puts a non-integer into
age on three rows past row 100:
# data rows 150, 250, 350 -> age becomes "55.5" (fails Integer parsing)
awk -F, 'BEGIN{OFS=","} NR==1{print; next}
{n++; if(n==150||n==250||n==350){$8="55.5"} print}' diabetes.csv > diabetes_b2.csv
- Upload
diabetes_b2.csv as a dataset and point a CSV File Scan operator at it.
- Run the workflow and look at the scan operator's output row count.
Expected: 403 rows accounted for, or a visible signal that 3 rows were skipped.
Actual: the scan emits 400 rows. The 3 rows containing 55.5 are gone, with no warning, no skipped-row count, and nothing in the console — silent data loss. (JSONL, ParallelCSV, and the csvOld variant share the same behavior.)
Version/Branch
1.3.0-incubating-SNAPSHOT (main)
Commit Hash (Optional)
No response
What browsers are you seeing the problem on?
No response
Relevant log output
What happened?
The CSV and JSONL scan source operators infer each column's type from only the first
INFER_READ_LIMIT(= 100) rows. When a later row holds a value that does not parse as the inferred type, the operator silently discards the entire row — no warning, no skipped-row count, no console message, no log entry. The user has no way to learn that data was lost, so every downstream count, aggregate, join, and model is computed on a silently truncated dataset.Cells that are simply empty are kept as
null(the row survives), while cells that fail to parse cause the whole row to vanish. The operator treats "missing" and "malformed" differently, and the more destructive path is the silent one. Rejecting malformed input is defensible but doing it silently would be a bugWhat I expected to happen: the loss should not be silent. At minimum the number of skipped rows should be surfaced (execution statistic / console warning), or the scan should fail loudly and name the offending row and column. Losing data with zero signal is the defect.
How to reproduce?
Using the
diabetesdataset (400 data rows). Type inference only samples the first 100 rows, so inject a bad value after row 100 into a column that is all-integer in the first 100 rows (e.g.age), which makes inference stably INTEGER:ageon three rows past row 100:diabetes_b2.csvas a dataset and point a CSV File Scan operator at it.Expected: 403 rows accounted for, or a visible signal that 3 rows were skipped.
Actual: the scan emits 400 rows. The 3 rows containing
55.5are gone, with no warning, no skipped-row count, and nothing in the console — silent data loss. (JSONL, ParallelCSV, and thecsvOldvariant share the same behavior.)Version/Branch
1.3.0-incubating-SNAPSHOT (main)
Commit Hash (Optional)
No response
What browsers are you seeing the problem on?
No response
Relevant log output