You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Generate parquet files with varied physical layout rather than querying one fixed file. Today the input space is examples/nano_hits.parquet plus a handful of targeted files added alongside recent fixes (batch_size_alignment.rs:37, page_index.rs) — all written by the same writer, and none produced systematically.
Properties to vary
Row group size (including files with many small row groups, so a scan crosses boundaries mid-batch)
Data page row count limit — a small limit gives the page index enough granularity to prune within a row group, which is what produces a row selection starting partway in
Dictionary encoding on/off
Per-column statistics level
Writer version
Compression codec
Null density, including all-null and no-null columns
Single file vs multiple files in a listing table
write_parquet in batch_size_alignment.rs:37-56 is the right starting shape — it already takes optional row group and page row limits.
Note on scope
This dimension is a weaker fit for #10 than it looks. That bug needed a file advertising a page index while one column chunk carries no offset index, which is a shape the Arrow writer does not readily produce. A corpus of files written by other tools is the more reliable answer to that class and is tracked separately.
Acceptance
File properties are part of the seeded generation and appear in the reproduce output.
The report counts iterations by row-group and page-limit configuration, so a run that only ever generated single-row-group files is visible rather than silent.
Activity