You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Nested-column conjuncts are silently dropped from the row filter #21
Same class as #19, different drop path, still open after #20.
pushdown_columns (src/datafusion/src/reader/plantime/row_filter.rs) returns None for three reasons: an empty column set (fixed in #20), non_primitive_columns, and projected_columns. build_row_filter then .flatten()s those candidates away. Because DataFusion has already removed the FilterExec, the liquid scan is the only place the predicate is applied, so a dropped conjunct widens the filter.
Repro — 8 rows, st is struct<a int>, one row has st.a = 3:
Fails on both cold and warm cache. Upstream DataFusion 54 supports struct-field-access pushdown (struct_field_accesses in its own PushdownChecker), so it plans the predicate as fully pushed while the liquid fork of row_filter.rs drops it.
The honest fix is to refuse the pushdown when a conjunct cannot be evaluated, so a FilterExec is retained, rather than dropping it. The doc paragraph on build_row_filter that sanctions dropping conjuncts ("the returned RowFilter may not contain all conjuncts ... will be ignored") encodes the assumption that #19 disproved and should go with it.
Activity