Conversation
Author
|
The two failing check families are unrelated to this change:
The changed crate is green locally: all 337 |
…() (apache#25306) - Closes apache#25305. `named_struct(...)` and `struct(...)` never produce a NULL row: `invoke_with_args` builds the output `StructArray` with no null buffer. Both functions nevertheless reported a nullable return field, and `ExprSchemable::nullable` for a scalar function reads that field. As a result the simplifier rule that rewrites `a IS NOT NULL` to `true` for a non-nullable `a` never fired on a struct constructor. The user-visible cost is lost pruning. A guard such as `WHERE s IS NOT NULL` on a struct built by a view cannot remove any row, but it keeps the whole struct expression alive, so projection pushdown can no longer prune the scan to the fields that are read: ```sql SET datafusion.explain.format = 'indent'; CREATE TABLE t (a INT, b INT, c INT) AS VALUES (1, 2, 3), (NULL, 5, 6); CREATE VIEW v AS SELECT named_struct('a', a, 'b', b, 'c', c) AS s FROM t; EXPLAIN SELECT s['b'] FROM v WHERE s IS NOT NULL; ``` Before: ``` logical_plan Projection: __datafusion_extracted_1 AS v.s[b] SubqueryAlias: v Projection: __datafusion_extracted_1 Filter: named_struct(Utf8("a"), t.a, Utf8("b"), t.b, Utf8("c"), t.c) IS NOT NULL Projection: t.b AS __datafusion_extracted_1, t.a, t.b, t.c TableScan: t projection=[a, b, c] ``` After: ``` logical_plan Projection: __datafusion_extracted_1 AS v.s[b] SubqueryAlias: v Projection: t.b AS __datafusion_extracted_1 TableScan: t projection=[b] ``` This matches the plan the same query already gets without the guard, and lets the existing `get_field(named_struct(...), 'f')` simplification (apache#22239) remove the constructor. - `datafusion/functions/src/core/named_struct.rs`: `return_field_from_args` now reports a non-nullable field instead of a nullable one. - `datafusion/functions/src/core/struct.rs`: adds a `return_field_from_args` override that reports a non-nullable field. The default implementation it previously used made the field nullable. Both are a one-line nullability correction; the inner struct fields stay nullable, since an individual member value can still be NULL. - New `sqllogictest` cases at the end of `datafusion/sqllogictest/test_files/struct.slt` covering the folded scalar results (`named_struct(...) IS NOT NULL`, `struct(...) IS NULL`), the `EXPLAIN` for `WHERE s IS NOT NULL` over a view showing `TableScan: t projection=[b]`, the values that query returns, and the `CASE WHEN named_struct(...) IS NOT NULL` fold. - Existing `datafusion-functions` unit tests: `cargo test -p datafusion-functions --lib`, 356 passed. - The full `sqllogictest` suite: all 518 files pass, with no other expectation changes needed. Yes, but no API change. - The schema nullability of the output of `named_struct()` and `struct()` changes from nullable to non-nullable. This now matches the array those functions actually build. - `IS NULL` and `IS NOT NULL` applied directly to one of these constructors constant-fold to `false` and `true`. Query results are unchanged; plans get smaller and can prune more. (cherry picked from commit 32662e7)
strawgate
force-pushed
the
codex/named-struct-non-nullable
branch
from
September 16, 2026 00:56
6740b05 to
d9a475b
Compare
strawgate
changed the title
Author
|
Closing as a duplicate of #75, which merged the same upstream backport into |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Backports apache/datafusion#25306 onto the DF55 branch used by Platform.
named_struct(...)andstruct(...)always construct aStructArraywithout an outer null buffer, but DF55 reports their return fields as nullable. Correcting that contract lets the existing simplifier fold redundant null guards and prune unused struct fields. Inner fields remain nullable.Validation
cargo test -p datafusion-functions --lib: 336 passedcargo test --profile=ci --test sqllogictests -- struct: passedcargo fmt --all -- --checkcargo clippy -p datafusion-functions --all-targets --all-features -- -D warningsThe backport retains the upstream constructor and projection-pruning coverage. The unrelated
list_cast_limittest that landed later on upstreammainwas not backported.