Skip to content

Join swap in join_selection flips schema-level metadata, failing "Schema mismatch" #23461

Description

@fangchenli

Describe the bug

When join reordering swaps a join's inputs, the swapped plan keeps a different value for a conflicting schema-level metadata key than the original, so OptimizationInvariantChecker rejects it:

Internal error: PhysicalOptimizer rule 'join_selection' failed. Schema mismatch.

swap_inputs rebuilds the join with inputs reversed and wraps it in a reverting ProjectionExec (reorder_output_after_swap), which restores column order and field metadata but not the schema-level metadata. So reversing two inputs that disagree on a key flips the winner, and the pre/post-swap schemas differ. This is on main, independent of merge direction, and shared by cross / hash / NLJ joins.

To Reproduce

// l: 1000 rows, schema metadata {"pandas": "LEFT"}
// r: 2 rows,    schema metadata {"pandas": "RIGHT"}
ctx.sql("SELECT * FROM l CROSS JOIN r").await?.collect().await?;
// -> Internal error: ... 'join_selection' failed. Schema mismatch.

set datafusion.optimizer.join_reordering = false; makes it pass.

Expected behavior

A swap must preserve the output schema, including schema-level metadata.

Additional context

Found while fixing the cross-join initial-plan case in #23434 / PR #23442, which leaves the shared swap path untouched.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Fields

    No fields configured for Bug.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions