fix: avoid dictionary key overflow in group value output

[StealthEyeLLC]

Which issue does this PR close?

Closes #23127.

Rationale for this change

GROUP BY over dictionary-encoded columns can produce more distinct group values than the key type of any individual input dictionary can represent.

For example, every input Dictionary(UInt8, Utf8) batch can be valid while the aggregate result contains more than 256 distinct groups. Re-encoding the complete result with UInt8 keys then fails with DictionaryKeyOverflowError.

What changes are included in this PR?

This changes dictionary group-value output to promote key types only when required:

• Top-level dictionary group columns are materialized to their value type for row interning.
• Emission starts with the planned dictionary key type.
• On key overflow, the key type grows one step at a time, such as UInt8 to UInt16 or Int8 to Int16.
• Once promoted, a key type does not shrink on later emissions.
• Final aggregate streams update their advertised schema when emitted dictionary keys grow.

Partial aggregate output still crosses fixed-schema repartition and coalescing boundaries. To keep that transport stable, partial output is materialized once, sliced at the planned dictionary key capacity, and each slice is compactly re-encoded to the planned schema. Spill IPC similarly uses a stable internal dictionary width.

What regression coverage is included?

• UInt8 remains UInt8 through 256 distinct values.
• 257 distinct values promote UInt8 to UInt16.
• Signed dictionary keys promote to the next signed width.
• Promoted output does not shrink on later emissions.
• Partial output is emitted as schema-stable batches and final aggregation promotes after combining all groups.
• Legacy and migrated aggregate paths.
• A repartitioned SQL GROUP BY regression.

Are these changes tested?

Yes. The current source was validated with:

cargo fmt --check
cargo test -p datafusion-physical-plan dictionary_groups_keep_partial_schema_and_promote_final_output -- --nocapture
cargo test -p datafusion-physical-plan dict_uint8_utf8_promotes_and_does_not_shrink -- --nocapture
cargo test -p datafusion-physical-plan dictionary_group_key_promotes_runtime_schema -- --nocapture
cargo test --test sqllogictests -- dictionary_group_by_key_promotion
cargo clippy -p datafusion-physical-plan --lib --tests -- -D warnings

The upstream Apache workflows still require maintainer approval before they can run on this PR.

[Rich-T-kid]

thank you for the PR @StealthEyeLLC, I left some of my thoughts.

[Rich-T-kid]

🤔 I don't think making every dictionary key type u64 is the best option here. changing a key from i8 to u64 just because a group by had 128 groups is overkill in my opinion.

[Rich-T-kid]

The root of the issue is here, we don't know how many groups we will end up with until we've called emit(). to resolve #23127 I think we need to allow for schemas to be somewhat dynamic. In the sense that a dictionary's key type can group but no shrink.

[StealthEyeLLC]

Thanks for the review, @Rich-T-kid. I reworked the implementation around dynamic, monotonic key promotion rather than widening every dictionary key to UInt64.

The current behavior is:

• emission begins with the planned dictionary key type
• the key grows one width only when Arrow reports DictionaryKeyOverflowError
• UInt8 remains UInt8 through 256 distinct values, and 257 values promote it to UInt16
• once promoted, the key type does not shrink on later emissions
• top-level dictionary values are materialized for row interning so aggregate stages can accept promoted dictionary keys
• partial aggregate batches remain schema-stable across repartition/coalescing by being sliced at the planned key capacity and compactly re-encoded
• final, ordered-final, and legacy streams update their advertised schema when runtime output promotes
• spill IPC uses a stable internal dictionary width

I added coverage for the promotion boundary, signed keys, no-shrink behavior, partial-to-final aggregation, the legacy path, and a repartitioned SQL regression.

The current source passes the targeted Rust tests, SQLLogicTest, formatting, and clippy with warnings denied. The Apache workflow runs are still waiting for maintainer approval. Could you please take another look and approve the workflows when convenient?