Skip to content

[FLINK-37926][table] Support casting from VARIANT to MAP - #29118

Merged
twalthr merged 3 commits into
apache:masterfrom
raminqaf:FLINK-37926-variant-to-map
Sep 7, 2026
Merged

[FLINK-37926][table] Support casting from VARIANT to MAP#29118
twalthr merged 3 commits into
apache:masterfrom
raminqaf:FLINK-37926-variant-to-map

Conversation

@raminqaf

@raminqaf raminqaf commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

What is the purpose of the change

This pull request adds CAST and TRY_CAST from VARIANT to MAP<STRING, V>. It is the last of the stacked constructed-cast PRs and builds on the same design: ARRAY (merged), ROW/STRUCTURED (#29092), and now MAP. It is stacked on #29092 and should be reviewed and merged after it.

MAP is the schemaless read of an object: unlike ROW, no field is named in advance, so it is the way to read an object whose keys are dynamic or unknown at query time.

Key semantics:

  • The variant must be an object. Each field name becomes a key and each value casts to V by the same VARIANT-to-value rules, recursively.
  • The key type must be a character string, since a variant object's keys are always strings. A non-string key type is rejected at validation.
  • A value present but set to a variant null maps to SQL NULL when V is nullable and fails the cast when V is NOT NULL. An empty object casts to an empty map.
  • A MAP<STRING, VARIANT> value type is the identity, so a variant null is kept as a variant null rather than downgraded to SQL NULL. This matches ARRAY<VARIANT> and a ROW with VARIANT fields.
  • Leaves are never parsed, and TRY_CAST returns NULL for the entire value on any failure.

Brief change log

  • Add VariantToMapCastRule that casts a variant object to MAP<STRING, V>, mapping field names to keys and casting each value with the existing VARIANT-to-value rules.
  • Allow the explicit cast in LogicalTypeCasts when the key is a character string and VARIANT casts to the value type.
  • Document the VARIANT to MAP cast and mark VARIANTMAP in the cast-support matrix.

Verifying this change

This change added tests and can be verified as follows:

  • CastRulesTest: key/value mapping, an empty object, a non-object input, and MAP<STRING, VARIANT> with and without a variant null value.
  • CastFunctionITCase: end-to-end SQL and Table API cases (the Table API cases use the parseJson() expression method).
  • LogicalTypeCastsTest and CastRuleProviderTest: castability and rule resolution, including the non-string-key rejection.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? docs

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [Tool Name and Version]

@flinkbot

flinkbot commented Sep 6, 2026

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@raminqaf
raminqaf force-pushed the FLINK-37926-variant-to-map branch from 31e2b3f to 48e2b5f Compare September 6, 2026 11:28
@raminqaf raminqaf changed the title [FLINK-37926][table] Support casting from VARIANT to ROW and STRUCTURED [FLINK-37926][table] Support casting from VARIANT to MAP Sep 6, 2026
@raminqaf
raminqaf force-pushed the FLINK-37926-variant-to-map branch 3 times, most recently from 44c712e to 1683ccc Compare September 7, 2026 08:33
@raminqaf
raminqaf force-pushed the FLINK-37926-variant-to-map branch from 1683ccc to 0c706e9 Compare September 7, 2026 08:36
@raminqaf
raminqaf marked this pull request as ready for review September 7, 2026 08:37

@twalthr twalthr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @raminqaf

@twalthr
twalthr merged commit 78b0df6 into apache:master Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants