Skip to content

[FLINK-40580][table] Render a VARIANT object or array as JSON when casting to a character string #29125

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
raminqaf wants to merge 2 commits into
base: master
Choose a base branch
from
Open

[FLINK-40580][table] Render a VARIANT object or array as JSON when casting to a character string #29125

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6d2a38e
[FLINK-40580][table] Render a VARIANT object or array as a SQL string…
raminqaf Sep 7, 2026
aa20b8a
[FLINK-40580] Address feedbacks
raminqaf Sep 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions docs/content.zh/docs/sql/reference/data-types.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1583,10 +1583,11 @@ CAST(CAST(PARSE_JSON('3.9') AS DECIMAL(2, 1)) AS INT) -- returns 3 (truncate

A cast to a character string renders the value exactly as a regular SQL cast of the stored kind
would, so a boolean becomes `TRUE`, a timestamp uses the SQL format, a `TIMESTAMP_LTZ` is shifted into
the session time zone, and a binary value is read as UTF-8. Only an object or an array has no such
rendering. Use `JSON_STRING` for the JSON representation instead, where a string stays quoted as
`"foo"` and an object or array is serialized. A variant that stores a JSON `null` casts to SQL
`NULL`.
the session time zone, and a binary value is read as UTF-8. An object or an array has no scalar
form, so it renders like a regular `ARRAY` or `MAP` cast to a string: an array as `[e1, e2]` and an
object as `{k1=v1, k2=v2}`, with each value rendered by these same rules and a nested variant null
shown as `NULL`. A string is never quoted, at any depth. Use `JSON_STRING` for the JSON form with
quoted strings. A variant that stores a JSON `null` casts to SQL `NULL`.

A `VARIANT` can also be cast to a constructed target, which imposes a schema on it. A variant array
casts to `ARRAY<T>`. The variant must be an array, otherwise the cast fails. Each element is itself a
Expand Down Expand Up @@ -1873,7 +1874,7 @@ COALESCE(TRY_CAST('non-number' AS INT), 0) --- 结果返回数字 0 的 INT 格
| `ROW` | Y | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | !³ | N | N | N | N |
| `STRUCTURED` | Y | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | !³ | N | N | N |
| `RAW` | Y | ! | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | Y⁴ | N | N |
| `VARIANT` | N | ! | ! | ! | ! | ! | ! | ! | ! | ! | ! | N | ! | ! | N | !³ | N | !³ | !³ | !³ | N | Y | N |
| `VARIANT` | ! | ! | ! | ! | ! | ! | ! | ! | ! | ! | ! | N | ! | ! | N | !³ | N | !³ | !³ | !³ | N | Y | N |
| `BITMAP` | Y | Y⁷ | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N |

备注：
Expand Down
11 changes: 6 additions & 5 deletions docs/content/docs/sql/reference/data-types.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1591,10 +1591,11 @@ CAST(CAST(PARSE_JSON('3.9') AS DECIMAL(2, 1)) AS INT) -- returns 3 (truncate

A cast to a character string renders the value exactly as a regular SQL cast of the stored kind
would, so a boolean becomes `TRUE`, a timestamp uses the SQL format, a `TIMESTAMP_LTZ` is shifted into
the session time zone, and a binary value is read as UTF-8. Only an object or an array has no such
rendering. Use `JSON_STRING` for the JSON representation instead, where a string stays quoted as
`"foo"` and an object or array is serialized. A variant that stores a JSON `null` casts to SQL
`NULL`.
the session time zone, and a binary value is read as UTF-8. An object or an array has no scalar
form, so it renders like a regular `ARRAY` or `MAP` cast to a string: an array as `[e1, e2]` and an
object as `{k1=v1, k2=v2}`, with each value rendered by these same rules and a nested variant null
shown as `NULL`. A string is never quoted, at any depth. Use `JSON_STRING` for the JSON form with
quoted strings. A variant that stores a JSON `null` casts to SQL `NULL`.

A `VARIANT` can also be cast to a constructed target, which imposes a schema on it. A variant array
casts to `ARRAY<T>`. The variant must be an array, otherwise the cast fails. Each element is itself a
Expand Down Expand Up @@ -1882,7 +1883,7 @@ The matrix below describes the supported cast pairs, where "Y" means supported,
| `ROW` | Y | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | !³ | N | N | N | N |
| `STRUCTURED` | Y | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | !³ | N | N | N |
| `RAW` | Y | ! | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | Y⁴ | N | N |
| `VARIANT` | N | ! | ! | ! | ! | ! | ! | ! | ! | ! | ! | N | ! | ! | N | !³ | N | !³ | !³ | !³ | N | Y | N |
| `VARIANT` | ! | ! | ! | ! | ! | ! | ! | ! | ! | ! | ! | N | ! | ! | N | !³ | N | !³ | !³ | !³ | N | Y | N |
| `BITMAP` | Y | Y⁷ | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N |

Notes:
Expand Down
10 changes: 6 additions & 4 deletions ...c/main/java/org/apache/flink/table/planner/functions/casting/VariantToStringCastRule.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,9 @@
* <p>Renders the scalar value the way a regular SQL cast of the stored kind would, so a boolean
* becomes {@code TRUE}, a timestamp uses the SQL format, and a {@code TIMESTAMP_LTZ} is shifted
* into the session time zone and a binary value is read as UTF-8. A variant holding an object or
* array has no scalar rendering and fails; use {@code JSON_STRING} for its JSON representation.
* array has no scalar form, so it renders like a regular {@code MAP} or {@code ARRAY} to string
* cast, an array as {@code [e1, e2]} and an object as {@code {k1=v1, k2=v2}}, with strings unquoted
* at every depth. This is a SQL rendering, not JSON; use {@code JSON_STRING} for the JSON form.
*
* <p>A binary value that is not well-formed UTF-8 fails instead of decoding to {@code U+FFFD}. Cast
* it to {@code BYTES} to inspect the raw value, or wrap that in {@code MAKE_VALID_UTF8} to accept
Expand Down Expand Up @@ -97,9 +99,9 @@ public String generateExpression(
LogicalType inputLogicalType,
LogicalType targetLogicalType) {
if (context.isPrinting()) {
// Every result has to be displayable, including an object or an array, which have no
// scalar rendering and would fail the cast. toJson returns a String, so it needs the
// wrap that toStringValue applies itself.
// Printing renders every variant as JSON, so a scalar string shows quoted rather than
// extracted as the cast below would. toJson returns a String, so it needs the wrap that
// toStringValue applies itself.
return staticCall(BINARY_STRING_DATA_FROM_STRING(), methodCall(inputTerm, "toJson"));
}
return staticCall(
Expand Down
41 changes: 33 additions & 8 deletions ...le-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -301,21 +301,27 @@ private static List<TestSetSpec> variantPrimitiveCasts() {
"CAST(PARSE_JSON('true') AS STRING)",
"TRUE",
STRING().notNull())
// An object or array has no scalar value, so the error points to
// JSON_STRING.
.testTableApiRuntimeError(
lit("[\"a\", \"b\"]").parseJson().cast(STRING()), "JSON_STRING")
// An object or array has no scalar form, so it renders like a regular ARRAY
// or MAP to string cast, with strings unquoted at every depth.
.testResult(
lit("[\"a\", \"b\"]").parseJson().cast(STRING()),
"CAST(PARSE_JSON('[\"a\", \"b\"]') AS STRING)",
"[a, b]",
STRING().notNull())
.testResult(
lit("[\"a\", \"b\"]").parseJson().tryCast(STRING()),
"TRY_CAST(PARSE_JSON('[\"a\", \"b\"]') AS STRING)",
null,
"[a, b]",
STRING())
.testTableApiRuntimeError(
lit("{\"a\": 1}").parseJson().cast(STRING()), "JSON_STRING")
.testResult(
lit("{\"a\": 1}").parseJson().cast(STRING()),
"CAST(PARSE_JSON('{\"a\": 1}') AS STRING)",
"{a=1}",
STRING().notNull())
.testResult(
lit("{\"a\": 1}").parseJson().tryCast(STRING()),
"TRY_CAST(PARSE_JSON('{\"a\": 1}') AS STRING)",
null,
"{a=1}",
STRING())
// A bounded CHAR/VARCHAR target trims a longer value, and CHAR pads a
// shorter one to its fixed width, the same as a regular cast into it.
Expand Down Expand Up @@ -354,6 +360,25 @@ private static List<TestSetSpec> variantPrimitiveCasts() {
"TRY_CAST(PARSE_JSON('\"ab\"') AS CHAR(5))",
"ab ",
CHAR(5))
// A nested variant null renders as NULL and a nested container renders in
// full, with strings unquoted, like a regular ARRAY or MAP to string cast.
.testResult(
lit("[\"a\", null, 1]").parseJson().cast(STRING()),
"CAST(PARSE_JSON('[\"a\", null, 1]') AS STRING)",
"[a, NULL, 1]",
STRING().notNull())
.testResult(
lit("{\"k\": [\"a\", \"b\"]}").parseJson().cast(STRING()),
"CAST(PARSE_JSON('{\"k\": [\"a\", \"b\"]}') AS STRING)",
"{k=[a, b]}",
STRING().notNull())
// A container renders in full and is then trimmed to a bounded target, the
// same as any other value longer than the target.
.testResult(
lit("[1, 2, 3]").parseJson().cast(VARCHAR(5)),
"CAST(PARSE_JSON('[1, 2, 3]') AS VARCHAR(5))",
"[1, 2",
VARCHAR(5).notNull())
// A variant holding a JSON null casts to SQL NULL, not to the text 'null'.
// The length of that text must not be checked against the target either.
.testResult(
Expand Down
31 changes: 26 additions & 5 deletions ...planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1847,11 +1847,32 @@ Stream<CastTestSpecBuilder> testCases() {
VARIANT(),
VARIANT_BUILDER.of(INVALID_UTF8),
TableRuntimeException.class)
// an object or an array has no scalar rendering and fails the cast
.fail(VARIANT(), VARIANT_ARRAY, TableRuntimeException.class)
.fail(VARIANT(), VARIANT_OBJECT, TableRuntimeException.class)
// printing is not a cast and cannot fail, so it renders JSON instead,
// which leaves a stored string quoted
// a byte value nested in a container is rendered unbounded, so its error
// names STRING rather than the container's target; the same holds for an
// object field value
.fail(
VARIANT(),
VARIANT_BUILDER
.array()
.add(VARIANT_BUILDER.of(INVALID_UTF8))
.build(),
TableRuntimeException.class,
"binary value to STRING")
.fail(
VARIANT(),
VARIANT_BUILDER
.object()
.add("k", VARIANT_BUILDER.of(INVALID_UTF8))
.build(),
TableRuntimeException.class,
"binary value to STRING")
// an object or an array has no scalar form, so it renders like a regular
// ARRAY or MAP to string cast, with strings unquoted and a nested null
// shown as NULL
.fromCase(VARIANT(), VARIANT_ARRAY, fromString("[1, two, FALSE, NULL]"))
.fromCase(VARIANT(), VARIANT_OBJECT, fromString("{k=[1, 2]}"))
// printing renders every variant as JSON instead, so a nested string is
// quoted and a null is the JSON null
.fromCasePrinting(
VARIANT(), VARIANT_ARRAY, fromString("[1,\"two\",false,null]"))
.fromCasePrinting(VARIANT(), VARIANT_OBJECT, fromString("{\"k\":[1,2]}"))
Expand Down
Loading