diff --git a/docs/content.zh/docs/sql/reference/data-types.md b/docs/content.zh/docs/sql/reference/data-types.md index 9bc67dd733e92..fa20d4c273f03 100644 --- a/docs/content.zh/docs/sql/reference/data-types.md +++ b/docs/content.zh/docs/sql/reference/data-types.md @@ -1583,10 +1583,11 @@ CAST(CAST(PARSE_JSON('3.9') AS DECIMAL(2, 1)) AS INT) -- returns 3 (truncate A cast to a character string renders the value exactly as a regular SQL cast of the stored kind would, so a boolean becomes `TRUE`, a timestamp uses the SQL format, a `TIMESTAMP_LTZ` is shifted into -the session time zone, and a binary value is read as UTF-8. Only an object or an array has no such -rendering. Use `JSON_STRING` for the JSON representation instead, where a string stays quoted as -`"foo"` and an object or array is serialized. A variant that stores a JSON `null` casts to SQL -`NULL`. +the session time zone, and a binary value is read as UTF-8. An object or an array has no scalar +form, so it renders like a regular `ARRAY` or `MAP` cast to a string: an array as `[e1, e2]` and an +object as `{k1=v1, k2=v2}`, with each value rendered by these same rules and a nested variant null +shown as `NULL`. A string is never quoted, at any depth. Use `JSON_STRING` for the JSON form with +quoted strings. A variant that stores a JSON `null` casts to SQL `NULL`. A `VARIANT` can also be cast to a constructed target, which imposes a schema on it. A variant array casts to `ARRAY`. The variant must be an array, otherwise the cast fails. Each element is itself a @@ -1873,7 +1874,7 @@ COALESCE(TRY_CAST('non-number' AS INT), 0) --- 结果返回数字 0 的 INT 格 | `ROW` | Y | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | !³ | N | N | N | N | | `STRUCTURED` | Y | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | !³ | N | N | N | | `RAW` | Y | ! | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | Y⁴ | N | N | -| `VARIANT` | N | ! | ! | ! | ! | ! | ! | ! | ! | ! | ! | N | ! | ! | N | !³ | N | !³ | !³ | !³ | N | Y | N | +| `VARIANT` | ! | ! | ! | ! | ! | ! | ! | ! | ! | ! | ! | N | ! | ! | N | !³ | N | !³ | !³ | !³ | N | Y | N | | `BITMAP` | Y | Y⁷ | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | 备注： diff --git a/docs/content/docs/sql/reference/data-types.md b/docs/content/docs/sql/reference/data-types.md index 8e43c2ec12ee2..e62f1997d29b0 100644 --- a/docs/content/docs/sql/reference/data-types.md +++ b/docs/content/docs/sql/reference/data-types.md @@ -1591,10 +1591,11 @@ CAST(CAST(PARSE_JSON('3.9') AS DECIMAL(2, 1)) AS INT) -- returns 3 (truncate A cast to a character string renders the value exactly as a regular SQL cast of the stored kind would, so a boolean becomes `TRUE`, a timestamp uses the SQL format, a `TIMESTAMP_LTZ` is shifted into -the session time zone, and a binary value is read as UTF-8. Only an object or an array has no such -rendering. Use `JSON_STRING` for the JSON representation instead, where a string stays quoted as -`"foo"` and an object or array is serialized. A variant that stores a JSON `null` casts to SQL -`NULL`. +the session time zone, and a binary value is read as UTF-8. An object or an array has no scalar +form, so it renders like a regular `ARRAY` or `MAP` cast to a string: an array as `[e1, e2]` and an +object as `{k1=v1, k2=v2}`, with each value rendered by these same rules and a nested variant null +shown as `NULL`. A string is never quoted, at any depth. Use `JSON_STRING` for the JSON form with +quoted strings. A variant that stores a JSON `null` casts to SQL `NULL`. A `VARIANT` can also be cast to a constructed target, which imposes a schema on it. A variant array casts to `ARRAY`. The variant must be an array, otherwise the cast fails. Each element is itself a @@ -1882,7 +1883,7 @@ The matrix below describes the supported cast pairs, where "Y" means supported, | `ROW` | Y | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | !³ | N | N | N | N | | `STRUCTURED` | Y | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | !³ | N | N | N | | `RAW` | Y | ! | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | Y⁴ | N | N | -| `VARIANT` | N | ! | ! | ! | ! | ! | ! | ! | ! | ! | ! | N | ! | ! | N | !³ | N | !³ | !³ | !³ | N | Y | N | +| `VARIANT` | ! | ! | ! | ! | ! | ! | ! | ! | ! | ! | ! | N | ! | ! | N | !³ | N | !³ | !³ | !³ | N | Y | N | | `BITMAP` | Y | Y⁷ | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | N | Notes: diff --git a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToStringCastRule.java b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToStringCastRule.java index eeb9fcec9f651..08cef8e0b3f35 100644 --- a/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToStringCastRule.java +++ b/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/casting/VariantToStringCastRule.java @@ -36,7 +36,9 @@ *

Renders the scalar value the way a regular SQL cast of the stored kind would, so a boolean * becomes {@code TRUE}, a timestamp uses the SQL format, and a {@code TIMESTAMP_LTZ} is shifted * into the session time zone and a binary value is read as UTF-8. A variant holding an object or - * array has no scalar rendering and fails; use {@code JSON_STRING} for its JSON representation. + * array has no scalar form, so it renders like a regular {@code MAP} or {@code ARRAY} to string + * cast, an array as {@code [e1, e2]} and an object as {@code {k1=v1, k2=v2}}, with strings unquoted + * at every depth. This is a SQL rendering, not JSON; use {@code JSON_STRING} for the JSON form. * *

A binary value that is not well-formed UTF-8 fails instead of decoding to {@code U+FFFD}. Cast * it to {@code BYTES} to inspect the raw value, or wrap that in {@code MAKE_VALID_UTF8} to accept @@ -97,9 +99,9 @@ public String generateExpression( LogicalType inputLogicalType, LogicalType targetLogicalType) { if (context.isPrinting()) { - // Every result has to be displayable, including an object or an array, which have no - // scalar rendering and would fail the cast. toJson returns a String, so it needs the - // wrap that toStringValue applies itself. + // Printing renders every variant as JSON, so a scalar string shows quoted rather than + // extracted as the cast below would. toJson returns a String, so it needs the wrap that + // toStringValue applies itself. return staticCall(BINARY_STRING_DATA_FROM_STRING(), methodCall(inputTerm, "toJson")); } return staticCall( diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java index b73054fbbf145..7dbbc379c67dd 100644 --- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java +++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/CastFunctionITCase.java @@ -301,21 +301,27 @@ private static List variantPrimitiveCasts() { "CAST(PARSE_JSON('true') AS STRING)", "TRUE", STRING().notNull()) - // An object or array has no scalar value, so the error points to - // JSON_STRING. - .testTableApiRuntimeError( - lit("[\"a\", \"b\"]").parseJson().cast(STRING()), "JSON_STRING") + // An object or array has no scalar form, so it renders like a regular ARRAY + // or MAP to string cast, with strings unquoted at every depth. + .testResult( + lit("[\"a\", \"b\"]").parseJson().cast(STRING()), + "CAST(PARSE_JSON('[\"a\", \"b\"]') AS STRING)", + "[a, b]", + STRING().notNull()) .testResult( lit("[\"a\", \"b\"]").parseJson().tryCast(STRING()), "TRY_CAST(PARSE_JSON('[\"a\", \"b\"]') AS STRING)", - null, + "[a, b]", STRING()) - .testTableApiRuntimeError( - lit("{\"a\": 1}").parseJson().cast(STRING()), "JSON_STRING") + .testResult( + lit("{\"a\": 1}").parseJson().cast(STRING()), + "CAST(PARSE_JSON('{\"a\": 1}') AS STRING)", + "{a=1}", + STRING().notNull()) .testResult( lit("{\"a\": 1}").parseJson().tryCast(STRING()), "TRY_CAST(PARSE_JSON('{\"a\": 1}') AS STRING)", - null, + "{a=1}", STRING()) // A bounded CHAR/VARCHAR target trims a longer value, and CHAR pads a // shorter one to its fixed width, the same as a regular cast into it. @@ -354,6 +360,25 @@ private static List variantPrimitiveCasts() { "TRY_CAST(PARSE_JSON('\"ab\"') AS CHAR(5))", "ab ", CHAR(5)) + // A nested variant null renders as NULL and a nested container renders in + // full, with strings unquoted, like a regular ARRAY or MAP to string cast. + .testResult( + lit("[\"a\", null, 1]").parseJson().cast(STRING()), + "CAST(PARSE_JSON('[\"a\", null, 1]') AS STRING)", + "[a, NULL, 1]", + STRING().notNull()) + .testResult( + lit("{\"k\": [\"a\", \"b\"]}").parseJson().cast(STRING()), + "CAST(PARSE_JSON('{\"k\": [\"a\", \"b\"]}') AS STRING)", + "{k=[a, b]}", + STRING().notNull()) + // A container renders in full and is then trimmed to a bounded target, the + // same as any other value longer than the target. + .testResult( + lit("[1, 2, 3]").parseJson().cast(VARCHAR(5)), + "CAST(PARSE_JSON('[1, 2, 3]') AS VARCHAR(5))", + "[1, 2", + VARCHAR(5).notNull()) // A variant holding a JSON null casts to SQL NULL, not to the text 'null'. // The length of that text must not be checked against the target either. .testResult( diff --git a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java index 274626557d172..440ff8a5826ac 100644 --- a/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java +++ b/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/functions/casting/CastRulesTest.java @@ -1847,11 +1847,32 @@ Stream testCases() { VARIANT(), VARIANT_BUILDER.of(INVALID_UTF8), TableRuntimeException.class) - // an object or an array has no scalar rendering and fails the cast - .fail(VARIANT(), VARIANT_ARRAY, TableRuntimeException.class) - .fail(VARIANT(), VARIANT_OBJECT, TableRuntimeException.class) - // printing is not a cast and cannot fail, so it renders JSON instead, - // which leaves a stored string quoted + // a byte value nested in a container is rendered unbounded, so its error + // names STRING rather than the container's target; the same holds for an + // object field value + .fail( + VARIANT(), + VARIANT_BUILDER + .array() + .add(VARIANT_BUILDER.of(INVALID_UTF8)) + .build(), + TableRuntimeException.class, + "binary value to STRING") + .fail( + VARIANT(), + VARIANT_BUILDER + .object() + .add("k", VARIANT_BUILDER.of(INVALID_UTF8)) + .build(), + TableRuntimeException.class, + "binary value to STRING") + // an object or an array has no scalar form, so it renders like a regular + // ARRAY or MAP to string cast, with strings unquoted and a nested null + // shown as NULL + .fromCase(VARIANT(), VARIANT_ARRAY, fromString("[1, two, FALSE, NULL]")) + .fromCase(VARIANT(), VARIANT_OBJECT, fromString("{k=[1, 2]}")) + // printing renders every variant as JSON instead, so a nested string is + // quoted and a null is the JSON null .fromCasePrinting( VARIANT(), VARIANT_ARRAY, fromString("[1,\"two\",false,null]")) .fromCasePrinting(VARIANT(), VARIANT_OBJECT, fromString("{\"k\":[1,2]}")) diff --git a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java index f4920a4d3532c..ddd86cdda64da 100644 --- a/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java +++ b/flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/functions/VariantCastUtils.java @@ -263,36 +263,74 @@ public static byte[] toBytes(Variant variant, int targetLength, boolean fixedLen } /** - * Casts a scalar {@code VARIANT} to a character string, rendering the value the way a regular - * SQL cast of the stored kind would. A value longer than {@code targetLength} is trimmed, and a - * {@code CHAR} target pads a shorter value to its fixed width. Both are measured in code - * points, so a character outside the BMP fills a single position even though it occupies two - * UTF-16 units. - * - *

A stored binary value has to be well-formed UTF-8, since a character string cannot carry - * bytes that no character maps to. Invalid input is rejected rather than decoded into {@code - * U+FFFD}, which would silently substitute a character the value never held. + * Casts a {@code VARIANT} to a character string, rendering the stored kind the way a regular + * SQL cast would. An object or array renders like a {@code MAP} or {@code ARRAY} cast, an array + * as {@code [e1, e2]} and an object as {@code {k1=v1, k2=v2}}, with a nested null shown as + * {@code NULL}. Strings are never quoted, at any depth; this is a SQL rendering, not JSON, so + * use {@code JSON_STRING} for the JSON form. A value longer than {@code targetLength} is + * trimmed and a {@code CHAR} target pads a shorter one, both counted in code points rather than + * UTF-16 units. A binary value must be well-formed UTF-8, and is rejected rather than decoded + * into {@code U+FFFD}. * * @param sessionZone the session time zone, applied to a {@code TIMESTAMP_LTZ} value */ public static BinaryStringData toStringValue( Variant variant, TimeZone sessionZone, int targetLength, boolean charTarget) { - final String value = getVariantTypeAsString(variant, sessionZone, targetLength, charTarget); + final String value = renderValue(variant, sessionZone, targetLength, charTarget); // numChars and substring both count code points, so a character outside the BMP fills one // position rather than the two UTF-16 units it occupies. - final BinaryStringData result = BinaryStringData.fromString(value); - final int length = result.numChars(); - if (length > targetLength) { - return result.substring(0, targetLength); + return variantKey(value, targetLength, charTarget); + } + + /** + * Renders a variant as a character string. An array becomes {@code [e1, e2]} and an object + * becomes {@code {k1=v1, k2=v2}}, matching how a regular {@code ARRAY} or {@code MAP} casts to + * a string. Elements and field values recurse through the same rendering, so a string stays + * unquoted at every depth. A scalar renders like a regular cast of its stored kind. + */ + private static String renderValue( + final Variant variant, + final TimeZone sessionZone, + final int targetLength, + final boolean charTarget) { + if (variant.isArray()) { + final int size = variant.getArraySize(); + final StringBuilder sb = new StringBuilder(); + sb.append('['); + for (int i = 0; i < size; i++) { + if (i > 0) { + sb.append(", "); + } + sb.append(renderElement(variant.getElement(i), sessionZone)); + } + return sb.append(']').toString(); } - if (charTarget && length < targetLength) { - return BinaryStringDataUtil.concat( - result, BinaryStringData.blankString(targetLength - length)); + if (variant.isObject()) { + final StringBuilder sb = new StringBuilder(); + sb.append('{'); + boolean first = true; + for (final String fieldName : variant.getFieldNames()) { + if (!first) { + sb.append(", "); + } + first = false; + sb.append(fieldName) + .append('=') + .append(renderElement(variant.getField(fieldName), sessionZone)); + } + return sb.append('}').toString(); } - return result; + return renderScalar(variant, sessionZone, targetLength, charTarget); } - private static String getVariantTypeAsString( + /** Renders one array element or object field value; a nested null shows as {@code NULL}. */ + private static String renderElement(final Variant element, final TimeZone sessionZone) { + return element.isNull() + ? "NULL" + : renderValue(element, sessionZone, Integer.MAX_VALUE, false); + } + + private static String renderScalar( final Variant variant, final TimeZone sessionZone, final int targetLength, @@ -324,6 +362,7 @@ private static String getVariantTypeAsString( final int invalidAt = StringUtf8Utils.firstInvalidUtf8ByteIndex(utf8, 0, utf8.length); if (invalidAt >= 0) { + final String targetDescription = characterTarget(targetLength, charTarget); throw new TableRuntimeException( String.format( "Cannot cast the VARIANT binary value to %s because it is not " @@ -331,9 +370,7 @@ private static String getVariantTypeAsString( + "of %d. Cast to BYTES to inspect the raw value, or " + "wrap that in MAKE_VALID_UTF8 to replace every " + "invalid byte with the U+FFFD replacement character.", - characterTarget(targetLength, charTarget), - invalidAt, - utf8.length)); + targetDescription, invalidAt, utf8.length)); } value = new String(utf8, StandardCharsets.UTF_8); break; @@ -359,13 +396,14 @@ private static String getVariantTypeAsString( case NULL: // Only reachable for a NOT NULL target. A nullable target maps a null-valued // variant to SQL NULL before this method is called. + final String targetDescription = characterTarget(targetLength, charTarget); throw new TableRuntimeException( String.format( "Cannot cast a VARIANT null value to %s because the target does not " + "accept NULL.", - characterTarget(targetLength, charTarget))); + targetDescription)); default: - // An object or array has no scalar rendering. + // Any remaining kind has no scalar string rendering; JSON_STRING serializes it. throw new TableRuntimeException( String.format( "Cannot cast a VARIANT %s value to a character string. Use the " @@ -376,7 +414,13 @@ private static String getVariantTypeAsString( } private static String characterTarget(int targetLength, boolean charTarget) { - return String.format("%s(%d)", charTarget ? "CHAR" : "VARCHAR", targetLength); + if (charTarget) { + return String.format("%s(%d)", "CHAR", targetLength); + } + if (targetLength == Integer.MAX_VALUE) { + return "STRING"; + } + return String.format("%s(%d)", "VARCHAR", targetLength); } private static Number numeric(Variant variant, String targetType) {