[SPARK-59401][SQL] Read INT96 timestamp columns as nanosecond timestamps - #58693
[SPARK-59401][SQL] Read INT96 timestamp columns as nanosecond timestamps#58693stevomitric wants to merge 2 commits into
Conversation
### What changes were proposed in this pull request? Extend the nanosecond-timestamp Parquet read path (which already reads INT64 TIMESTAMP(MICROS)) to also read a legacy INT96 timestamp column as a nanosecond type -- the read side of widening a TIMESTAMP(6) column stored as INT96 to nanosecond precision. INT96 carries no logical unit and decodes to microseconds (ParquetRowConverter.binaryToSQLTimestamp), so each value is promoted to the internal (epochMicros, nanosWithinMicro = 0) representation -- the same promotion the MICROS path uses. The requested type's family decides handling, mirroring the INT96 arms of ParquetRowConverter: the LTZ family applies the INT96 Julian rebase and any timezone conversion; the NTZ family applies neither. INT96 spans the full date range, so the promotion is range-complete. - Row-based: TimestampNanosParquetOps.newConverter accepts an INT96 column via a new int96AsNanosConverter (guarded by isInt96Timestamp). - Vectorized: a new Int96AsTimestampNanosUpdater, dispatched for a nanos type over an INT96 column. ### Why are the changes needed? INT96 is the default on-disk encoding for LTZ timestamps, so without this an INT96-encoded TIMESTAMP(6) column could not be widened to nanoseconds and read back. This complements the INT64 TIMESTAMP(MICROS) read support. ### Does this PR introduce _any_ user-facing change? Yes, behind the preview flag spark.sql.timestampNanosTypes.enabled: reading an INT96 timestamp column under a nanosecond timestamp type now succeeds (previously threw). ### How was this patch tested? ParquetTypeWideningSuite now covers INT96 -> LTZ nanos (moved from the unsupported list; TIMESTAMP_MILLIS stays unsupported), across vectorized/row-based readers and dictionary on/off. ### Was this patch authored or co-authored using generative AI tooling? Co-authored-by: Isaac <no-reply@databricks.com>
uros-b
left a comment
uros-b
left a comment
There was a problem hiding this comment.
[WIP][SPARK-56822][SQL] Read INT96 timestamp columns as nanosecond timestamps
Are we ready drop WIP here?
Also, SPARK-56822 doesn't look right here. Should it be SPARK-59284?
| SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true", | ||
| SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> ParquetOutputTimestampType.INT96.toString, | ||
| SQLConf.PARQUET_INT96_REBASE_MODE_IN_WRITE.key -> LegacyBehaviorPolicy.CORRECTED.toString, | ||
| SQLConf.PARQUET_INT96_REBASE_MODE_IN_READ.key -> LegacyBehaviorPolicy.CORRECTED.toString) { |
There was a problem hiding this comment.
Tests pin INT96 rebase to CORRECTED and use 2020 / 5138. The MICROS widening loop documents pre-1582 LEGACY rebase; INT96 needs the same, plus EXCEPTION failIfRebase. Dictionary decode is implemented, keep checkAllParquetReaders dict on/off.
There was a problem hiding this comment.
done, the INT96 read now runs under both CORRECTED and LEGACY rebase over a pre-1582 value.
| int96RebaseTz); | ||
| } | ||
| } | ||
| } else if (sparkType instanceof TimestampNTZNanosType) { |
There was a problem hiding this comment.
Spark writes NTZ as INT64 MICROS, but the read path still accepts INT96 as TimestampNTZNanosType. If that widening is intentional, please add a read test; otherwise drop the arm.
There was a problem hiding this comment.
this is reachable only through .schema(...). default is int64 for timestamp ntz. I've added a test.
…/NTZ reads Address review feedback on reading INT96 timestamp columns as a nanosecond type. INT96 stores nanoseconds-of-day, so a foreign file (e.g. Impala/Hive) can carry true sub-microsecond digits. binaryToSQLTimestamp floors to micros, so the read now recovers the remainder straight from the raw INT96 (timeOfDayNanos % 1000) and truncates it to the read precision via the shared DateTimeUtils.truncateNanosWithinMicroToPrecision, instead of hardcoding nanosWithinMicro = 0. A whole-microsecond rebase/timezone shift never perturbs the remainder. Applied to both readers; the vectorized updater now carries the read precision. Other changes: - Build the INT96 rebase closure only for the LTZ family (identity for NTZ), matching microsAsNanosConverter. - Clarify that INT96 has no time-zone family, so it can be requested as either LTZ or NTZ nanos (mirroring Spark's existing INT96 -> TimestampType / TimestampNTZType reads); the same-family guard applies only to the annotated micros path. Comments no longer claim INT96 has no sub-microsecond digits. Tests: - ParquetTypeWideningSuite now runs the INT96 read under both CORRECTED and LEGACY rebase over a pre-1582 value, keeping dictionary on/off. - TimestampNanosParquetOpsSuite gains INT96 unit tests: sub-microsecond preservation (both families), precision truncation, LTZ rebase (LEGACY differs from CORRECTED, EXCEPTION throws) and timezone conversion, and NTZ ignoring both. Co-authored-by: Isaac <no-reply@databricks.com>
stevomitric
changed the title
| expectError = false) | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
ParquetVectorUpdaterFactory.java (Int96AsTimestampNanosUpdater) / ParquetTypeWideningSuite.scala -- The new vectorized updater is exercised end-to-end only for the LTZ family with micro-aligned, Spark-written values (CORRECTED + LEGACY rebase, dictionary on/off, vectorized on/off, via checkAllParquetReaders). Its sub-microsecond recovery -- the PR's headline behavior ("a foreign nanosecond INT96 is not silently floored to micros") -- plus its NTZ arm and its EXCEPTION (failIfRebase) arm have no vectorized coverage: Spark only writes micro-aligned INT96, so the widening loop always yields nanosWithinMicro == 0, and those corners are pinned only on the row-based converter (TimestampNanosParquetOpsSuite.decodeInt96), while the default vectorized reader runs the parallel putInt96AsNanos implementation with a non-zero remainder untested. This maps onto uros-b's inline ask to test the vectorized NTZ arm (ParquetVectorUpdaterFactory.java:232), which was answered with a row-based unit test rather than one that drives the vectorized updater. A vectorized-reader test carrying a hand-crafted sub-microsecond INT96 (and an NTZ variant) would pin the default path's headline behavior. The code itself is verified correct against the row path and the existing INT96->Timestamp updaters, so this is a coverage-rigor gap on a new code path, not a suspected defect.
2 participants
What changes were proposed in this pull request?
Extend the nanosecond-timestamp Parquet read path (which already reads INT64 TIMESTAMP(MICROS)) to also read a legacy INT96 timestamp column as a nanosecond type - the read side of widening a TIMESTAMP(6) column stored as INT96 to nanosecond precision.
INT96 carries no logical unit and decodes to microseconds (ParquetRowConverter.binaryToSQLTimestamp), so each value is promoted to the internal (epochMicros, nanosWithinMicro = 0) representation - the same promotion the MICROS path uses. The requested type's family decides handling, mirroring the INT96 arms of ParquetRowConverter: the LTZ family applies the INT96 Julian rebase and any timezone conversion; the NTZ family applies neither. INT96 spans the full date range, so the promotion is range-complete.
Why are the changes needed?
INT96 is the default on-disk encoding for LTZ timestamps, so without this an INT96-encoded TIMESTAMP(6) column could not be widened to nanoseconds and read back. This complements the INT64 TIMESTAMP(MICROS) read support.
Does this PR introduce any user-facing change?
Yes, behind the preview flag spark.sql.timestampNanosTypes.enabled, reading an INT96 timestamp column under a nanosecond timestamp type now succeeds (previously threw).
How was this patch tested?
extended ParquetTypeWideningSuite.
Was this patch authored or co-authored using generative AI tooling?
Co-authored-by: Claude Code 4.8