Skip to content

[deps] Upgrade Hudi to 0.14.1 - #878

Open
the-other-tim-brown wants to merge 1 commit into
apache:branch-0.4from
the-other-tim-brown:hudi-0.14.2
Open

[deps] Upgrade Hudi to 0.14.1#878
the-other-tim-brown wants to merge 1 commit into
apache:branch-0.4from
the-other-tim-brown:hudi-0.14.2

Conversation

@the-other-tim-brown

@the-other-tim-brown the-other-tim-brown commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

What

Bumps the hudi.version property on branch-0.4 from 0.14.0 to 0.14.1.

Note on the branch name. This branch is still called hudi-0.14.2 from the original attempt. GitHub does not allow retargeting an open PR's head branch, so the name is stale — the change itself is 0.14.00.14.1.

Why not 0.14.2

0.14.2 was the original target, but it regresses the column stats index and breaks TestHudiFileStatsExtractor.columnStatsWithMetadataTable (9 indexed columns → 4).

The cause is apache/hudi#18132 "fix: timestamp logical types" (merged 2026-04-10), a combined backport of #13711 and #14161. It rewrote HoodieTableMetadataUtil.getColumnsToIndex to filter configured columns through:

Schema.Field field = writerSchema.getField(colName);
return field != null && !isTimestampMillisField(field.schema());

Schema.getField() matches top-level field names only. Any nested column configured in dot notation via hoodie.metadata.index.column.stats.column.list resolves to null and is silently dropped from the index. TestAbstractHudiTable configures exactly such a list, deliberately — "all the columns in dot notation so that all columns will have stats saved to metadata table for more thorough testing."

Verified against the test schema — three column-list configurations, all yielding the same result on 0.14.2:

column list configuration columns indexed
dot-notation leaf paths (current test behavior) 4
top-level field names only 4
not configured (Hudi default) 4

The survivors are exactly the top-level primitives that are not timestamp-millis: long_field, key, date_field, decimal_field. Dropped are the four nested paths plus timestamp_field. No configuration recovers them — the only code path that skips the filter is .orElse(targetColumns), reached only when the writer schema fails to resolve.

Excluding timestamp-millis columns is intended and is shared with mainline. Dropping nested columns is not. Hudi master resolves the same lookup with a nested-aware helper:

Option<Pair<String, HoodieSchemaField>> fieldPairOpt = HoodieSchemaUtils.getNestedField(tableSchema, indexCol);
return fieldPairOpt.isPresent() && !isTimestampMillisField(fieldPairOpt.get().getRight().schema());

So this is a defect specific to the 0.14.x backport, which substituted the top-level-only getField. It is not tracked upstream: PR #18132 carries no JIRA key, ASF JIRA lists exactly one issue against fixVersion = 0.14.2 (HUDI-8066, an unrelated Flink cherry-pick), and the Hudi website publishes no release notes for 0.14.1 or 0.14.2. Since 0.14.x is dormant and the bug does not exist on mainline, no fix is pending.

0.14.1 predates all of this and is unaffected.

Trade-off

Staying on 0.14.1 forgoes 0.14.2's timestamp logical-type correctness fixes for tables whose timestamp-millis columns were mislabeled as micros. That was judged the better trade for a maintenance branch than silently losing column stats coverage. Happy to revisit if reviewers weigh it differently.

Scope

One-line change. Every Hudi coordinate in the reactor already resolves through ${hudi.version}xtable-core, xtable-aws, xtable-hive-metastore, xtable-hudi-support-extensions, xtable-utilities, and xtable-spark-runtime all inherit it, so no other file needs updating.

Testing

xtable-core test suite passes locally against 0.14.1, including all 4 tests in TestHudiFileStatsExtractor, with all 9 expected columns present in the index. No test changes were needed. CI covers the full ./mvnw clean install suite plus the Spark 3.4/3.5 xtable-spark-runtime bundle ITs.

Intentionally not changed

  • The docs (features-and-limitations.md, athena.md, glue-catalog.md, spark.md, how-to.md) state Hudi 0.14.0 is required when reading a Hudi target table. That is a minimum-reader-version statement, not the build dependency, so raising it would incorrectly narrow what users can read with.
  • demo/notebook/demo.ipynb pins Hudi 0.14.0 but is already stale (it references xtable-*-0.2.0-SNAPSHOT.jar). Updating only its Hudi line would be a half-fix; happy to refresh it separately.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CyVF3fyK2VQE7DPd31mRLv

Bump the hudi.version property from 0.14.0 to 0.14.1 on branch-0.4.

0.14.2 was evaluated first but regresses the column stats index. Its
getColumnsToIndex filter resolves configured column names with
Schema.getField(), which matches top-level names only, so any nested
column configured via hoodie.metadata.index.column.stats.column.list is
silently dropped. 0.14.1 does not contain that code.

All Hudi coordinates across the reactor resolve through this single
property, so no other changes are required.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CyVF3fyK2VQE7DPd31mRLv
@the-other-tim-brown the-other-tim-brown changed the title [deps] Upgrade Hudi to 0.14.2 [deps] Upgrade Hudi to 0.14.1 Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant