XTable 4th Release

[vinishjail97]

XTable 4th release updates (0.4.0-incubating).

Release PRs:

• fix(release): exclude xtable-service/examples from source tarball and repair deploy_staging_jars.sh #852 — Fix release scripts (exclude xtable-service/examples from source tarball; repair deploy_staging_jars.sh)
• fix(build): restore delombok execution so release builds generate javadoc jars #853 — Restore delombok execution so release builds generate javadoc jars

[vinishjail97]

Release Notes for release-0.4.0-incubating-rc1

Apache XTable™ (Incubating) is a cross-table converter for table formats that facilitates omni-directional interoperability across data processing systems and query engines. Currently, Apache XTable™ supports widely adopted open-source table formats such as Apache Hudi, Apache Iceberg, Delta Lake, and (as a source) Apache Paimon.

✨ What's Changed

🔄 Core Functionality Enhancements

• New formats:
  • Added a Delta Kernel conversion target (Delta kernel conversion target #801, Delta Kernel Draft PR #729).
  • Added Apache Paimon source support, with incremental sync and column stats (Paimon Source Support #742, Implement Paimon Source Incremental Sync #780, [755] Support column stats for paimon #767).
  • Added Apache Parquet source support, with partition extraction and incremental sync ([553] parquet source with partition extraction #728, [553] Parquet Schema and Column Stats Converters #669, Parquet Incremental Sync #768, Parquet source: Expand column stats support, fix bugs in schema conversion #805).
• Sync improvements:
  • Delta source: extract nested field statistics (Delta source: extract nested field's stats #769).
  • Nested-column partition support in HudiPartitionValuesExtractor with hive-style partitioning ([739] Nested column partition support in HudiPartitionValuesExtractor when hive style partitioning is enabled #738).
  • Fall back to parquet footers when metadata-table column stats are missing (Fallback to parquet footers when metadata table col stats are missing #818).
  • Enable Delta checkpoint CRC exception throwing by default (Enable Delta checkpoint CRC exception throwing by default #817).
  • Add additional properties when converting for Hudi (Add additional properties when doing conversion for hudi #714).
• Bug fixes:
  • Delta: null partition value for composite generated-column partitions (fix(delta): null partition value for composite generated-column partitions #828), map-key path handling (Fix map key path handling in Delta schema extractors #826), NPE for binary in map/array schemas (fix(delta): avoid NPE for binary in map/array schemas #795), column-mapping-enabled source tables (Fix Delta to Iceberg not working on column mapping enabled Delta source table #766).
  • Hudi: batch INSERT_OVERWRITE replacecommits dropping adds ([bug] Fix batch INSERT_OVERWRITE replacecommits dropping adds in HudiDataFileExtractor #816), empty EarliestCommitToRetain (Handle empty EarliestCommitToRetain #732).
  • Iceberg: nested comments with qualified name (fix(iceberg): nested comments with qualified name #797), schema sync ([750] Fix schema sync for Iceberg tables #749), empty source table (Handle empty Iceberg table source in conversion #756).
  • Parquet: snapshot sync on multiple commits with partitions ([bug] Parquet Source: snapshot sync fails on multiple commits with partitions #806), schema-conversion bugs (Parquet source: Expand column stats support, fix bugs in schema conversion #805).

📦 Packaging & Distribution

• Added xtable-spark-runtime, a thin drop-in Spark bundle for Spark 3.4 and 3.5 ([836] xtable-spark-runtime: thin drop-in Spark bundle for 0.4.x (Spark 3.4 + 3.5) #843).
• Made the bundled xtable-utilities jar runnable again ([736] Make the bundled xtable-utilities jar runnable again (Hudi 0.x) + CI on branch-0.4 #841, Missing org.openjdk.jol from imported dependencies in xtable-utilities makes it to fail with NoClassDefFoundError #736).
• Fixed the scope of spark-catalyst in the utilities bundle ([789] Fix scope of spark catalyst in utilities bundle #790).

🧪 Testing & Quality

• Added an initial integration test for the XTable REST service (Add intial IT for XTable REST Service #716).
• Improved TestHudiInstantUtils with parameterization and more cases (Improved all tests in TestHudiInstantUtils by parameterising and adding more test cases #703).
• Removed wildcard imports and enforced with Spotless (Remove wildcard imports and enforce with spotless #809).
• Removed redundant getSnapshotAt calls per commit (Remove redundant getSnapshotAt calls per commit #791).

📘 Documentation & Community

• Added a 0.3.0 release page ([DOCS] Add a new 0.3.0 release page #725) and updated the community sync page (Update community sync page #734).

🧰 Infrastructure & Maintenance

• Upgraded Apache Iceberg to 1.9.2 and Avro to 1.12.0 (Upgrade Iceberg to 1.9.2 and Avro to 1.12.0 #784).
• Added a Maven wrapper for consistent builds ([787] Add maven wrapper for consistent builds #788).
• Automated license file generation and validation (Automate license file generation and validation #822); removed a duplicate javassist LICENSE entry ([LICENSE] Remove duplicate entry for org.javassist:javassist:3.25.0-G… #745).
• Restored the delombok execution so release builds generate javadoc jars (fix(build): restore delombok execution so release builds generate javadoc jars #853).
• Added .sdkmanrc to .gitignore (add .sdkmanrc to .gitignore #793); bumped GitHub Actions (checkout, setup-node, setup-java).

Detailed Change Log

• [553] Parquet Schema and Column Stats Converters by @sapienza88 in [553] Parquet Schema and Column Stats Converters #669
• Add additional properties when doing conversion for hudi by @rahil-c in Add additional properties when doing conversion for hudi #714
• Improved all tests in TestHudiInstantUtils by parameterising and adding more test cases by @Monilnarang in Improved all tests in TestHudiInstantUtils by parameterising and adding more test cases #703
• Fix typo in InternalTable by @vinishjail97 in Fix typo in InternalTable #720
• [DOCS] Add a new 0.3.0 release page by @sagarlakshmipathy in [DOCS] Add a new 0.3.0 release page #725
• Add intial IT for XTable REST Service by @rahil-c in Add intial IT for XTable REST Service #716
• Update community sync page by @pratyakshsharma in Update community sync page #734
• Bump actions/setup-java from 4 to 5 by @dependabot[bot] in Bump actions/setup-java from 4 to 5 #740
• [739] Nested column partition support in HudiPartitionValuesExtractor when hive style partitioning is enabled by @kroushan-nit in [739] Nested column partition support in HudiPartitionValuesExtractor when hive style partitioning is enabled #738
• Bump actions/setup-node from 4 to 5 by @dependabot[bot] in Bump actions/setup-node from 4 to 5 #741
• Bump actions/checkout from 4 to 5 by @dependabot[bot] in Bump actions/checkout from 4 to 5 #737
• [553] parquet source with partition extraction by @sapienza88 in [553] parquet source with partition extraction #728
• [LICENSE] Remove duplicate entry for org.javassist:javassist:3.25.0-G… by @piyushdubey in [LICENSE] Remove duplicate entry for org.javassist:javassist:3.25.0-G… #745
• [750] Fix schema sync for Iceberg tables by @the-other-tim-brown in [750] Fix schema sync for Iceberg tables #749
• Paimon Source Support by @mikedias in Paimon Source Support #742
• Handle empty Iceberg table source in conversion by @brishi19791 in Handle empty Iceberg table source in conversion #756
• Bump actions/checkout from 5 to 6 by @dependabot[bot] in Bump actions/checkout from 5 to 6 #761
• Delta Kernel Draft PR by @vaibhavk1992 in Delta Kernel Draft PR #729
• Delta source: extract nested field's stats by @emilie-wang in Delta source: extract nested field's stats #769
• Fix Delta to Iceberg not working on column mapping enabled Delta source table by @xr-chen in Fix Delta to Iceberg not working on column mapping enabled Delta source table #766
• Implement Paimon Source Incremental Sync by @mikedias in Implement Paimon Source Incremental Sync #780
• [787] Add maven wrapper for consistent builds by @the-other-tim-brown in [787] Add maven wrapper for consistent builds #788
• [755] Support column stats for paimon by @mao-liu in [755] Support column stats for paimon #767
• [789] Fix scope of spark catalyst in utilities bundle by @the-other-tim-brown in [789] Fix scope of spark catalyst in utilities bundle #790
• Upgrade Iceberg to 1.9.2 and Avro to 1.12.0 by @kevinjqliu in Upgrade Iceberg to 1.9.2 and Avro to 1.12.0 #784
• fix(delta): avoid NPE for binary in map/array schemas by @parisni in fix(delta): avoid NPE for binary in map/array schemas #795
• Remove redundant getSnapshotAt calls per commit by @brishi19791 in Remove redundant getSnapshotAt calls per commit #791
• fix(iceberg): nested comments with qualified name by @parisni in fix(iceberg): nested comments with qualified name #797
• add .sdkmanrc to .gitignore by @kevinjqliu in add .sdkmanrc to .gitignore #793
• Remove wildcard imports and enforce with spotless by @the-other-tim-brown in Remove wildcard imports and enforce with spotless #809
• Parquet source: Expand column stats support, fix bugs in schema conversion by @the-other-tim-brown in Parquet source: Expand column stats support, fix bugs in schema conversion #805
• [bug] Parquet Source: snapshot sync fails on multiple commits with partitions by @the-other-tim-brown in [bug] Parquet Source: snapshot sync fails on multiple commits with partitions #806
• Enable Delta checkpoint CRC exception throwing by default by @vinishjail97 in Enable Delta checkpoint CRC exception throwing by default #817
• Bump actions/setup-node from 5 to 6 by @dependabot[bot] in Bump actions/setup-node from 5 to 6 #753
• Handle empty EarliestCommitToRetain by @alexr17 in Handle empty EarliestCommitToRetain #732
• [bug] Fix batch INSERT_OVERWRITE replacecommits dropping adds in HudiDataFileExtractor by @vinishjail97 in [bug] Fix batch INSERT_OVERWRITE replacecommits dropping adds in HudiDataFileExtractor #816
• Fallback to parquet footers when metadata table col stats are missing by @vinishjail97 in Fallback to parquet footers when metadata table col stats are missing #818
• Automate license file generation and validation by @the-other-tim-brown in Automate license file generation and validation #822
• Delta kernel conversion target by @vaibhavk1992 in Delta kernel conversion target #801
• Fix map key path handling in Delta schema extractors by @brishi19791 in Fix map key path handling in Delta schema extractors #826
• Parquet Incremental Sync by @sapienza88 in Parquet Incremental Sync #768
• [MINOR] Preserve original cause in ReadException on close failure by @alealandreev in [MINOR] Preserve original cause in ReadException on close failure #827
• fix(delta): null partition value for composite generated-column partitions by @alealandreev in fix(delta): null partition value for composite generated-column partitions #828
• [736] Make the bundled xtable-utilities jar runnable again (Hudi 0.x) + CI on branch-0.4 by @vinishjail97 in [736] Make the bundled xtable-utilities jar runnable again (Hudi 0.x) + CI on branch-0.4 #841
• [836] xtable-spark-runtime: thin drop-in Spark bundle for 0.4.x (Spark 3.4 + 3.5) by @vinishjail97 in [836] xtable-spark-runtime: thin drop-in Spark bundle for 0.4.x (Spark 3.4 + 3.5) #843
• fix(build): restore delombok execution so release builds generate javadoc jars by @vinishjail97 in fix(build): restore delombok execution so release builds generate javadoc jars #853

New Contributors

• @sapienza88 made their first contribution in [553] Parquet Schema and Column Stats Converters #669
• @Monilnarang made their first contribution in Improved all tests in TestHudiInstantUtils by parameterising and adding more test cases #703
• @pratyakshsharma made their first contribution in Update community sync page #734
• @mikedias made their first contribution in Paimon Source Support #742
• @xr-chen made their first contribution in Fix Delta to Iceberg not working on column mapping enabled Delta source table #766
• @mao-liu made their first contribution in [755] Support column stats for paimon #767
• @kevinjqliu made their first contribution in Upgrade Iceberg to 1.9.2 and Avro to 1.12.0 #784
• @parisni made their first contribution in fix(delta): avoid NPE for binary in map/array schemas #795
• @alexr17 made their first contribution in Handle empty EarliestCommitToRetain #732
• @alealandreev made their first contribution in [MINOR] Preserve original cause in ReadException on close failure #827

Full Changelog: 0.3.0-incubating...0.4.0-incubating-rc1