From 5d56802cc0096c252e9fdaf6fe106ecc23412141 Mon Sep 17 00:00:00 2001 From: Vinish Reddy Date: Mon, 20 Jul 2026 15:25:49 -0700 Subject: [PATCH 1/4] [docs] Add xtable-spark-runtime module README and Spark quickstart docs Documents the xtable-spark-runtime bundle: a spark-submit quickstart with examples, the CLI options, a source/target format matrix, and Spark 3.4/3.5 compatibility (Delta auto-switches to the Spark-free Delta Kernel on 3.5+). Adds a "Run XTable on Apache Spark" docs page (linked from Installation and the sidebar) and a module README. Note: depends on the xtable-spark-runtime module (PR #843); intended to merge after the 0.4.0 release once the module lands on main. Co-Authored-By: Claude Opus 4.8 (1M context) --- website/docs/setup.md | 5 +- website/docs/spark-runtime.md | 113 +++++++++++++++++++++++++++++++++ website/sidebars.js | 3 +- xtable-spark-runtime/README.md | 94 +++++++++++++++++++++++++++ 4 files changed, 213 insertions(+), 2 deletions(-) create mode 100644 website/docs/spark-runtime.md create mode 100644 xtable-spark-runtime/README.md diff --git a/website/docs/setup.md b/website/docs/setup.md index d7b38da6c..d7b4484be 100644 --- a/website/docs/setup.md +++ b/website/docs/setup.md @@ -23,4 +23,7 @@ For more information on the steps, follow the project's GitHub [README.md](https ## Next Steps See the [Quickstart](/docs/how-to) guide to learn to use Apache XTable™ (Incubating) to add interoperability between -different table formats. \ No newline at end of file +different table formats. + +To run a sync on an existing Apache Spark cluster with `spark-submit`, see +[Run XTable on Apache Spark](/docs/spark-runtime). \ No newline at end of file diff --git a/website/docs/spark-runtime.md b/website/docs/spark-runtime.md new file mode 100644 index 000000000..8084afee9 --- /dev/null +++ b/website/docs/spark-runtime.md @@ -0,0 +1,113 @@ +--- +title: "Run XTable on Apache Spark" +--- + +# Running Apache XTable™ (Incubating) on Apache Spark + +The `xtable-spark-runtime` module publishes a self-contained (shaded) bundle jar that runs an +XTable metadata sync with `spark-submit` on an existing Apache Spark cluster. It is the +`spark-submit` equivalent of the `RunSync` utility: no data is rewritten, only the target table +format metadata is generated alongside the existing data files. + +Use this when you already run Spark (EMR, Dataproc, HDInsight, Databricks, or a local install) and +want to add interoperability without standing up a separate process. + +## Build the bundle + +From the project root: + +```shell md title="shell" +./mvnw clean package -pl xtable-spark-runtime -am -DskipTests +``` + +This produces the shaded bundle at +`xtable-spark-runtime/target/xtable-spark-runtime_2.12-.jar`. + +## Quick start + +Sync an existing Hudi table to Iceberg and Delta: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + --master 'local[*]' \ + xtable-spark-runtime_2.12-.jar \ + --basepath /path/to/hudi_table \ + --sourceformat HUDI \ + --targets ICEBERG,DELTA +``` + +Sync an Iceberg table to Hudi: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + xtable-spark-runtime_2.12-.jar \ + --basepath /path/to/iceberg_table \ + --sourceformat ICEBERG \ + --targets HUDI +``` + +Sync a partitioned Hudi table (pass the source partition spec): + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + xtable-spark-runtime_2.12-.jar \ + --basepath /path/to/hudi_table \ + --sourceformat HUDI \ + --targets DELTA \ + --partitionspec level:VALUE +``` + +The engine libraries (Hudi, Iceberg, Delta, Avro, Parquet) are `provided` — the bundle expects them +on the Spark runtime classpath, which is the case on a standard Spark install with the relevant +format support. + +## Command-line options + +| Option | Required | Description | +| --- | --- | --- | +| `--basepath` | yes | Base path of the source table. | +| `--sourceformat` | yes | Source table format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON`, or `PARQUET`. | +| `--targets` | yes | Comma-separated target formats, e.g. `ICEBERG,DELTA`. | +| `--datapath` | no | Path to the data files if different from the base path (e.g. Iceberg keeps data under `/data`). | +| `--tablename` | no | Table name; defaults to the last segment of the base path. | +| `--namespace` | no | Dot-separated table namespace. | +| `--partitionspec` | no | Hudi source partition field spec, e.g. `level:VALUE`. | +| `--usedeltakernel` | no | Force the Spark-free Delta Kernel for the Delta source/target. Auto-enabled on Spark 3.5+ (see below). | +| `--help` | no | Print usage. | + +## Supported formats + +Paimon and Parquet are read-only sources (there is no corresponding write target). + +| Source ↓ / Target → | Hudi | Iceberg | Delta | +| --- | :---: | :---: | :---: | +| **Hudi** | – | ✅ | ✅ | +| **Iceberg** | ✅ | – | ✅ | +| **Delta** | ✅ | ✅ | – | +| **Paimon** | ✅ | ✅ | ✅ | +| **Parquet** | ✅ | ✅ | ✅ | + +## Spark version compatibility + +Hudi and Iceberg conversion use only Spark-free core classes, so they run on any of the Spark lines +below. Delta is the only Spark-version-sensitive engine, and the bundle picks the right +implementation automatically: + +| Spark version | Hudi / Iceberg | Delta implementation | +| --- | :---: | --- | +| 3.4.x | ✅ | Delta Standalone (`delta-core`) | +| 3.5.x and newer | ✅ | Delta Kernel (Spark-free), selected automatically | + +On Spark 3.5+, the bundled `delta-core` does not run, so a Delta source or target is routed through +the Spark-free [Delta Kernel](https://docs.delta.io/latest/delta-kernel.html) implementation +automatically — no flag needed. To force Kernel on any Spark version (e.g. Spark 3.4), pass +`--usedeltakernel`. + +## Next steps + +- See [Installation](/docs/setup) for building the project. +- See the [Quickstart](/docs/how-to) for an end-to-end interoperability walkthrough. +- To query a synced table from Spark, see [Apache Spark](/docs/spark). diff --git a/website/sidebars.js b/website/sidebars.js index 9f0b55943..ce8c4b2cb 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -19,7 +19,8 @@ module.exports = { collapsed: false, items: [ 'how-to', - 'how-to-catalog-sync' + 'how-to-catalog-sync', + 'spark-runtime' ], }, { diff --git a/xtable-spark-runtime/README.md b/xtable-spark-runtime/README.md new file mode 100644 index 000000000..dd75e6a36 --- /dev/null +++ b/xtable-spark-runtime/README.md @@ -0,0 +1,94 @@ + + +# XTable Spark Runtime + +`xtable-spark-runtime` publishes a self-contained (shaded, relocated) bundle jar that runs an +Apache XTable™ metadata sync with `spark-submit` on an existing Apache Spark cluster. It is the +`spark-submit` equivalent of the `RunSync` utility — no data is rewritten, only the target table +format metadata is generated alongside the existing data files. + +The engine libraries (Hudi, Iceberg, Delta, Avro, Parquet) are `provided`: the user brings their +own engine versions from the Spark runtime, so the thin bundle stays compatible across versions. + +## Build + +From the project root: + +```shell +./mvnw clean package -pl xtable-spark-runtime -am -DskipTests +``` + +The shaded bundle is written to +`xtable-spark-runtime/target/xtable-spark-runtime_2.12-.jar`. + +## Usage + +```shell +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + --master 'local[*]' \ + xtable-spark-runtime_2.12-.jar \ + --basepath /path/to/hudi_table \ + --sourceformat HUDI \ + --targets ICEBERG,DELTA +``` + +| Option | Required | Description | +| --- | --- | --- | +| `--basepath` | yes | Base path of the source table. | +| `--sourceformat` | yes | `HUDI`, `ICEBERG`, `DELTA`, `PAIMON`, or `PARQUET`. | +| `--targets` | yes | Comma-separated target formats, e.g. `ICEBERG,DELTA`. | +| `--datapath` | no | Path to the data files if different from the base path. | +| `--tablename` | no | Table name; defaults to the last segment of the base path. | +| `--namespace` | no | Dot-separated table namespace. | +| `--partitionspec` | no | Hudi source partition field spec, e.g. `level:VALUE`. | +| `--usedeltakernel` | no | Force the Spark-free Delta Kernel for Delta; auto-enabled on Spark 3.5+. | +| `--help` | no | Print usage. | + +Paimon and Parquet are read-only sources; targets are Hudi, Iceberg, and Delta. + +## Spark version compatibility + +Hudi and Iceberg conversion use only Spark-free core classes and run on any Spark line below. Delta +is the only Spark-version-sensitive engine, and the bundle selects the implementation automatically: + +| Spark version | Hudi / Iceberg | Delta | +| --- | :---: | --- | +| 3.4.x | ✅ | Delta Standalone (`delta-core`) | +| 3.5.x and newer | ✅ | Delta Kernel (Spark-free), selected automatically | + +On Spark 3.5+ the bundled `delta-core` does not run, so a Delta source/target is routed through the +Spark-free Delta Kernel automatically. Pass `--usedeltakernel` to force it on any Spark version. + +## Bundle validation IT + +`ITXTableSparkRuntimeBundle` `spark-submit`s the shaded jar against a real Spark distribution, one +case per direction across Hudi, Iceberg, and Delta, and asserts the target is data-equivalent to +the source. It self-skips unless `SPARK_HOME` is set, so it does not run in the normal build: + +```shell +SPARK_LOCAL_IP=127.0.0.1 SPARK_HOME=/path/to/spark-3.5.x-bin-hadoop3 \ + ./mvnw verify -pl xtable-spark-runtime +``` + +CI runs this IT on both Spark 3.4 and 3.5 (see +`.github/workflows/spark-runtime-validation.yml`). + +For more, see the docs: [Run XTable on Apache Spark](https://xtable.apache.org/docs/spark-runtime). From aab4104f972923d423a851fe6de6e834c7698eb9 Mon Sep 17 00:00:00 2001 From: Vinish Reddy Date: Tue, 21 Jul 2026 11:11:42 -0700 Subject: [PATCH 2/4] [docs] xtable-spark-runtime: document existing-job use, multi-table --datasetconfig, TPC-DS example - Add "Adding to an existing Spark job" section: drop the thin bundle on an existing cluster's classpath (engines are provided) and run XTableSparkSync against a table the job already writes to add other formats; no data rewrite. - Document the --datasetconfig multi-table mode and add it to the options table (mutually exclusive with --basepath/--sourceformat/--targets). - Add examples/tpcds-dataset.yaml: a 24-table TPC-DS Hudi -> Iceberg,Delta config. Co-Authored-By: Claude Opus 4.8 (1M context) --- website/docs/spark-runtime.md | 63 ++++++++++++++++++- xtable-spark-runtime/README.md | 55 +++++++++++++++- .../examples/tpcds-dataset.yaml | 60 ++++++++++++++++++ 3 files changed, 172 insertions(+), 6 deletions(-) create mode 100644 xtable-spark-runtime/examples/tpcds-dataset.yaml diff --git a/website/docs/spark-runtime.md b/website/docs/spark-runtime.md index 8084afee9..759504a4c 100644 --- a/website/docs/spark-runtime.md +++ b/website/docs/spark-runtime.md @@ -64,13 +64,70 @@ The engine libraries (Hudi, Iceberg, Delta, Avro, Parquet) are `provided` — th on the Spark runtime classpath, which is the case on a standard Spark install with the relevant format support. +## Adding to an existing Spark job + +A common setup is a Spark job that already writes a table in one format — for example a job that +writes Hudi — where the same table should also be readable as Iceberg or Delta. Because the bundle's +engine libraries are `provided`, it reuses the Hudi, Iceberg, and Delta libraries already present on +the Spark runtime that job uses; the only additional artifact is this jar. Add it to that Spark +runtime's classpath and run the sync — no separate installation is required. + +Run `XTableSparkSync` on the same Spark cluster, pointed at the table the job writes, as a follow-on +step after the write completes. Set `--sourceformat` to the format the job writes and list the +formats to add in `--targets`: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + xtable-spark-runtime_2.12-.jar \ + --basepath s3://example-warehouse/db/orders \ + --sourceformat HUDI \ + --targets ICEBERG,DELTA +``` + +The sync reads the table's existing source-format metadata and writes the Iceberg and Delta metadata +alongside the data files the job already produced; no data is rewritten. + +## Sync multiple tables + +To sync more than one table in a single submit, pass `--datasetconfig` with a YAML file instead of +the per-table `--basepath`/`--sourceformat`/`--targets` flags (the two modes are mutually +exclusive). `sourceFormat` and `targetFormats` apply to every table; each entry under `datasets` +needs only a `tableBasePath`, with `tableName`, `tableDataPath`, `namespace`, and `partitionSpec` +optional. The config path may be local or on cloud storage, and each table is synced in turn. + +```yaml md title="dataset.yaml" +sourceFormat: HUDI +targetFormats: + - ICEBERG + - DELTA +datasets: + - tableBasePath: s3://tpcds-datasets/100GB/store_sales + - tableBasePath: s3://tpcds-datasets/100GB/store_returns + - tableBasePath: s3://tpcds-datasets/100GB/item +``` + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + xtable-spark-runtime_2.12-.jar \ + --datasetconfig dataset.yaml +``` + +A complete example covering the 24 tables of the TPC-DS schema is provided at +[`xtable-spark-runtime/examples/tpcds-dataset.yaml`](https://github.com/apache/incubator-xtable/blob/main/xtable-spark-runtime/examples/tpcds-dataset.yaml). + ## Command-line options +One of `--basepath` (single table) or `--datasetconfig` (multiple tables) is required; the two are +mutually exclusive. + | Option | Required | Description | | --- | --- | --- | -| `--basepath` | yes | Base path of the source table. | -| `--sourceformat` | yes | Source table format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON`, or `PARQUET`. | -| `--targets` | yes | Comma-separated target formats, e.g. `ICEBERG,DELTA`. | +| `--basepath` | yes\* | Base path of the source table. Required unless `--datasetconfig` is given. | +| `--sourceformat` | yes\* | Source table format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON`, or `PARQUET`. Required unless `--datasetconfig` is given. | +| `--targets` | yes\* | Comma-separated target formats, e.g. `ICEBERG,DELTA`. Required unless `--datasetconfig` is given. | +| `--datasetconfig` | yes\* | Path (local or cloud) to a YAML dataset config for syncing multiple tables. Mutually exclusive with `--basepath`/`--sourceformat`/`--targets`. | | `--datapath` | no | Path to the data files if different from the base path (e.g. Iceberg keeps data under `/data`). | | `--tablename` | no | Table name; defaults to the last segment of the base path. | | `--namespace` | no | Dot-separated table namespace. | diff --git a/xtable-spark-runtime/README.md b/xtable-spark-runtime/README.md index dd75e6a36..d8c9a21e5 100644 --- a/xtable-spark-runtime/README.md +++ b/xtable-spark-runtime/README.md @@ -52,9 +52,10 @@ $SPARK_HOME/bin/spark-submit \ | Option | Required | Description | | --- | --- | --- | -| `--basepath` | yes | Base path of the source table. | -| `--sourceformat` | yes | `HUDI`, `ICEBERG`, `DELTA`, `PAIMON`, or `PARQUET`. | -| `--targets` | yes | Comma-separated target formats, e.g. `ICEBERG,DELTA`. | +| `--basepath` | yes\* | Base path of the source table. Required unless `--datasetconfig` is given. | +| `--sourceformat` | yes\* | `HUDI`, `ICEBERG`, `DELTA`, `PAIMON`, or `PARQUET`. Required unless `--datasetconfig` is given. | +| `--targets` | yes\* | Comma-separated target formats, e.g. `ICEBERG,DELTA`. Required unless `--datasetconfig` is given. | +| `--datasetconfig` | yes\* | Path (local or cloud) to a YAML dataset config for syncing multiple tables. Mutually exclusive with `--basepath`/`--sourceformat`/`--targets`. | | `--datapath` | no | Path to the data files if different from the base path. | | `--tablename` | no | Table name; defaults to the last segment of the base path. | | `--namespace` | no | Dot-separated table namespace. | @@ -64,6 +65,54 @@ $SPARK_HOME/bin/spark-submit \ Paimon and Parquet are read-only sources; targets are Hudi, Iceberg, and Delta. +## Adding to an existing Spark job + +If a Spark job already writes a table in one format (for example Hudi) and the same table should +also be readable as Iceberg or Delta, the bundle's `provided` engine libraries let it reuse the +Hudi, Iceberg, and Delta libraries already on that job's Spark runtime — the only additional +artifact is this jar. Add it to that Spark runtime's classpath and run `XTableSparkSync` against the +table as a follow-on step after the write completes, with `--sourceformat` set to the format the job +writes and the formats to add listed in `--targets`: + +```shell +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + xtable-spark-runtime_2.12-.jar \ + --basepath s3://example-warehouse/db/orders \ + --sourceformat HUDI \ + --targets ICEBERG,DELTA +``` + +The sync reads the existing source-format metadata and writes the target metadata alongside the data +files the job already produced; no data is rewritten. + +## Sync multiple tables + +Pass `--datasetconfig` with a YAML file instead of the per-table flags to sync several tables in one +submit. `sourceFormat` and `targetFormats` apply to every table; each `datasets` entry needs only a +`tableBasePath` (`tableName`, `tableDataPath`, `namespace`, and `partitionSpec` are optional): + +```yaml +sourceFormat: HUDI +targetFormats: + - ICEBERG + - DELTA +datasets: + - tableBasePath: s3://tpcds-datasets/100GB/store_sales + - tableBasePath: s3://tpcds-datasets/100GB/store_returns + - tableBasePath: s3://tpcds-datasets/100GB/item +``` + +```shell +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + xtable-spark-runtime_2.12-.jar \ + --datasetconfig dataset.yaml +``` + +A complete example covering the 24 tables of the TPC-DS schema is at +[`examples/tpcds-dataset.yaml`](examples/tpcds-dataset.yaml). + ## Spark version compatibility Hudi and Iceberg conversion use only Spark-free core classes and run on any Spark line below. Delta diff --git a/xtable-spark-runtime/examples/tpcds-dataset.yaml b/xtable-spark-runtime/examples/tpcds-dataset.yaml new file mode 100644 index 000000000..2b0b4d326 --- /dev/null +++ b/xtable-spark-runtime/examples/tpcds-dataset.yaml @@ -0,0 +1,60 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +# Example --datasetconfig for a multi-table sync. This syncs the 24 tables of the +# TPC-DS schema from Hudi to Iceberg and Delta in a single spark-submit: +# +# $SPARK_HOME/bin/spark-submit \ +# --class org.apache.xtable.spark.XTableSparkSync \ +# xtable-spark-runtime_2.12-.jar \ +# --datasetconfig examples/tpcds-dataset.yaml +# +# sourceFormat and targetFormats apply to every table. Each entry needs only a +# tableBasePath; tableName, tableDataPath, namespace, and partitionSpec are optional +# (a Hudi source reads its partitioning from the table itself). Replace the bucket and +# prefix with the location of your tables. +sourceFormat: HUDI +targetFormats: + - ICEBERG + - DELTA +datasets: + - tableBasePath: s3://tpcds-datasets/100GB/call_center + - tableBasePath: s3://tpcds-datasets/100GB/catalog_page + - tableBasePath: s3://tpcds-datasets/100GB/catalog_returns + - tableBasePath: s3://tpcds-datasets/100GB/catalog_sales + - tableBasePath: s3://tpcds-datasets/100GB/customer + - tableBasePath: s3://tpcds-datasets/100GB/customer_address + - tableBasePath: s3://tpcds-datasets/100GB/customer_demographics + - tableBasePath: s3://tpcds-datasets/100GB/date_dim + - tableBasePath: s3://tpcds-datasets/100GB/household_demographics + - tableBasePath: s3://tpcds-datasets/100GB/income_band + - tableBasePath: s3://tpcds-datasets/100GB/inventory + - tableBasePath: s3://tpcds-datasets/100GB/item + - tableBasePath: s3://tpcds-datasets/100GB/promotion + - tableBasePath: s3://tpcds-datasets/100GB/reason + - tableBasePath: s3://tpcds-datasets/100GB/ship_mode + - tableBasePath: s3://tpcds-datasets/100GB/store + - tableBasePath: s3://tpcds-datasets/100GB/store_returns + - tableBasePath: s3://tpcds-datasets/100GB/store_sales + - tableBasePath: s3://tpcds-datasets/100GB/time_dim + - tableBasePath: s3://tpcds-datasets/100GB/warehouse + - tableBasePath: s3://tpcds-datasets/100GB/web_page + - tableBasePath: s3://tpcds-datasets/100GB/web_returns + - tableBasePath: s3://tpcds-datasets/100GB/web_sales + - tableBasePath: s3://tpcds-datasets/100GB/web_site From 06e1175c318eb30885fcd949c79d5c668beb2329 Mon Sep 17 00:00:00 2001 From: Vinish Reddy Date: Tue, 21 Jul 2026 12:08:29 -0700 Subject: [PATCH 3/4] [docs] xtable-spark-runtime: call it a runtime jar, tighten prose Rename "bundle jar" to "runtime jar" throughout to match the module name and Iceberg's convention; keep "(shaded, relocated)" once in the developer README. Drop em-dashes from prose. Co-Authored-By: Claude Opus 4.8 (1M context) --- website/docs/spark-runtime.md | 30 +++++++++++++++--------------- xtable-spark-runtime/README.md | 20 ++++++++++---------- 2 files changed, 25 insertions(+), 25 deletions(-) diff --git a/website/docs/spark-runtime.md b/website/docs/spark-runtime.md index 759504a4c..a6cbff5df 100644 --- a/website/docs/spark-runtime.md +++ b/website/docs/spark-runtime.md @@ -4,7 +4,7 @@ title: "Run XTable on Apache Spark" # Running Apache XTable™ (Incubating) on Apache Spark -The `xtable-spark-runtime` module publishes a self-contained (shaded) bundle jar that runs an +The `xtable-spark-runtime` module publishes a self-contained runtime jar that runs an XTable metadata sync with `spark-submit` on an existing Apache Spark cluster. It is the `spark-submit` equivalent of the `RunSync` utility: no data is rewritten, only the target table format metadata is generated alongside the existing data files. @@ -12,7 +12,7 @@ format metadata is generated alongside the existing data files. Use this when you already run Spark (EMR, Dataproc, HDInsight, Databricks, or a local install) and want to add interoperability without standing up a separate process. -## Build the bundle +## Build the runtime jar From the project root: @@ -20,7 +20,7 @@ From the project root: ./mvnw clean package -pl xtable-spark-runtime -am -DskipTests ``` -This produces the shaded bundle at +This produces the runtime jar at `xtable-spark-runtime/target/xtable-spark-runtime_2.12-.jar`. ## Quick start @@ -60,17 +60,17 @@ $SPARK_HOME/bin/spark-submit \ --partitionspec level:VALUE ``` -The engine libraries (Hudi, Iceberg, Delta, Avro, Parquet) are `provided` — the bundle expects them -on the Spark runtime classpath, which is the case on a standard Spark install with the relevant +The engine libraries (Hudi, Iceberg, Delta, Avro, Parquet) are `provided`. The runtime jar expects +them on the Spark runtime classpath, which is the case on a standard Spark install with the relevant format support. ## Adding to an existing Spark job -A common setup is a Spark job that already writes a table in one format — for example a job that -writes Hudi — where the same table should also be readable as Iceberg or Delta. Because the bundle's -engine libraries are `provided`, it reuses the Hudi, Iceberg, and Delta libraries already present on -the Spark runtime that job uses; the only additional artifact is this jar. Add it to that Spark -runtime's classpath and run the sync — no separate installation is required. +A common setup is a Spark job that already writes a table in one format (for example a job that +writes Hudi) where the same table should also be readable as Iceberg or Delta. Because the runtime +jar's engine libraries are `provided`, it reuses the Hudi, Iceberg, and Delta libraries already +present on the Spark runtime that job uses; the only additional artifact is this jar. Add it to that +Spark runtime's classpath and run the sync, with no separate installation required. Run `XTableSparkSync` on the same Spark cluster, pointed at the table the job writes, as a follow-on step after the write completes. Set `--sourceformat` to the format the job writes and list the @@ -150,7 +150,7 @@ Paimon and Parquet are read-only sources (there is no corresponding write target ## Spark version compatibility Hudi and Iceberg conversion use only Spark-free core classes, so they run on any of the Spark lines -below. Delta is the only Spark-version-sensitive engine, and the bundle picks the right +below. Delta is the only Spark-version-sensitive engine, and the runtime jar picks the right implementation automatically: | Spark version | Hudi / Iceberg | Delta implementation | @@ -158,10 +158,10 @@ implementation automatically: | 3.4.x | ✅ | Delta Standalone (`delta-core`) | | 3.5.x and newer | ✅ | Delta Kernel (Spark-free), selected automatically | -On Spark 3.5+, the bundled `delta-core` does not run, so a Delta source or target is routed through -the Spark-free [Delta Kernel](https://docs.delta.io/latest/delta-kernel.html) implementation -automatically — no flag needed. To force Kernel on any Spark version (e.g. Spark 3.4), pass -`--usedeltakernel`. +On Spark 3.5+, the `delta-core` in the runtime jar does not run, so a Delta source or target is +routed through the Spark-free [Delta Kernel](https://docs.delta.io/latest/delta-kernel.html) +implementation automatically, with no flag needed. To force Kernel on any Spark version (e.g. Spark +3.4), pass `--usedeltakernel`. ## Next steps diff --git a/xtable-spark-runtime/README.md b/xtable-spark-runtime/README.md index d8c9a21e5..267d5bda6 100644 --- a/xtable-spark-runtime/README.md +++ b/xtable-spark-runtime/README.md @@ -19,13 +19,13 @@ # XTable Spark Runtime -`xtable-spark-runtime` publishes a self-contained (shaded, relocated) bundle jar that runs an +`xtable-spark-runtime` publishes a self-contained (shaded, relocated) runtime jar that runs an Apache XTable™ metadata sync with `spark-submit` on an existing Apache Spark cluster. It is the -`spark-submit` equivalent of the `RunSync` utility — no data is rewritten, only the target table +`spark-submit` equivalent of the `RunSync` utility: no data is rewritten, only the target table format metadata is generated alongside the existing data files. The engine libraries (Hudi, Iceberg, Delta, Avro, Parquet) are `provided`: the user brings their -own engine versions from the Spark runtime, so the thin bundle stays compatible across versions. +own engine versions from the Spark runtime, so the thin runtime jar stays compatible across versions. ## Build @@ -35,7 +35,7 @@ From the project root: ./mvnw clean package -pl xtable-spark-runtime -am -DskipTests ``` -The shaded bundle is written to +The runtime jar is written to `xtable-spark-runtime/target/xtable-spark-runtime_2.12-.jar`. ## Usage @@ -68,8 +68,8 @@ Paimon and Parquet are read-only sources; targets are Hudi, Iceberg, and Delta. ## Adding to an existing Spark job If a Spark job already writes a table in one format (for example Hudi) and the same table should -also be readable as Iceberg or Delta, the bundle's `provided` engine libraries let it reuse the -Hudi, Iceberg, and Delta libraries already on that job's Spark runtime — the only additional +also be readable as Iceberg or Delta, the runtime jar's `provided` engine libraries let it reuse the +Hudi, Iceberg, and Delta libraries already on that job's Spark runtime; the only additional artifact is this jar. Add it to that Spark runtime's classpath and run `XTableSparkSync` against the table as a follow-on step after the write completes, with `--sourceformat` set to the format the job writes and the formats to add listed in `--targets`: @@ -116,19 +116,19 @@ A complete example covering the 24 tables of the TPC-DS schema is at ## Spark version compatibility Hudi and Iceberg conversion use only Spark-free core classes and run on any Spark line below. Delta -is the only Spark-version-sensitive engine, and the bundle selects the implementation automatically: +is the only Spark-version-sensitive engine, and the runtime jar selects the implementation automatically: | Spark version | Hudi / Iceberg | Delta | | --- | :---: | --- | | 3.4.x | ✅ | Delta Standalone (`delta-core`) | | 3.5.x and newer | ✅ | Delta Kernel (Spark-free), selected automatically | -On Spark 3.5+ the bundled `delta-core` does not run, so a Delta source/target is routed through the +On Spark 3.5+ the `delta-core` in the runtime jar does not run, so a Delta source/target is routed through the Spark-free Delta Kernel automatically. Pass `--usedeltakernel` to force it on any Spark version. -## Bundle validation IT +## Runtime jar validation IT -`ITXTableSparkRuntimeBundle` `spark-submit`s the shaded jar against a real Spark distribution, one +`ITXTableSparkRuntimeBundle` `spark-submit`s the runtime jar against a real Spark distribution, one case per direction across Hudi, Iceberg, and Delta, and asserts the target is data-equivalent to the source. It self-skips unless `SPARK_HOME` is set, so it does not run in the normal build: From d391270a1152b84bf777ec36e1243ebd1ce367bc Mon Sep 17 00:00:00 2001 From: Vinish Reddy Date: Tue, 21 Jul 2026 12:29:54 -0700 Subject: [PATCH 4/4] [docs] Fix Spotless YAML license header on tpcds-dataset.yaml Normalize the ASF header to the canonical wrapping in style/text-license-header and prefix the example comment with ## so Spotless treats it as content instead of folding it into the license-header region. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../examples/tpcds-dataset.yaml | 29 +++++++++---------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/xtable-spark-runtime/examples/tpcds-dataset.yaml b/xtable-spark-runtime/examples/tpcds-dataset.yaml index 2b0b4d326..0ee8c4373 100644 --- a/xtable-spark-runtime/examples/tpcds-dataset.yaml +++ b/xtable-spark-runtime/examples/tpcds-dataset.yaml @@ -1,23 +1,20 @@ # -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at # -# http://www.apache.org/licenses/LICENSE-2.0 +# http://www.apache.org/licenses/LICENSE-2.0 # -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. # - -# Example --datasetconfig for a multi-table sync. This syncs the 24 tables of the +## Example --datasetconfig for a multi-table sync. This syncs the 24 tables of the # TPC-DS schema from Hudi to Iceberg and Delta in a single spark-submit: # # $SPARK_HOME/bin/spark-submit \