[docs] xtable-spark-runtime README + Spark quickstart docs

website/docs/setup.md

@@ -23,4 +23,7 @@ For more information on the steps, follow the project's GitHub [README.md](https
 
 ## Next Steps
 See the [Quickstart](/docs/how-to) guide to learn to use Apache XTable™ (Incubating) to add interoperability between
-different table formats.
+different table formats.
+
+To run a sync on an existing Apache Spark cluster with `spark-submit`, see
+[Run XTable on Apache Spark](/docs/spark-runtime).

website/docs/spark-runtime.md

@@ -0,0 +1,170 @@
+---
+title: "Run XTable on Apache Spark"
+---
+
+# Running Apache XTable™ (Incubating) on Apache Spark
+
+The `xtable-spark-runtime` module publishes a self-contained runtime jar that runs an
+XTable metadata sync with `spark-submit` on an existing Apache Spark cluster. It is the
+`spark-submit` equivalent of the `RunSync` utility: no data is rewritten, only the target table
+format metadata is generated alongside the existing data files.
+
+Use this when you already run Spark (EMR, Dataproc, HDInsight, Databricks, or a local install) and
+want to add interoperability without standing up a separate process.
+
+## Build the runtime jar
+
+From the project root:
+
+```shell md title="shell"
+./mvnw clean package -pl xtable-spark-runtime -am -DskipTests
+```
+
+This produces the runtime jar at
+`xtable-spark-runtime/target/xtable-spark-runtime_2.12-<version>.jar`.
+
+## Quick start
+
+Sync an existing Hudi table to Iceberg and Delta:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  --master 'local[*]' \
+  xtable-spark-runtime_2.12-<version>.jar \
+  --basepath /path/to/hudi_table \
+  --sourceformat HUDI \
+  --targets ICEBERG,DELTA
+```
+
+Sync an Iceberg table to Hudi:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  xtable-spark-runtime_2.12-<version>.jar \
+  --basepath /path/to/iceberg_table \
+  --sourceformat ICEBERG \
+  --targets HUDI
+```
+
+Sync a partitioned Hudi table (pass the source partition spec):
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  xtable-spark-runtime_2.12-<version>.jar \
+  --basepath /path/to/hudi_table \
+  --sourceformat HUDI \
+  --targets DELTA \
+  --partitionspec level:VALUE
+```
+
+The engine libraries (Hudi, Iceberg, Delta, Avro, Parquet) are `provided`. The runtime jar expects
+them on the Spark runtime classpath, which is the case on a standard Spark install with the relevant
+format support.
+
+## Adding to an existing Spark job
+
+A common setup is a Spark job that already writes a table in one format (for example a job that
+writes Hudi) where the same table should also be readable as Iceberg or Delta. Because the runtime
+jar's engine libraries are `provided`, it reuses the Hudi, Iceberg, and Delta libraries already
+present on the Spark runtime that job uses; the only additional artifact is this jar. Add it to that
+Spark runtime's classpath and run the sync, with no separate installation required.
+
+Run `XTableSparkSync` on the same Spark cluster, pointed at the table the job writes, as a follow-on
+step after the write completes. Set `--sourceformat` to the format the job writes and list the
+formats to add in `--targets`:
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  xtable-spark-runtime_2.12-<version>.jar \
+  --basepath s3://example-warehouse/db/orders \
+  --sourceformat HUDI \
+  --targets ICEBERG,DELTA
+```
+
+The sync reads the table's existing source-format metadata and writes the Iceberg and Delta metadata
+alongside the data files the job already produced; no data is rewritten.
+
+## Sync multiple tables
+
+To sync more than one table in a single submit, pass `--datasetconfig` with a YAML file instead of
+the per-table `--basepath`/`--sourceformat`/`--targets` flags (the two modes are mutually
+exclusive). `sourceFormat` and `targetFormats` apply to every table; each entry under `datasets`
+needs only a `tableBasePath`, with `tableName`, `tableDataPath`, `namespace`, and `partitionSpec`
+optional. The config path may be local or on cloud storage, and each table is synced in turn.
+
+```yaml md title="dataset.yaml"
+sourceFormat: HUDI
+targetFormats:
+  - ICEBERG
+  - DELTA
+datasets:
+  - tableBasePath: s3://tpcds-datasets/100GB/store_sales
+  - tableBasePath: s3://tpcds-datasets/100GB/store_returns
+  - tableBasePath: s3://tpcds-datasets/100GB/item
+```
+
+```shell md title="shell"
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  xtable-spark-runtime_2.12-<version>.jar \
+  --datasetconfig dataset.yaml
+```
+
+A complete example covering the 24 tables of the TPC-DS schema is provided at
+[`xtable-spark-runtime/examples/tpcds-dataset.yaml`](https://github.com/apache/incubator-xtable/blob/main/xtable-spark-runtime/examples/tpcds-dataset.yaml).
+
+## Command-line options
+
+One of `--basepath` (single table) or `--datasetconfig` (multiple tables) is required; the two are
+mutually exclusive.
+
+| Option | Required | Description |
+| --- | --- | --- |
+| `--basepath` | yes\* | Base path of the source table. Required unless `--datasetconfig` is given. |
+| `--sourceformat` | yes\* | Source table format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON`, or `PARQUET`. Required unless `--datasetconfig` is given. |
+| `--targets` | yes\* | Comma-separated target formats, e.g. `ICEBERG,DELTA`. Required unless `--datasetconfig` is given. |
+| `--datasetconfig` | yes\* | Path (local or cloud) to a YAML dataset config for syncing multiple tables. Mutually exclusive with `--basepath`/`--sourceformat`/`--targets`. |
+| `--datapath` | no | Path to the data files if different from the base path (e.g. Iceberg keeps data under `<basePath>/data`). |
+| `--tablename` | no | Table name; defaults to the last segment of the base path. |
+| `--namespace` | no | Dot-separated table namespace. |
+| `--partitionspec` | no | Hudi source partition field spec, e.g. `level:VALUE`. |
+| `--usedeltakernel` | no | Force the Spark-free Delta Kernel for the Delta source/target. Auto-enabled on Spark 3.5+ (see below). |
+| `--help` | no | Print usage. |
+
+## Supported formats
+
+Paimon and Parquet are read-only sources (there is no corresponding write target).
+
+| Source ↓ / Target → | Hudi | Iceberg | Delta |
+| --- | :---: | :---: | :---: |
+| **Hudi** | – | ✅ | ✅ |
+| **Iceberg** | ✅ | – | ✅ |
+| **Delta** | ✅ | ✅ | – |
+| **Paimon** | ✅ | ✅ | ✅ |
+| **Parquet** | ✅ | ✅ | ✅ |
+
+## Spark version compatibility
+
+Hudi and Iceberg conversion use only Spark-free core classes, so they run on any of the Spark lines
+below. Delta is the only Spark-version-sensitive engine, and the runtime jar picks the right
+implementation automatically:
+
+| Spark version | Hudi / Iceberg | Delta implementation |
+| --- | :---: | --- |
+| 3.4.x | ✅ | Delta Standalone (`delta-core`) |
+| 3.5.x and newer | ✅ | Delta Kernel (Spark-free), selected automatically |
+
+On Spark 3.5+, the `delta-core` in the runtime jar does not run, so a Delta source or target is
+routed through the Spark-free [Delta Kernel](https://docs.delta.io/latest/delta-kernel.html)
+implementation automatically, with no flag needed. To force Kernel on any Spark version (e.g. Spark
+3.4), pass `--usedeltakernel`.
+
+## Next steps
+
+- See [Installation](/docs/setup) for building the project.
+- See the [Quickstart](/docs/how-to) for an end-to-end interoperability walkthrough.
+- To query a synced table from Spark, see [Apache Spark](/docs/spark).

website/sidebars.js

@@ -19,7 +19,8 @@ module.exports = {
             collapsed: false,
             items: [
                 'how-to',
-                'how-to-catalog-sync'
+                'how-to-catalog-sync',
+                'spark-runtime'
             ],
         },
         {

xtable-spark-runtime/README.md

@@ -0,0 +1,143 @@
+<!--
+  - Licensed to the Apache Software Foundation (ASF) under one
+  - or more contributor license agreements.  See the NOTICE file
+  - distributed with this work for additional information
+  - regarding copyright ownership.  The ASF licenses this file
+  - to you under the Apache License, Version 2.0 (the
+  - "License"); you may not use this file except in compliance
+  - with the License.  You may obtain a copy of the License at
+  -
+  -   http://www.apache.org/licenses/LICENSE-2.0
+  -
+  - Unless required by applicable law or agreed to in writing,
+  - software distributed under the License is distributed on an
+  - "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  - KIND, either express or implied.  See the License for the
+  - specific language governing permissions and limitations
+  - under the License.
+  -->
+
+# XTable Spark Runtime
+
+`xtable-spark-runtime` publishes a self-contained (shaded, relocated) runtime jar that runs an
+Apache XTable™ metadata sync with `spark-submit` on an existing Apache Spark cluster. It is the
+`spark-submit` equivalent of the `RunSync` utility: no data is rewritten, only the target table
+format metadata is generated alongside the existing data files.
+
+The engine libraries (Hudi, Iceberg, Delta, Avro, Parquet) are `provided`: the user brings their
+own engine versions from the Spark runtime, so the thin runtime jar stays compatible across versions.
+
+## Build
+
+From the project root:
+
+```shell
+./mvnw clean package -pl xtable-spark-runtime -am -DskipTests
+```
+
+The runtime jar is written to
+`xtable-spark-runtime/target/xtable-spark-runtime_2.12-<version>.jar`.
+
+## Usage
+
+```shell
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  --master 'local[*]' \
+  xtable-spark-runtime_2.12-<version>.jar \
+  --basepath /path/to/hudi_table \
+  --sourceformat HUDI \
+  --targets ICEBERG,DELTA
+```
+
+| Option | Required | Description |
+| --- | --- | --- |
+| `--basepath` | yes\* | Base path of the source table. Required unless `--datasetconfig` is given. |
+| `--sourceformat` | yes\* | `HUDI`, `ICEBERG`, `DELTA`, `PAIMON`, or `PARQUET`. Required unless `--datasetconfig` is given. |
+| `--targets` | yes\* | Comma-separated target formats, e.g. `ICEBERG,DELTA`. Required unless `--datasetconfig` is given. |
+| `--datasetconfig` | yes\* | Path (local or cloud) to a YAML dataset config for syncing multiple tables. Mutually exclusive with `--basepath`/`--sourceformat`/`--targets`. |
+| `--datapath` | no | Path to the data files if different from the base path. |
+| `--tablename` | no | Table name; defaults to the last segment of the base path. |
+| `--namespace` | no | Dot-separated table namespace. |
+| `--partitionspec` | no | Hudi source partition field spec, e.g. `level:VALUE`. |
+| `--usedeltakernel` | no | Force the Spark-free Delta Kernel for Delta; auto-enabled on Spark 3.5+. |
+| `--help` | no | Print usage. |
+
+Paimon and Parquet are read-only sources; targets are Hudi, Iceberg, and Delta.
+
+## Adding to an existing Spark job
+
+If a Spark job already writes a table in one format (for example Hudi) and the same table should
+also be readable as Iceberg or Delta, the runtime jar's `provided` engine libraries let it reuse the
+Hudi, Iceberg, and Delta libraries already on that job's Spark runtime; the only additional
+artifact is this jar. Add it to that Spark runtime's classpath and run `XTableSparkSync` against the
+table as a follow-on step after the write completes, with `--sourceformat` set to the format the job
+writes and the formats to add listed in `--targets`:
+
+```shell
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  xtable-spark-runtime_2.12-<version>.jar \
+  --basepath s3://example-warehouse/db/orders \
+  --sourceformat HUDI \
+  --targets ICEBERG,DELTA
+```
+
+The sync reads the existing source-format metadata and writes the target metadata alongside the data
+files the job already produced; no data is rewritten.
+
+## Sync multiple tables
+
+Pass `--datasetconfig` with a YAML file instead of the per-table flags to sync several tables in one
+submit. `sourceFormat` and `targetFormats` apply to every table; each `datasets` entry needs only a
+`tableBasePath` (`tableName`, `tableDataPath`, `namespace`, and `partitionSpec` are optional):
+
+```yaml
+sourceFormat: HUDI
+targetFormats:
+  - ICEBERG
+  - DELTA
+datasets:
+  - tableBasePath: s3://tpcds-datasets/100GB/store_sales
+  - tableBasePath: s3://tpcds-datasets/100GB/store_returns
+  - tableBasePath: s3://tpcds-datasets/100GB/item
+```
+
+```shell
+$SPARK_HOME/bin/spark-submit \
+  --class org.apache.xtable.spark.XTableSparkSync \
+  xtable-spark-runtime_2.12-<version>.jar \
+  --datasetconfig dataset.yaml
+```
+
+A complete example covering the 24 tables of the TPC-DS schema is at
+[`examples/tpcds-dataset.yaml`](examples/tpcds-dataset.yaml).
+
+## Spark version compatibility
+
+Hudi and Iceberg conversion use only Spark-free core classes and run on any Spark line below. Delta
+is the only Spark-version-sensitive engine, and the runtime jar selects the implementation automatically:
+
+| Spark version | Hudi / Iceberg | Delta |
+| --- | :---: | --- |
+| 3.4.x | ✅ | Delta Standalone (`delta-core`) |
+| 3.5.x and newer | ✅ | Delta Kernel (Spark-free), selected automatically |
+
+On Spark 3.5+ the `delta-core` in the runtime jar does not run, so a Delta source/target is routed through the
+Spark-free Delta Kernel automatically. Pass `--usedeltakernel` to force it on any Spark version.
+
+## Runtime jar validation IT
+
+`ITXTableSparkRuntimeBundle` `spark-submit`s the runtime jar against a real Spark distribution, one
+case per direction across Hudi, Iceberg, and Delta, and asserts the target is data-equivalent to
+the source. It self-skips unless `SPARK_HOME` is set, so it does not run in the normal build:
+
+```shell
+SPARK_LOCAL_IP=127.0.0.1 SPARK_HOME=/path/to/spark-3.5.x-bin-hadoop3 \
+  ./mvnw verify -pl xtable-spark-runtime
+```
+
+CI runs this IT on both Spark 3.4 and 3.5 (see
+`.github/workflows/spark-runtime-validation.yml`).
+
+For more, see the docs: [Run XTable on Apache Spark](https://xtable.apache.org/docs/spark-runtime).

xtable-spark-runtime/examples/tpcds-dataset.yaml

@@ -0,0 +1,57 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+## Example --datasetconfig for a multi-table sync. This syncs the 24 tables of the
+# TPC-DS schema from Hudi to Iceberg and Delta in a single spark-submit:
+#
+#   $SPARK_HOME/bin/spark-submit \
+#     --class org.apache.xtable.spark.XTableSparkSync \
+#     xtable-spark-runtime_2.12-<version>.jar \
+#     --datasetconfig examples/tpcds-dataset.yaml
+#
+# sourceFormat and targetFormats apply to every table. Each entry needs only a
+# tableBasePath; tableName, tableDataPath, namespace, and partitionSpec are optional
+# (a Hudi source reads its partitioning from the table itself). Replace the bucket and
+# prefix with the location of your tables.
+sourceFormat: HUDI
+targetFormats:
+  - ICEBERG
+  - DELTA
+datasets:
+  - tableBasePath: s3://tpcds-datasets/100GB/call_center
+  - tableBasePath: s3://tpcds-datasets/100GB/catalog_page
+  - tableBasePath: s3://tpcds-datasets/100GB/catalog_returns
+  - tableBasePath: s3://tpcds-datasets/100GB/catalog_sales
+  - tableBasePath: s3://tpcds-datasets/100GB/customer
+  - tableBasePath: s3://tpcds-datasets/100GB/customer_address
+  - tableBasePath: s3://tpcds-datasets/100GB/customer_demographics
+  - tableBasePath: s3://tpcds-datasets/100GB/date_dim
+  - tableBasePath: s3://tpcds-datasets/100GB/household_demographics
+  - tableBasePath: s3://tpcds-datasets/100GB/income_band
+  - tableBasePath: s3://tpcds-datasets/100GB/inventory
+  - tableBasePath: s3://tpcds-datasets/100GB/item
+  - tableBasePath: s3://tpcds-datasets/100GB/promotion
+  - tableBasePath: s3://tpcds-datasets/100GB/reason
+  - tableBasePath: s3://tpcds-datasets/100GB/ship_mode
+  - tableBasePath: s3://tpcds-datasets/100GB/store
+  - tableBasePath: s3://tpcds-datasets/100GB/store_returns
+  - tableBasePath: s3://tpcds-datasets/100GB/store_sales
+  - tableBasePath: s3://tpcds-datasets/100GB/time_dim
+  - tableBasePath: s3://tpcds-datasets/100GB/warehouse
+  - tableBasePath: s3://tpcds-datasets/100GB/web_page
+  - tableBasePath: s3://tpcds-datasets/100GB/web_returns
+  - tableBasePath: s3://tpcds-datasets/100GB/web_sales
+  - tableBasePath: s3://tpcds-datasets/100GB/web_site