diff --git a/website/docs/setup.md b/website/docs/setup.md index d7b38da6c..d7b4484be 100644 --- a/website/docs/setup.md +++ b/website/docs/setup.md @@ -23,4 +23,7 @@ For more information on the steps, follow the project's GitHub [README.md](https ## Next Steps See the [Quickstart](/docs/how-to) guide to learn to use Apache XTable™ (Incubating) to add interoperability between -different table formats. \ No newline at end of file +different table formats. + +To run a sync on an existing Apache Spark cluster with `spark-submit`, see +[Run XTable on Apache Spark](/docs/spark-runtime). \ No newline at end of file diff --git a/website/docs/spark-runtime.md b/website/docs/spark-runtime.md new file mode 100644 index 000000000..a6cbff5df --- /dev/null +++ b/website/docs/spark-runtime.md @@ -0,0 +1,170 @@ +--- +title: "Run XTable on Apache Spark" +--- + +# Running Apache XTable™ (Incubating) on Apache Spark + +The `xtable-spark-runtime` module publishes a self-contained runtime jar that runs an +XTable metadata sync with `spark-submit` on an existing Apache Spark cluster. It is the +`spark-submit` equivalent of the `RunSync` utility: no data is rewritten, only the target table +format metadata is generated alongside the existing data files. + +Use this when you already run Spark (EMR, Dataproc, HDInsight, Databricks, or a local install) and +want to add interoperability without standing up a separate process. + +## Build the runtime jar + +From the project root: + +```shell md title="shell" +./mvnw clean package -pl xtable-spark-runtime -am -DskipTests +``` + +This produces the runtime jar at +`xtable-spark-runtime/target/xtable-spark-runtime_2.12-.jar`. + +## Quick start + +Sync an existing Hudi table to Iceberg and Delta: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + --master 'local[*]' \ + xtable-spark-runtime_2.12-.jar \ + --basepath /path/to/hudi_table \ + --sourceformat HUDI \ + --targets ICEBERG,DELTA +``` + +Sync an Iceberg table to Hudi: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + xtable-spark-runtime_2.12-.jar \ + --basepath /path/to/iceberg_table \ + --sourceformat ICEBERG \ + --targets HUDI +``` + +Sync a partitioned Hudi table (pass the source partition spec): + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + xtable-spark-runtime_2.12-.jar \ + --basepath /path/to/hudi_table \ + --sourceformat HUDI \ + --targets DELTA \ + --partitionspec level:VALUE +``` + +The engine libraries (Hudi, Iceberg, Delta, Avro, Parquet) are `provided`. The runtime jar expects +them on the Spark runtime classpath, which is the case on a standard Spark install with the relevant +format support. + +## Adding to an existing Spark job + +A common setup is a Spark job that already writes a table in one format (for example a job that +writes Hudi) where the same table should also be readable as Iceberg or Delta. Because the runtime +jar's engine libraries are `provided`, it reuses the Hudi, Iceberg, and Delta libraries already +present on the Spark runtime that job uses; the only additional artifact is this jar. Add it to that +Spark runtime's classpath and run the sync, with no separate installation required. + +Run `XTableSparkSync` on the same Spark cluster, pointed at the table the job writes, as a follow-on +step after the write completes. Set `--sourceformat` to the format the job writes and list the +formats to add in `--targets`: + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + xtable-spark-runtime_2.12-.jar \ + --basepath s3://example-warehouse/db/orders \ + --sourceformat HUDI \ + --targets ICEBERG,DELTA +``` + +The sync reads the table's existing source-format metadata and writes the Iceberg and Delta metadata +alongside the data files the job already produced; no data is rewritten. + +## Sync multiple tables + +To sync more than one table in a single submit, pass `--datasetconfig` with a YAML file instead of +the per-table `--basepath`/`--sourceformat`/`--targets` flags (the two modes are mutually +exclusive). `sourceFormat` and `targetFormats` apply to every table; each entry under `datasets` +needs only a `tableBasePath`, with `tableName`, `tableDataPath`, `namespace`, and `partitionSpec` +optional. The config path may be local or on cloud storage, and each table is synced in turn. + +```yaml md title="dataset.yaml" +sourceFormat: HUDI +targetFormats: + - ICEBERG + - DELTA +datasets: + - tableBasePath: s3://tpcds-datasets/100GB/store_sales + - tableBasePath: s3://tpcds-datasets/100GB/store_returns + - tableBasePath: s3://tpcds-datasets/100GB/item +``` + +```shell md title="shell" +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + xtable-spark-runtime_2.12-.jar \ + --datasetconfig dataset.yaml +``` + +A complete example covering the 24 tables of the TPC-DS schema is provided at +[`xtable-spark-runtime/examples/tpcds-dataset.yaml`](https://github.com/apache/incubator-xtable/blob/main/xtable-spark-runtime/examples/tpcds-dataset.yaml). + +## Command-line options + +One of `--basepath` (single table) or `--datasetconfig` (multiple tables) is required; the two are +mutually exclusive. + +| Option | Required | Description | +| --- | --- | --- | +| `--basepath` | yes\* | Base path of the source table. Required unless `--datasetconfig` is given. | +| `--sourceformat` | yes\* | Source table format: `HUDI`, `ICEBERG`, `DELTA`, `PAIMON`, or `PARQUET`. Required unless `--datasetconfig` is given. | +| `--targets` | yes\* | Comma-separated target formats, e.g. `ICEBERG,DELTA`. Required unless `--datasetconfig` is given. | +| `--datasetconfig` | yes\* | Path (local or cloud) to a YAML dataset config for syncing multiple tables. Mutually exclusive with `--basepath`/`--sourceformat`/`--targets`. | +| `--datapath` | no | Path to the data files if different from the base path (e.g. Iceberg keeps data under `/data`). | +| `--tablename` | no | Table name; defaults to the last segment of the base path. | +| `--namespace` | no | Dot-separated table namespace. | +| `--partitionspec` | no | Hudi source partition field spec, e.g. `level:VALUE`. | +| `--usedeltakernel` | no | Force the Spark-free Delta Kernel for the Delta source/target. Auto-enabled on Spark 3.5+ (see below). | +| `--help` | no | Print usage. | + +## Supported formats + +Paimon and Parquet are read-only sources (there is no corresponding write target). + +| Source ↓ / Target → | Hudi | Iceberg | Delta | +| --- | :---: | :---: | :---: | +| **Hudi** | – | ✅ | ✅ | +| **Iceberg** | ✅ | – | ✅ | +| **Delta** | ✅ | ✅ | – | +| **Paimon** | ✅ | ✅ | ✅ | +| **Parquet** | ✅ | ✅ | ✅ | + +## Spark version compatibility + +Hudi and Iceberg conversion use only Spark-free core classes, so they run on any of the Spark lines +below. Delta is the only Spark-version-sensitive engine, and the runtime jar picks the right +implementation automatically: + +| Spark version | Hudi / Iceberg | Delta implementation | +| --- | :---: | --- | +| 3.4.x | ✅ | Delta Standalone (`delta-core`) | +| 3.5.x and newer | ✅ | Delta Kernel (Spark-free), selected automatically | + +On Spark 3.5+, the `delta-core` in the runtime jar does not run, so a Delta source or target is +routed through the Spark-free [Delta Kernel](https://docs.delta.io/latest/delta-kernel.html) +implementation automatically, with no flag needed. To force Kernel on any Spark version (e.g. Spark +3.4), pass `--usedeltakernel`. + +## Next steps + +- See [Installation](/docs/setup) for building the project. +- See the [Quickstart](/docs/how-to) for an end-to-end interoperability walkthrough. +- To query a synced table from Spark, see [Apache Spark](/docs/spark). diff --git a/website/sidebars.js b/website/sidebars.js index 9f0b55943..ce8c4b2cb 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -19,7 +19,8 @@ module.exports = { collapsed: false, items: [ 'how-to', - 'how-to-catalog-sync' + 'how-to-catalog-sync', + 'spark-runtime' ], }, { diff --git a/xtable-spark-runtime/README.md b/xtable-spark-runtime/README.md new file mode 100644 index 000000000..267d5bda6 --- /dev/null +++ b/xtable-spark-runtime/README.md @@ -0,0 +1,143 @@ + + +# XTable Spark Runtime + +`xtable-spark-runtime` publishes a self-contained (shaded, relocated) runtime jar that runs an +Apache XTable™ metadata sync with `spark-submit` on an existing Apache Spark cluster. It is the +`spark-submit` equivalent of the `RunSync` utility: no data is rewritten, only the target table +format metadata is generated alongside the existing data files. + +The engine libraries (Hudi, Iceberg, Delta, Avro, Parquet) are `provided`: the user brings their +own engine versions from the Spark runtime, so the thin runtime jar stays compatible across versions. + +## Build + +From the project root: + +```shell +./mvnw clean package -pl xtable-spark-runtime -am -DskipTests +``` + +The runtime jar is written to +`xtable-spark-runtime/target/xtable-spark-runtime_2.12-.jar`. + +## Usage + +```shell +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + --master 'local[*]' \ + xtable-spark-runtime_2.12-.jar \ + --basepath /path/to/hudi_table \ + --sourceformat HUDI \ + --targets ICEBERG,DELTA +``` + +| Option | Required | Description | +| --- | --- | --- | +| `--basepath` | yes\* | Base path of the source table. Required unless `--datasetconfig` is given. | +| `--sourceformat` | yes\* | `HUDI`, `ICEBERG`, `DELTA`, `PAIMON`, or `PARQUET`. Required unless `--datasetconfig` is given. | +| `--targets` | yes\* | Comma-separated target formats, e.g. `ICEBERG,DELTA`. Required unless `--datasetconfig` is given. | +| `--datasetconfig` | yes\* | Path (local or cloud) to a YAML dataset config for syncing multiple tables. Mutually exclusive with `--basepath`/`--sourceformat`/`--targets`. | +| `--datapath` | no | Path to the data files if different from the base path. | +| `--tablename` | no | Table name; defaults to the last segment of the base path. | +| `--namespace` | no | Dot-separated table namespace. | +| `--partitionspec` | no | Hudi source partition field spec, e.g. `level:VALUE`. | +| `--usedeltakernel` | no | Force the Spark-free Delta Kernel for Delta; auto-enabled on Spark 3.5+. | +| `--help` | no | Print usage. | + +Paimon and Parquet are read-only sources; targets are Hudi, Iceberg, and Delta. + +## Adding to an existing Spark job + +If a Spark job already writes a table in one format (for example Hudi) and the same table should +also be readable as Iceberg or Delta, the runtime jar's `provided` engine libraries let it reuse the +Hudi, Iceberg, and Delta libraries already on that job's Spark runtime; the only additional +artifact is this jar. Add it to that Spark runtime's classpath and run `XTableSparkSync` against the +table as a follow-on step after the write completes, with `--sourceformat` set to the format the job +writes and the formats to add listed in `--targets`: + +```shell +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + xtable-spark-runtime_2.12-.jar \ + --basepath s3://example-warehouse/db/orders \ + --sourceformat HUDI \ + --targets ICEBERG,DELTA +``` + +The sync reads the existing source-format metadata and writes the target metadata alongside the data +files the job already produced; no data is rewritten. + +## Sync multiple tables + +Pass `--datasetconfig` with a YAML file instead of the per-table flags to sync several tables in one +submit. `sourceFormat` and `targetFormats` apply to every table; each `datasets` entry needs only a +`tableBasePath` (`tableName`, `tableDataPath`, `namespace`, and `partitionSpec` are optional): + +```yaml +sourceFormat: HUDI +targetFormats: + - ICEBERG + - DELTA +datasets: + - tableBasePath: s3://tpcds-datasets/100GB/store_sales + - tableBasePath: s3://tpcds-datasets/100GB/store_returns + - tableBasePath: s3://tpcds-datasets/100GB/item +``` + +```shell +$SPARK_HOME/bin/spark-submit \ + --class org.apache.xtable.spark.XTableSparkSync \ + xtable-spark-runtime_2.12-.jar \ + --datasetconfig dataset.yaml +``` + +A complete example covering the 24 tables of the TPC-DS schema is at +[`examples/tpcds-dataset.yaml`](examples/tpcds-dataset.yaml). + +## Spark version compatibility + +Hudi and Iceberg conversion use only Spark-free core classes and run on any Spark line below. Delta +is the only Spark-version-sensitive engine, and the runtime jar selects the implementation automatically: + +| Spark version | Hudi / Iceberg | Delta | +| --- | :---: | --- | +| 3.4.x | ✅ | Delta Standalone (`delta-core`) | +| 3.5.x and newer | ✅ | Delta Kernel (Spark-free), selected automatically | + +On Spark 3.5+ the `delta-core` in the runtime jar does not run, so a Delta source/target is routed through the +Spark-free Delta Kernel automatically. Pass `--usedeltakernel` to force it on any Spark version. + +## Runtime jar validation IT + +`ITXTableSparkRuntimeBundle` `spark-submit`s the runtime jar against a real Spark distribution, one +case per direction across Hudi, Iceberg, and Delta, and asserts the target is data-equivalent to +the source. It self-skips unless `SPARK_HOME` is set, so it does not run in the normal build: + +```shell +SPARK_LOCAL_IP=127.0.0.1 SPARK_HOME=/path/to/spark-3.5.x-bin-hadoop3 \ + ./mvnw verify -pl xtable-spark-runtime +``` + +CI runs this IT on both Spark 3.4 and 3.5 (see +`.github/workflows/spark-runtime-validation.yml`). + +For more, see the docs: [Run XTable on Apache Spark](https://xtable.apache.org/docs/spark-runtime). diff --git a/xtable-spark-runtime/examples/tpcds-dataset.yaml b/xtable-spark-runtime/examples/tpcds-dataset.yaml new file mode 100644 index 000000000..0ee8c4373 --- /dev/null +++ b/xtable-spark-runtime/examples/tpcds-dataset.yaml @@ -0,0 +1,57 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +## Example --datasetconfig for a multi-table sync. This syncs the 24 tables of the +# TPC-DS schema from Hudi to Iceberg and Delta in a single spark-submit: +# +# $SPARK_HOME/bin/spark-submit \ +# --class org.apache.xtable.spark.XTableSparkSync \ +# xtable-spark-runtime_2.12-.jar \ +# --datasetconfig examples/tpcds-dataset.yaml +# +# sourceFormat and targetFormats apply to every table. Each entry needs only a +# tableBasePath; tableName, tableDataPath, namespace, and partitionSpec are optional +# (a Hudi source reads its partitioning from the table itself). Replace the bucket and +# prefix with the location of your tables. +sourceFormat: HUDI +targetFormats: + - ICEBERG + - DELTA +datasets: + - tableBasePath: s3://tpcds-datasets/100GB/call_center + - tableBasePath: s3://tpcds-datasets/100GB/catalog_page + - tableBasePath: s3://tpcds-datasets/100GB/catalog_returns + - tableBasePath: s3://tpcds-datasets/100GB/catalog_sales + - tableBasePath: s3://tpcds-datasets/100GB/customer + - tableBasePath: s3://tpcds-datasets/100GB/customer_address + - tableBasePath: s3://tpcds-datasets/100GB/customer_demographics + - tableBasePath: s3://tpcds-datasets/100GB/date_dim + - tableBasePath: s3://tpcds-datasets/100GB/household_demographics + - tableBasePath: s3://tpcds-datasets/100GB/income_band + - tableBasePath: s3://tpcds-datasets/100GB/inventory + - tableBasePath: s3://tpcds-datasets/100GB/item + - tableBasePath: s3://tpcds-datasets/100GB/promotion + - tableBasePath: s3://tpcds-datasets/100GB/reason + - tableBasePath: s3://tpcds-datasets/100GB/ship_mode + - tableBasePath: s3://tpcds-datasets/100GB/store + - tableBasePath: s3://tpcds-datasets/100GB/store_returns + - tableBasePath: s3://tpcds-datasets/100GB/store_sales + - tableBasePath: s3://tpcds-datasets/100GB/time_dim + - tableBasePath: s3://tpcds-datasets/100GB/warehouse + - tableBasePath: s3://tpcds-datasets/100GB/web_page + - tableBasePath: s3://tpcds-datasets/100GB/web_returns + - tableBasePath: s3://tpcds-datasets/100GB/web_sales + - tableBasePath: s3://tpcds-datasets/100GB/web_site