build: enable Spark SQL tests for released Spark 4.2.0

.github/workflows/README.md

@@ -43,6 +43,12 @@ required one, so a red 3.4 there changes nothing. It is the next push with
 the label still applied that runs 3.4 under `Required Checks`, and with the
 queue run gone that push is the only thing that makes a 3.4 failure blocking.
 
+`spark_4_2` is in the nightly tier despite Spark 4.2 support being
+experimental. Nightly is what keeps `dev/diffs/4.2.0.diff` honest: the diff
+files for the supported versions are updated together whenever a Comet change
+needs one, and a 4.2 suite that only ran on request would let its diff rot
+unnoticed between requests.
+
 Heavy jobs have no `push` tier. The queue already tested the exact tree that
 lands, so re-running them on push to main would double the cost of every
 merge. Two jobs are still on `push`: `docs`, because it deploys to `asf-site`
@@ -113,6 +119,7 @@ tiers partition the list and that the `pr` tier is exactly the default profile.
                                         (other profiles)
                                       spark_3_5           run-spark-3.5-tests
                                       spark_4_0           run-spark-4.0-tests
+                                      spark_4_2           run-spark-4.2-tests
                                       iceberg_1_8         run-iceberg-tests
                                       iceberg_1_9         run-iceberg-tests
                                       iceberg_1_10        run-iceberg-tests
@@ -157,6 +164,7 @@ tiers partition the list and that the `pr` tier is exactly the default profile.
 | `spark_4_1`          | merge group, **or** PR with `run-spark-4.1-tests`; the `sql_hive` shards alone with `run-spark-4.1-hive-tests`                                                                                                                                         | Spark 4.1 sources                   |
 | `spark_3_4`          | PR with `run-spark-3.4-tests`, or dispatch                                                                                                                                                                                                             | Spark 3.4 sources                   |
 | `spark_4_0`          | nightly, **or** PR with `run-spark-4.0-tests`                                                                                                                                                                                                          | Spark 4.0 sources                   |
+| `spark_4_2`          | nightly, **or** PR with `run-spark-4.2-tests`                                                                                                                                                                                                          | Spark 4.2 sources                   |
 | `iceberg_1_11`       | merge group, **or** PR with `run-iceberg-tests`                                                                                                                                                                                                        | Iceberg sources                     |
 | `iceberg_1_8`        | nightly, **or** PR with `run-iceberg-tests`                                                                                                                                                                                                            | Iceberg sources                     |
 | `iceberg_1_9`        | nightly, **or** PR with `run-iceberg-tests`                                                                                                                                                                                                            | Iceberg sources                     |
@@ -274,16 +282,16 @@ umbrella doesn't watch, or operate independently of the rest of CI:
 
 ## Reusable workflows (called by `ci.yml`)
 
-| File                              | Called from `ci.yml` job(s)                                  |
-| --------------------------------- | ------------------------------------------------------------ |
-| `pr_build_linux.yml`              | `pr_build_linux`                                             |
-| `pr_build_macos.yml`              | `pr_build_macos`                                             |
-| `pr_benchmark_check.yml`          | `pr_benchmark_check`                                         |
-| `delta_build_gate.yml`            | `delta_build_gate`                                           |
-| `pyarrow_udf_test.yml`            | `pyarrow_udf_test`                                           |
-| `docs.yaml`                       | `docs`                                                       |
-| `spark_sql_test_reusable.yml`     | `spark_3_4`, `spark_3_5`, `spark_4_0`, `spark_4_1`           |
-| `iceberg_spark_test_reusable.yml` | `iceberg_1_8`, `iceberg_1_9`, `iceberg_1_10`, `iceberg_1_11` |
+| File                              | Called from `ci.yml` job(s)                                     |
+| --------------------------------- | --------------------------------------------------------------- |
+| `pr_build_linux.yml`              | `pr_build_linux`                                                |
+| `pr_build_macos.yml`              | `pr_build_macos`                                                |
+| `pr_benchmark_check.yml`          | `pr_benchmark_check`                                            |
+| `delta_build_gate.yml`            | `delta_build_gate`                                              |
+| `pyarrow_udf_test.yml`            | `pyarrow_udf_test`                                              |
+| `docs.yaml`                       | `docs`                                                          |
+| `spark_sql_test_reusable.yml`     | `spark_3_4`, `spark_3_5`, `spark_4_0`, `spark_4_1`, `spark_4_2` |
+| `iceberg_spark_test_reusable.yml` | `iceberg_1_8`, `iceberg_1_9`, `iceberg_1_10`, `iceberg_1_11`    |
 
 ## Changing what runs when
 

.github/workflows/ci.yml

@@ -32,8 +32,8 @@
 #                 the default profile's.
 #   schedule      the nightly regression sweep of everything else: the Linux
 #                 test matrix against the other Spark profiles, Spark SQL on
-#                 Spark 3.5 and 4.0, and Iceberg 1.8/1.9/1.10, routed by the
-#                 same path filters over what landed on main since the last
+#                 Spark 3.5, 4.0 and 4.2, and Iceberg 1.8/1.9/1.10, routed by
+#                 the same path filters over what landed on main since the last
 #                 green nightly. A failure opens (or comments on) an issue
 #                 labelled `ci-nightly-failure`.
 #
@@ -97,6 +97,14 @@ on:
     - cron: '0 6 * * *'
   workflow_dispatch:
 
+# Least-privilege default for the whole pipeline: every job here builds and tests,
+# and nothing needs to write back to the repository. Jobs that call reusable
+# workflows inherit this, and a callee can only narrow it further -- so the one
+# job that does need to write (docs, which pushes the built site to asf-site)
+# raises it to `contents: write` on itself.
+permissions:
+  contents: read
+
 jobs:
   # ---------------------------------------------------------------------------
   # preflight: cheap checks that gate everything else. Failure short-circuits
@@ -243,6 +251,7 @@ jobs:
       spark_4_0:    ${{ steps.compute.outputs.spark_4_0 }}
       spark_4_1:    ${{ steps.compute.outputs.spark_4_1 }}
       spark_4_1_hive: ${{ steps.compute.outputs.spark_4_1_hive }}
+      spark_4_2:    ${{ steps.compute.outputs.spark_4_2 }}
       iceberg_1_8:  ${{ steps.compute.outputs.iceberg_1_8 }}
       iceberg_1_9:  ${{ steps.compute.outputs.iceberg_1_9 }}
       iceberg_1_10: ${{ steps.compute.outputs.iceberg_1_10 }}
@@ -399,6 +408,10 @@ jobs:
     needs: changes
     # docs deploys to asf-site, so only run on push-to-main (or a manual dispatch).
     if: needs.changes.outputs.docs == 'true'
+    # docs.yaml commits and pushes the generated site to the asf-site branch, so
+    # this job needs write access on top of the read-only default above.
+    permissions:
+      contents: write
     uses: ./.github/workflows/docs.yaml
 
   spark_3_4:
@@ -467,6 +480,20 @@ jobs:
             || needs.changes.outputs.spark_4_1_hive != 'true' && 'core'
             || 'all' }}
 
+  spark_4_2:
+    name: Spark SQL Tests (Spark 4.2)
+    needs: changes
+    # Nightly by default; PRs need the `run-spark-4.2-tests` label. Spark 4.2
+    # support is still experimental, so it sits in the nightly tier rather
+    # than the queue: a regression there blocks no merge, but it is caught
+    # within a day instead of whenever someone next asks for the suite.
+    if: needs.changes.outputs.spark_4_2 == 'true'
+    uses: ./.github/workflows/spark_sql_test_reusable.yml
+    with:
+      spark-short: '4.2'
+      spark-full: '4.2.0'
+      java: 17
+
   iceberg_1_8:
     name: Iceberg Spark SQL Tests (Iceberg 1.8)
     needs: changes
@@ -570,6 +597,7 @@ jobs:
       - spark_3_5
       - spark_4_0
       - spark_4_1
+      - spark_4_2
       - iceberg_1_8
       - iceberg_1_9
       - iceberg_1_10

dev/ci/check-ci-config.py

@@ -108,6 +108,7 @@
     "spark_4_0",
     "spark_4_1",
     "spark_4_1_hive",
+    "spark_4_2",
     "iceberg_1_8",
     "iceberg_1_9",
     "iceberg_1_10",
@@ -191,6 +192,7 @@
 NIGHTLY_TIER = {
     "spark_3_5",
     "spark_4_0",
+    "spark_4_2",
     "iceberg_1_8",
     "iceberg_1_9",
     "iceberg_1_10",
@@ -350,6 +352,21 @@
         },
         {"spark_3_4"},
     ),
+    # Spark 4.2 runs nightly, and its label brings that run forward onto a pull
+    # request, the same shape as 3.5 and 4.0. It pulls in no other Spark version.
+    (
+        {"name": "pull_request", "action": "synchronize", "labels": ["run-spark-4.2-tests"]},
+        PR_TIER | {"spark_4_2"},
+    ),
+    (
+        {
+            "name": "pull_request",
+            "action": "labeled",
+            "label": "run-spark-4.2-tests",
+            "labels": ["run-spark-4.2-tests"],
+        },
+        {"spark_4_2"},
+    ),
     (
         {"name": "pull_request", "action": "synchronize", "labels": ["run-iceberg-tests"]},
         PR_TIER | ICEBERG_OPT_IN,

dev/ci/compute-changes.py

@@ -297,6 +297,34 @@
         ".mvn/**",
         "mvnw",
     ],
+    "spark_4_2": [
+        "native/**/src/**",
+        "native/**/Cargo.toml",
+        "native/Cargo.lock",
+        "common/src/main/**",
+        "common/pom.xml",
+        "spark/src/main/**",
+        "!spark/src/main/spark-3.4/**",
+        "!spark/src/main/spark-3.5/**",
+        "!spark/src/main/spark-3.x/**",
+        "!spark/src/main/spark-4.0/**",
+        "!spark/src/main/spark-4.1/**",
+        "!spark/src/main/scala/org/apache/comet/GenerateDocs.scala",
+        "spark/pom.xml",
+        "dev/diffs/4.2.0.diff",
+        "pom.xml",
+        "rust-toolchain.toml",
+        ".github/workflows/ci.yml",
+        ".github/workflows/spark_sql_test_reusable.yml",
+        "dev/ci/spark-sql-modules.py",
+        ".github/actions/setup-builder/**",
+        ".github/actions/setup-spark-builder/**",
+        ".github/actions/upload-artifact-retry/**",
+        ".github/actions/download-artifact-retry/**",
+        ".github/actions/maven-bootstrap/**",
+        ".mvn/**",
+        "mvnw",
+    ],
     # Same inputs as spark_4_1: this is not a separate job but a second
     # POLICY decision for the same call, selecting the sql_hive matrix rows.
     # ci.yml folds the two outputs into the reusable workflow's `modules`
@@ -489,8 +517,8 @@
     # anyone who wants to check a change against 3.4 still can.
     "spark_3_4": ["label:run-spark-3.4-tests"],
     # Spark 4.1 is the default build profile and the one Spark SQL suite the
-    # queue runs; 3.5 and 4.0 run nightly, or on a pull request with their
-    # label.
+    # queue runs; 3.5, 4.0 and 4.2 run nightly, or on a pull request with
+    # their label.
     "spark_3_5": ["nightly", "label:run-spark-3.5-tests"],
     "spark_4_0": ["nightly", "label:run-spark-4.0-tests"],
     # No Spark SQL suite runs on a plain pull request. Spark 4.1 was the last
@@ -508,6 +536,14 @@
         "label:run-spark-4.1-tests",
         "label:run-spark-4.1-hive-tests",
     ],
+    # Spark 4.2 support is experimental, but the suite passes, so it sits in
+    # the nightly tier with the other non-default versions rather than being
+    # reachable only on demand. Nightly is the right tier for it twice over:
+    # a 4.2 regression blocks nobody's merge, and running it every night is
+    # what keeps the 4.2 diff in `dev/diffs` from silently rotting as the
+    # other diffs are updated -- the failure mode an on-demand suite hides
+    # until someone thinks to ask for it.
+    "spark_4_2": ["nightly", "label:run-spark-4.2-tests"],
     # Same shape for Iceberg: 1.11 is the only Spark 4.1 coverage, so it is
     # the one Iceberg version the queue runs; the three older versions run
     # nightly. One label opts a pull request into all four.