Enable SPARK-57298 collect_set tests when Spark 4.2 SQL diff lands

.github/workflows/README.md

@@ -43,6 +43,12 @@ required one, so a red 3.4 there changes nothing. It is the next push with
 the label still applied that runs 3.4 under `Required Checks`, and with the
 queue run gone that push is the only thing that makes a 3.4 failure blocking.
 
+`spark_4_2` is outside the tiers for the opposite reason: Spark 4.2 support
+is new and experimental rather than deprecated. It runs only when a pull
+request carries `run-spark-4.2-tests`, or from a `workflow_dispatch`, and the
+same label-run caveat above applies to it. Moving it into the nightly sweep
+is a follow-up once the version has settled.
+
 Heavy jobs have no `push` tier. The queue already tested the exact tree that
 lands, so re-running them on push to main would double the cost of every
 merge. Two jobs are still on `push`: `docs`, because it deploys to `asf-site`
@@ -107,6 +113,7 @@ tiers partition the list and that the `pr` tier is exactly the default profile.
   label or dispatch only                                        spark_4_1 sql_hive  run-spark-4.1-hive-tests
   ----------------------                                        iceberg_1_11        run-iceberg-tests
   spark_3_4  run-spark-3.4-tests
+  spark_4_2  run-spark-4.2-tests
                                       nightly tier, or PR with label
                                       ------------------------------
                                       pr_build_linux      run-all-spark-profiles
@@ -134,6 +141,9 @@ tiers partition the list and that the `pr` tier is exactly the default profile.
                                 |  issue                |
                                 +-----------------------+
 
+                                                  spark_4_2
+                                            (workflow_dispatch only)
+
   reusable workflows invoked via `uses:`:
     pr_build_linux.yml         spark_sql_test_reusable.yml
     pr_build_macos.yml         iceberg_spark_test_reusable.yml
@@ -157,6 +167,7 @@ tiers partition the list and that the `pr` tier is exactly the default profile.
 | `spark_4_1`          | merge group, **or** PR with `run-spark-4.1-tests`; the `sql_hive` shards alone with `run-spark-4.1-hive-tests`                                                                                                                                         | Spark 4.1 sources                   |
 | `spark_3_4`          | PR with `run-spark-3.4-tests`, or dispatch                                                                                                                                                                                                             | Spark 3.4 sources                   |
 | `spark_4_0`          | nightly, **or** PR with `run-spark-4.0-tests`                                                                                                                                                                                                          | Spark 4.0 sources                   |
+| `spark_4_2`          | PR with `run-spark-4.2-tests`, or dispatch                                                                                                                                                                                                             | Spark 4.2 sources                   |
 | `iceberg_1_11`       | merge group, **or** PR with `run-iceberg-tests`                                                                                                                                                                                                        | Iceberg sources                     |
 | `iceberg_1_8`        | nightly, **or** PR with `run-iceberg-tests`                                                                                                                                                                                                            | Iceberg sources                     |
 | `iceberg_1_9`        | nightly, **or** PR with `run-iceberg-tests`                                                                                                                                                                                                            | Iceberg sources                     |
@@ -261,16 +272,16 @@ umbrella doesn't watch, or operate independently of the rest of CI:
 
 ## Reusable workflows (called by `ci.yml`)
 
-| File                              | Called from `ci.yml` job(s)                                  |
-| --------------------------------- | ------------------------------------------------------------ |
-| `pr_build_linux.yml`              | `pr_build_linux`                                             |
-| `pr_build_macos.yml`              | `pr_build_macos`                                             |
-| `pr_benchmark_check.yml`          | `pr_benchmark_check`                                         |
-| `delta_build_gate.yml`            | `delta_build_gate`                                           |
-| `pyarrow_udf_test.yml`            | `pyarrow_udf_test`                                           |
-| `docs.yaml`                       | `docs`                                                       |
-| `spark_sql_test_reusable.yml`     | `spark_3_4`, `spark_3_5`, `spark_4_0`, `spark_4_1`           |
-| `iceberg_spark_test_reusable.yml` | `iceberg_1_8`, `iceberg_1_9`, `iceberg_1_10`, `iceberg_1_11` |
+| File                              | Called from `ci.yml` job(s)                                     |
+| --------------------------------- | --------------------------------------------------------------- |
+| `pr_build_linux.yml`              | `pr_build_linux`                                                |
+| `pr_build_macos.yml`              | `pr_build_macos`                                                |
+| `pr_benchmark_check.yml`          | `pr_benchmark_check`                                            |
+| `delta_build_gate.yml`            | `delta_build_gate`                                              |
+| `pyarrow_udf_test.yml`            | `pyarrow_udf_test`                                              |
+| `docs.yaml`                       | `docs`                                                          |
+| `spark_sql_test_reusable.yml`     | `spark_3_4`, `spark_3_5`, `spark_4_0`, `spark_4_1`, `spark_4_2` |
+| `iceberg_spark_test_reusable.yml` | `iceberg_1_8`, `iceberg_1_9`, `iceberg_1_10`, `iceberg_1_11`    |
 
 ## Changing what runs when
 

.github/workflows/ci.yml

@@ -77,6 +77,14 @@ on:
     - cron: '0 6 * * *'
   workflow_dispatch:
 
+# Least-privilege default for the whole pipeline: every job here builds and tests,
+# and nothing needs to write back to the repository. Jobs that call reusable
+# workflows inherit this, and a callee can only narrow it further -- so the one
+# job that does need to write (docs, which pushes the built site to asf-site)
+# raises it to `contents: write` on itself.
+permissions:
+  contents: read
+
 jobs:
   # ---------------------------------------------------------------------------
   # preflight: cheap checks that gate everything else. Failure short-circuits
@@ -191,6 +199,7 @@ jobs:
       spark_4_0:    ${{ steps.compute.outputs.spark_4_0 }}
       spark_4_1:    ${{ steps.compute.outputs.spark_4_1 }}
       spark_4_1_hive: ${{ steps.compute.outputs.spark_4_1_hive }}
+      spark_4_2:    ${{ steps.compute.outputs.spark_4_2 }}
       iceberg_1_8:  ${{ steps.compute.outputs.iceberg_1_8 }}
       iceberg_1_9:  ${{ steps.compute.outputs.iceberg_1_9 }}
       iceberg_1_10: ${{ steps.compute.outputs.iceberg_1_10 }}
@@ -347,6 +356,10 @@ jobs:
     needs: changes
     # docs deploys to asf-site, so only run on push-to-main (or a manual dispatch).
     if: needs.changes.outputs.docs == 'true'
+    # docs.yaml commits and pushes the generated site to the asf-site branch, so
+    # this job needs write access on top of the read-only default above.
+    permissions:
+      contents: write
     uses: ./.github/workflows/docs.yaml
 
   spark_3_4:
@@ -415,6 +428,22 @@ jobs:
             || needs.changes.outputs.spark_4_1_hive != 'true' && 'core'
             || 'all' }}
 
+  spark_4_2:
+    name: Spark SQL Tests (Spark 4.2)
+    needs: changes
+    # On demand only for now: Spark 4.2 support is brand new, so it sits
+    # outside the pull request, queue and nightly tiers, exactly like the
+    # deprecated 3.4 suite. It runs from `run-spark-4.2-tests` on a pull
+    # request, or from a workflow_dispatch; POLICY in compute-changes.py is
+    # where that is expressed, not an event check here. Folding it into the
+    # nightly is a follow-up once the version has settled.
+    if: needs.changes.outputs.spark_4_2 == 'true'
+    uses: ./.github/workflows/spark_sql_test_reusable.yml
+    with:
+      spark-short: '4.2'
+      spark-full: '4.2.0'
+      java: 17
+
   iceberg_1_8:
     name: Iceberg Spark SQL Tests (Iceberg 1.8)
     needs: changes
@@ -517,6 +546,7 @@ jobs:
       - spark_3_5
       - spark_4_0
       - spark_4_1
+      - spark_4_2
       - iceberg_1_8
       - iceberg_1_9
       - iceberg_1_10

dev/ci/check-ci-config.py

@@ -103,6 +103,7 @@
     "spark_4_0",
     "spark_4_1",
     "spark_4_1_hive",
+    "spark_4_2",
     "iceberg_1_8",
     "iceberg_1_9",
     "iceberg_1_10",
@@ -185,10 +186,16 @@
 # `schedule` cases below assert its absence rather than quietly accept it
 # coming back.
 SPARK_DEPRECATED = {"spark_3_4"}
+# Spark 4.2 is experimental rather than deprecated, but it is routed the same
+# way: outside the queue and nightly tiers entirely, reachable through
+# `run-spark-4.2-tests` on a pull request or a workflow_dispatch. Keeping it
+# in its own set is what makes the `merge_group` and `schedule` cases below
+# assert its absence rather than quietly accept it arriving.
+SPARK_EXPERIMENTAL = {"spark_4_2"}
 # One label opts a pull request into every Iceberg version, whichever tier
 # each sits in.
 ICEBERG_OPT_IN = {"iceberg_1_8", "iceberg_1_9", "iceberg_1_10", "iceberg_1_11"}
-ALL_JOBS = QUEUE_TIER | NIGHTLY_TIER | SPARK_DEPRECATED | {"docs"}
+ALL_JOBS = QUEUE_TIER | NIGHTLY_TIER | SPARK_DEPRECATED | SPARK_EXPERIMENTAL | {"docs"}
 assert not QUEUE_TIER & NIGHTLY_TIER, "a job is queue or nightly, never both"
 
 POLICY_CASES = [
@@ -333,6 +340,21 @@
         },
         {"spark_3_4"},
     ),
+    # Spark 4.2 is experimental and routed like 3.4: the label is the only way
+    # it reaches a pull request, and it pulls in no other Spark version.
+    (
+        {"name": "pull_request", "action": "synchronize", "labels": ["run-spark-4.2-tests"]},
+        PR_TIER | {"spark_4_2"},
+    ),
+    (
+        {
+            "name": "pull_request",
+            "action": "labeled",
+            "label": "run-spark-4.2-tests",
+            "labels": ["run-spark-4.2-tests"],
+        },
+        {"spark_4_2"},
+    ),
     (
         {"name": "pull_request", "action": "synchronize", "labels": ["run-iceberg-tests"]},
         PR_TIER | ICEBERG_OPT_IN,

dev/ci/compute-changes.py

@@ -286,6 +286,33 @@
         ".mvn/**",
         "mvnw",
     ],
+    "spark_4_2": [
+        "native/**/src/**",
+        "native/**/Cargo.toml",
+        "native/Cargo.lock",
+        "common/src/main/**",
+        "common/pom.xml",
+        "spark/src/main/**",
+        "!spark/src/main/spark-3.4/**",
+        "!spark/src/main/spark-3.5/**",
+        "!spark/src/main/spark-3.x/**",
+        "!spark/src/main/spark-4.0/**",
+        "!spark/src/main/spark-4.1/**",
+        "!spark/src/main/scala/org/apache/comet/GenerateDocs.scala",
+        "spark/pom.xml",
+        "dev/diffs/4.2.0.diff",
+        "pom.xml",
+        "rust-toolchain.toml",
+        ".github/workflows/ci.yml",
+        ".github/workflows/spark_sql_test_reusable.yml",
+        "dev/ci/spark-sql-modules.py",
+        ".github/actions/setup-builder/**",
+        ".github/actions/setup-spark-builder/**",
+        ".github/actions/upload-artifact-retry/**",
+        ".github/actions/download-artifact-retry/**",
+        ".mvn/**",
+        "mvnw",
+    ],
     # Same inputs as spark_4_1: this is not a separate job but a second
     # POLICY decision for the same call, selecting the sql_hive matrix rows.
     # ci.yml folds the two outputs into the reusable workflow's `modules`
@@ -493,6 +520,11 @@
         "label:run-spark-4.1-tests",
         "label:run-spark-4.1-hive-tests",
     ],
+    # Spark 4.2 support is new and experimental, so like the deprecated 3.4
+    # suite it sits outside all three tiers: it runs from `run-spark-4.2-tests`
+    # on a pull request, or a workflow_dispatch, and gates no merge. Folding it
+    # into the nightly is the follow-up once the version has settled.
+    "spark_4_2": ["label:run-spark-4.2-tests"],
     # Same shape for Iceberg: 1.11 is the only Spark 4.1 coverage, so it is
     # the one Iceberg version the queue runs; the three older versions run
     # nightly. One label opts a pull request into all four.