Conversation
Adds dev/diffs/4.2.0-preview4.diff so the Spark SQL test suite can run against Apache Spark 4.2.0-preview4 with Comet enabled, and wires the 4.2 profile into the spark_sql_test workflow matrix. The diff was seeded from 4.1.1.diff and reconciled against v4.2.0-preview4.
Inherited from 4.1.1.diff during reconciliation. The Spark build was trying to resolve comet-spark-spark4.1_2.13 instead of the 4.2 artifact, causing the sql_hive jobs to fail before any tests ran.
Spark 4.2 removes GeographyVal / GeometryVal and replaces the getGeography and getGeometry accessors on SpecializedGetters with a single getBinaryView returning BinaryView. The shim therefore can no longer be shared between 4.1 and 4.2 out of the spark-4.1+ source root: the major, minor-plus and minor shim directories are all added as compile source roots, so a spark-4.2 copy would be a duplicate class rather than an override. Move the trait into per-version spark-4.1 and spark-4.2 copies.
Point spark.version at the released 4.2.0 instead of 4.2.0-preview4. Spark 4.2.0 bundles its own copy of org.apache.datasketches.memory.internal.ResourceImpl inside spark-catalyst, which collides with the transitive datasketches-memory jar and fails the maven-enforcer BanDuplicateClasses rule, so ignore that class. Keep the test-scope Jetty pin at 11.0.26. Spark 4.2.0 ships Jetty 12, but the Iceberg REST catalog test helper needs jetty-servlet, which Jetty 12 replaced with jetty-ee10-servlet.
Reseed the Spark SQL test overrides from 4.1.2.diff against the v4.2.0 tag and drop the 4.2.0-preview4 diff. The notable reconciliation is the test harness refactor: Spark 4.2 shrinks SQLTestUtils to a deprecated empty alias and moves withSQLConf, stripSparkFilter and the test() override into QueryTest, with SharedSparkSession now extending QueryTest directly. Comet's hooks (isCometEnabled, the IgnoreComet skip and the stripSparkFilter Comet cases) move to QueryTest accordingly. isCometEnabled has to qualify classic.SparkSession there, since a bare SparkSession in that package resolves to the unified class.
Regenerated with dev/regenerate-golden-files.sh --spark-version 4.2. The q2, q5, q54 (v1_4) and q5a (v2_7) goldens are pruned: their plans now match the shared fallback under the released 4.2.0, so the divergences seen on 4.2.0-preview4 are gone. q77a (v2_7) is added because Spark 4.2 plans a OneRowRelation into its Union branches. Comet cannot convert that leaf, so the Unions fall back and the aggregates above them lose their Comet partial producer. The approved plan records the degraded plan; see apache#4949.
Add a spark_4_2 job to the ci.yml umbrella calling spark_sql_test_reusable.yml with Spark 4.2.0 on JDK 17, plus the matching spark_4_2 filter in compute-changes.py and output on the changes job. Gate it behind the run-spark-4.2-tests label so it runs on pushes to main but stays off the default PR path while 4.2 support is experimental. Update the user guide for the released 4.2.0 and refresh the new-Spark-version contributor guide, which still described the spark_sql_test.yml matrix that was replaced by the per-version umbrella jobs.
The preflight job only proceeds on a labeled event for known gating labels. Without run-spark-4.2-tests in that allowlist, labelling a PR to request the Spark 4.2 SQL tests skipped the whole pipeline instead of running them.
Spark 4.2.0 is now in Maven Central, so the spark-4.2 profile no longer needs to target a preview. Point spark.version at 4.2.0 and make the main sources compile against it. - Drop the WIP compile-only comment on the spark-4.2 profile. - Spark 4.2.0 bundles its own copy of org.apache.datasketches.memory.internal.ResourceImpl inside spark-catalyst, colliding with the transitive datasketches-memory jar and failing the BanDuplicateClasses enforcer rule; ignore that class. - Spark 4.2 removes GeographyVal / GeometryVal and replaces getGeography / getGeometry on SpecializedGetters with a single getBinaryView returning BinaryView. CometInternalRowShim can no longer be shared from the spark-4.1+ source root, so split it into per-version spark-4.1 and spark-4.2 copies. - Correct the stale Jetty comment that referenced preview4. This does not wire up the Spark SQL tests for 4.2; it only moves the profile onto the released version and keeps it compiling.
Regenerated with dev/regenerate-golden-files.sh --spark-version 4.2 against the released 4.2.0. The q2, q5, q54 (v1_4) and q5a (v2_7) goldens are pruned because their plans now match the shared fallback on 4.2.0, and q77a (v2_7) is added to record a degraded plan.
Two Spark 4.2.0 changes surfaced by moving the profile off preview4: Iceberg scans job: Spark 4.2.0 turned `connector.catalog.View` from an interface into a class. No Iceberg spark-runtime is published for 4.2, so the build reuses the 4.0 runtime, whose `SparkView implements View` now throws IncompatibleClassChangeError at class-load and aborts the Iceberg suites. Report Iceberg as unavailable on Spark 4.2 in the shared probes so the suites skip. Guard the fuzz suite's beforeAll with an early return (cancelling from beforeAll aborts the suite) and add the missing `assume(icebergAvailable)` to one native-scan test. Expressions job: Spark 4.2 normalizes NaN / -0.0 for array_distinct and the array set operations by wrapping their inputs as `KnownFloatingPointNormalized(ArrayTransform(arr, x -> NormalizeNaNAndZero(x)))`. The serde only handled a scalar `NormalizeNaNAndZero` child and fell back. Since `KnownFloatingPointNormalized` is a runtime no-op tag, serialize any other child directly and let its serde (the ArrayTransform codegen dispatcher) carry the normalization, keeping these operations native.
# Conflicts: # spark/src/main/scala/org/apache/comet/serde/contraintExpressions.scala
CometNativeScan serializes the full data and partition schema, not just the required columns. A table with a GEOMETRY or GEOGRAPHY column (Spark 4.2) has no proto representation for those types, so schema2Proto threw NoSuchElementException during planning instead of falling back. Reject the native scan when any data or partition schema field type cannot be serialized.
…issues Handle new-in-4.2 Spark SQL test suites that assert on Spark-internal execution mechanics Comet replaces wholesale, and repoint deferred assumes at specific issues instead of the closed generic tracker (apache#4142). - SLAM metric suites, segment-tree window suites, and the UnionExec whole-stage-codegen test are marked IgnoreComet / IgnoreCometSuite (apache#4963, apache#4964, apache#4965). - collect_set NaN/-0.0 normalization deferred (apache#4966). - Repoint assume(!isSpark42Plus) guards to specific issues: apache#4967 (ANSI arithmetic), apache#4968 (BloomFilter), apache#4969 (Iceberg REST catalog). - Geometry/geography scan tests now pass via native-scan fallback.
CodeQL flagged ci.yml as not limiting the GITHUB_TOKEN, so it inherits whatever the repository default is. Adds a top-level `permissions: contents: read`, matching the convention already used by codeql.yml and pyarrow_udf_test.yml. A blanket read-only default is not quite safe here, because ci.yml calls six reusable workflows and a callee inherits the caller's permissions and can only narrow them further. Audited all six: docs.yaml commits and pushes the generated site to the asf-site branch, so it needs write; the other five (pr_build_linux, pr_build_macos, pr_benchmark_check, spark_sql_test_reusable, iceberg_spark_test_reusable) only build and test. So the `docs` job raises itself to `contents: write` and nothing else changes. That job already only runs on push-to-main or workflow_dispatch, never on pull requests. Also checked what else might quietly depend on write access: no job in ci.yml or in any callee uses GITHUB_TOKEN, secrets, the gh CLI, git push (other than docs.yaml), containers or packages. The upload/download-artifact steps are all same-run, which uses the Actions runtime token rather than GITHUB_TOKEN, and none pass run-id or github-token, so none need `actions: read`. Verified the file parses and the blocks land where intended (top-level contents: read, docs job contents: write, no other job overriding). actionlint could not be run locally -- its installer is a curl-to-shell script -- so it will be exercised by the preflight job on this PR.
`PR Build [expressions]` (Spark 4.0 / 4.1 / 4.2) failed on `expressions/misc/variant.sql`: the new full-data-schema serializability check in `CometNativeScan.isSupported` ran before the projected-schema type check, so a `struct<v: variant>` column was reported as "Native scan does not support data type struct<v:variant>" instead of the expected "type VariantType". Run the schema-support check first so the more specific type reason wins; the geospatial case that motivated the check still falls back (and no longer crashes planning), now covered by a new `expressions/misc/geospatial_types.sql` fixture. `Spark SQL Tests (Spark 4.2)` sql_core failures: - Spark 4.2 allows `FILTER (WHERE ...)` on window aggregates. Comet dropped the filter (it is only serialized for Partial mode aggregates, and window aggregates are Complete mode) and returned wrong results for `window.sql`. Decline these window expressions in `CometWindowExec`, and turn the silent drop in `aggExprToProto` into a fallback for any non-Partial mode carrying a filter. Covered by `expressions/window/window_filter.sql`. - `dev/diffs/4.2.0.diff` was reseeded from `4.1.2.diff` and missed the updates main has since made to the 4.1 diff, which caused the `SubquerySuite`, `CachedBatchSerializerNoUnwrapSuite` and `DataFrameAggregateSuite` (x4 suites) failures. Ported: the `CometHashAggregateExec` arm for SPARK-22223, the `WholeStageCodegenExec(CometColumnarToRowExec(...))` scan shapes for SPARK-26893, the codegen-aware cached-plan transition check, the `VariantEndToEndSuite` patch, `spark.comet.shuffle.enabled` for the renamed config, and `comet.version` 1.1.0-SNAPSHOT. - `tuplesketch.sql` hits the same collated-string sketch problem as `thetasketch.sql`, so it joins the ignore list. `join-nearest-by.sql` records Spark operator names in EXPLAIN goldens and `in-order-by.sql` orders by a column with ties, so both run with Comet disabled (as explain*.sql and in-limit.sql already do). Also merges apache/main, which the branch needed for the diff sync.
# Conflicts: # docs/source/user-guide/latest/compatibility/spark-versions.md # docs/source/user-guide/latest/installation.md
Resolve the semantic conflicts left by merging apache/main: - `withFallbackReason` lost its roll-up varargs in apache#5236, which now happens centrally in `CometExecRule.rollUpFallbackReasons`. Drop the third argument from the two Spark 4.2 FILTER (WHERE ...) call sites. - Mirror the `RemoveRedundantWindowGroupLimitsSuite` change from apache#4870 into `dev/diffs/4.2.0.diff` so the suite accepts `CometWindowGroupLimitExec`.
- `CometNativeScan.isSupported` validated the raw relation data schema, so an unprojected `ARRAY<VARIANT>` / `MAP<STRING, VARIANT>` root forced a fallback even though `convert` prunes exactly those roots before serializing. Share the pruning between the two and validate the schema the scan actually serializes, keeping the full partition schema check. This is the `expressions/misc/variant.sql` failure on the Spark 4.0, 4.1 and 4.2 `[expressions]` jobs. - `extractWindowExpression` rebuilds the window expression for the decimal SUM / AVG shapes Spark's `DecimalAggregates` rule wraps, so the FILTER guard tagged a detached copy and `rollUpFallbackReasons` never saw it: strict mode reported an unexplained Window fallback. Lift the reasons onto the operator's own expression and cover decimal SUM / AVG FILTER in `window_filter.sql`. - `dev/diffs/4.2.0.diff`: restore the upstream `localReads.length == 2` assertion. The test is `IgnoreComet`-tagged, so its body only runs with Comet disabled, where the lowered count contradicted the `localReads(1)` access on the next line. - `ci.yml`: add `spark_4_2` to the `workflow_dispatch` key list so a manual run actually sets the output the Spark 4.2 job requires.
`ignoreList` registers a ScalaTest `ignore` before execution, so listing `thetasketch.sql` and `tuplesketch.sql` there skipped both files even with `ENABLE_COMET=false`, dropping 364 `SELECT`s from the Spark baseline. The recorded rationale was also wrong. The sketch aggregates run on Spark's `ObjectHashAggregate` and are never converted; what fails is the native scan reading the `t_string_collation` fixture, which contains invalid UTF-8 (`CAST(X'C1' AS STRING)`, `CAST(X'80' AS STRING)`): Parquet error: Arrow: Parquet argument error: Parquet error: encountered non UTF-8 data That is apache#4121, and it has nothing to do with collation: the uncollated query at tuplesketch.sql:825 fails identically. Switch both files to the `--SET spark.comet.enabled = false` header already used for `hll.sql`, which fails on the same fixture for the same reason. Every query then runs and is verified against the golden file in both the baseline and the Comet job, rather than the whole file being skipped.
Drop the `IgnoreCometSuite` mixin from `HiveUDFDynamicLoadSuite`. The justification carried by the 4.1 patch is that `hive-test-udfs.jar` is stripped from the release source tag under the ASF binary-artifact policy, so the suite cannot run. Spark 4.2 added `TestHiveUdfsJar`, which builds that jar at test runtime from sources under `src/test/resources/hive-test-udfs/`, so that reason no longer holds and the 4.2 patch should not carry the exclusion. The replacement rationale cited apache#4098, a closed umbrella for unrelated Spark 4.1 failures which states that Spark SQL tests should not reference it. Also record the two CI registration sites this PR needed but the checklist omitted: the `workflow_dispatch` forced-output list and the `preflight` label allowlist. Both are hard gates that silently skip the whole job.
andygrove
added
enhancement
The spark_4_2 job now runs only on workflow_dispatch, so it fires neither on pull requests nor on merges to main. Enabling it on the PR/push path is a follow-up once the version has settled.
# Conflicts: # spark/src/main/scala/org/apache/comet/serde/operator/CometNativeScan.scala
Four conflicts, all in the CI tiering that landed on main while this branch was out (apache#5926, apache#5939, apache#5963): - dev/ci/compute-changes.py: keep both the spark_4_2 FILTERS entry and main's spark_4_1_hive entry, and give spark_4_2 the POLICY entry it never had. compute() iterates FILTERS and calls event_allows(), which indexes POLICY, so a FILTERS key with no POLICY entry made the changes job raise KeyError: 'spark_4_2' on every event that is not a dispatch. The 4.2 filter list also picks up the inputs spark_4_1 gained since (spark-sql-modules.py, the artifact retry actions, .mvn and mvnw) so the shared-build routing cases cover it. - .github/workflows/ci.yml: expose both new outputs; take main's Detect changes script, whose schedule case replaces the dispatch key list; keep main's docs if: but keep this branch's permissions: contents: write, which the new top-level contents: read default makes load-bearing for the asf-site deploy. The spark_4_2 job's event check moves into POLICY, and the job joins required_checks.needs so it cannot fail without blocking the queue. - dev/ci/check-ci-config.py: declare spark_4_2 in BUILD_JOBS, in a SPARK_EXPERIMENTAL tier set feeding ALL_JOBS, and in POLICY_CASES. - .github/workflows/README.md: take main's tables and add the 4.2 rows. Spark 4.2 keeps its on demand only routing, now expressed as label:run-spark-4.2-tests (the label already exists on the repo) rather than an event check in ci.yml. adding_a_new_spark_version.md described the old dispatch gating, a key list main deleted, and a preflight label allowlist that no longer exists, so its CI section is rewritten to match. check-ci-config.py, actionlint and prettier all pass.
# Conflicts: # dev/diffs/4.2.0.diff
github-actions
Bot
added
area:scan
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #5209.
Stacked on #4950 and should be merged after it. This also relies on the native
collect_setnormalization from #5166.Rationale for this change
Spark 4.2 includes SPARK-57298 coverage for normalizing NaN and signed zero in
collect_set. Comet previously ignored the scalar test because nativecollect_setdid not implement those semantics.What changes are included in this PR?
Regenerates
dev/diffs/4.2.0.diffwithout the SPARK-57298IgnoreCometmarker. The nested complex-type test was already enabled, so both upstream tests now run without adding another diff file.How are these changes tested?
DataFrameAggregateSuitetests: 2 passed, 0 failed, 0 ignored.CometSqlFileTestSuite collect_set_normalizationwith Spark 4.2 against fix: normalize floating-point values in native collect_set #5166: 2 passed for Parquet dictionary encoding on and off.git diff --check.