Skip to content

Bump Spark 4.3.0 and call Spark EnsureRequirements via reflection in the Spark 4.3 extension - #7810

Closed
pan3793 wants to merge 3 commits into
apache:masterfrom
pan3793:spark-4.3.0-rc3
Closed

pan3793 wants to merge 3 commits into
apache:masterfrom
pan3793:spark-4.3.0-rc3

Conversation

@pan3793

@pan3793 pan3793 commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Why are the changes needed?

Bump Spark 4.3.0 from RC2 to the official 4.3.0 release.

Spark 4.3.0 RC3 passed the vote (vote thread) and became the official 4.3.0 release, and its artifacts are now published to Maven Central and the Apache release mirrors:

  • Drop the spark-staging repository (orgapachespark-1542) override in the spark-4.3 profile — Maven dependencies now resolve from Maven Central.
  • Drop the dist.apache.org RC archive override in pom.xml and CI — the binary archive now resolves from the default https://www.apache.org/dyn/closer.lua/spark/spark-4.3.0 location.

Also applies the change of #7805 to the Spark 4.3 extension: replace the forked KyuubiEnsureRequirements with a rule that calls Spark's private ensureDistributionAndOrdering via reflection, so it always matches the Spark runtime — that class changes frequently in recent Spark versions, even between RCs, and the RC1-based copy already missed the RC2 storage-partitioned join fixes.

How was this patch tested?

  • Pass GHA, including the verify-on-spark-4.3-binary job, which now downloads the Spark 4.3.0 release archive from the Apache mirror.
  • build/mvn -Pfast -Pspark-4.3 -Pscala-2.13 clean package -DskipTests
  • build/mvn clean install -pl extensions/spark/kyuubi-extension-spark-4-3 -am -Pspark-4.3 -Pscala-2.13, all tests pass, including the storage-partitioned join cases in InsertShuffleNodeBeforeJoinSuite (run against the RC3 artifacts, which are byte-identical to the voted 4.3.0 release).
  • Verified the spark-4.3 profile resolves to the GA locations:
    $ build/mvn -Pspark-4.3 help:evaluate -Dexpression=spark.archive.mirror -DforceStdout -q
    https://www.apache.org/dyn/closer.lua/spark/spark-4.3.0
    $ build/mvn -Pspark-4.3 help:evaluate -Dexpression=spark.archive.query -DforceStdout -q
    ?action=download
    

Was this patch assisted by generative AI tooling?

Assisted-by: Claude Opus 5.5
Assisted-by: ZCode with GLM-5.3-Flash

Assisted-by: Claude Opus 5.5
@github-actions github-actions Bot added kind:infra license, community building, project builds, asf infra related, etc. kind:build labels Oct 28, 2026
Apply the change of apache#7805 to the Spark 4.3 extension: replace the forked
KyuubiEnsureRequirements with a rule that calls Spark's private
ensureDistributionAndOrdering via reflection, so it always matches the
Spark runtime.

Assisted-by: Claude Opus 5.5
@pan3793 pan3793 changed the title Bump Spark 4.3.0 RC3 Bump Spark 4.3.0 RC3 and call Spark EnsureRequirements via reflection in the Spark 4.3 extension Oct 28, 2026
RC3 passed the vote and became the official 4.3.0 release; drop the RC
binary archive and Maven staging repository overrides so artifacts
resolve from the default Apache archive location and Maven Central.
@pan3793 pan3793 changed the title Bump Spark 4.3.0 RC3 and call Spark EnsureRequirements via reflection in the Spark 4.3 extension Bump Spark 4.3.0 and call Spark EnsureRequirements via reflection in the Spark 4.3 extension Nov 1, 2026
@pan3793 pan3793 self-assigned this Nov 1, 2026
@pan3793 pan3793 added this to the v1.13.0 milestone Nov 1, 2026
@pan3793 pan3793 closed this in 9d67ce6 Nov 2, 2026
@pan3793

pan3793 commented Nov 2, 2026

Copy link
Copy Markdown
Member Author

Merge Summary:

Posted by merge_kyuubi_pr.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind:build kind:infra license, community building, project builds, asf infra related, etc. module:extensions module:spark

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants