Skip to content

[MINOR][SQL][DOCS] Correct SPJ configuration prerequisites and example - #58904

Open
szehon-ho wants to merge 1 commit into
apache:masterfrom
szehon-ho:codex/spj-tuning-docs
Open

szehon-ho wants to merge 1 commit into
apache:masterfrom
szehon-ho:codex/spj-tuning-docs

Conversation

@szehon-ho

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Correct the Storage Partition Join guidance in the SQL performance tuning guide:

  • Remove pushPartValues.enabled=true as a prerequisite for allowKeysSubsetOfPartitionKeys.enabled.
  • Document that allowCompatibleTransforms.enabled requires partiallyClusteredDistribution.enabled=false.
  • Keep partial clustering at its default in the basic Iceberg example and describe it separately as optional skew tuning, including the extra reads that replication can cause.

Why are the changes needed?

The current table asks users to enable an unnecessary prerequisite for subset-key joins and omits an incompatible configuration combination. The basic example also enables partial clustering without explaining its replication cost.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Checked the documented prerequisites against EnsureRequirements.checkKeyGroupCompatible and KeyedShuffleSpec.canReduceKeys, and reviewed the existing SPARK-48065 coverage in KeyGroupedPartitioningSuite. git diff --check passes.

Documentation-only change; Spark test suites and the full documentation build were not run.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex 0.143.0-alpha.32

@szehon-ho

Copy link
Copy Markdown
Member Author

@ulysses-you @peter-toth hi i noticed these other discrepancy while reviewing the SPJ doc, can you help take a look? Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants