Skip to content

docs: explain that Comet's memory comes out of the executor container - #6033

Closed
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:comet-memory-configuration-guide-eaa36a30
Closed

andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:comet-memory-configuration-guide-eaa36a30

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

Closes #6032.

Rationale for this change

#6014 and #6029 established, in the contributor guide, where Comet's memory actually lives and which configuration value sizes each part of the executor container. Two of those facts are the ones users get wrong, and neither appeared anywhere in the tuning guide: spark.memory.offHeap.size is part of the container budget rather than headroom on top of it, and spark.executor.memoryOverhead is the container's only slack while sizing no budget any consumer can draw on.

The tuning guide opened the memory section with "specify how much memory Comet can use in addition to memory already allocated to Spark", which reads as though Comet's budget sits outside what the container already accounts for. That is exactly the misconception the contributor-guide diagram was changed to prevent, and the audience that needs it most does not read the contributor guide.

The practical consequence was that the guide gave no way to choose between the three settings that respond to an executor OOM kill, and they do not behave alike. Raising spark.executor.memoryOverhead widens the container and leaves Comet's budget alone. Lowering spark.comet.exec.memoryPool.fraction leaves the container alone and shrinks what Comet may reserve. Raising spark.memory.offHeap.size does both, which makes it the intuitive response and the least reliable one, since it also raises the ceiling on the unaccounted overshoot that caused the kill. The guide also never said what an OOM looks like from each of the three budgets, so there was nothing to tell a user whether the setting they were about to change was the one that overflowed.

What changes are included in this PR?

All in docs/source/user-guide/latest/tuning.md:

  • The Memory Tuning intro is rewritten to say Comet re-divides the executor's memory rather than adding to it.
  • A new How Comet's Memory Fits in the Executor Container section: Spark's container sizing sum, the two facts stated against it, and a table comparing how the three settings move the container and Comet's budget.
  • A new Diagnosing Out-of-Memory Failures section: a table distinguishing a container kill (exit 137 / OOMKilled), JVM heap exhaustion (exit 52), and a task-level SparkOutOfMemoryError, with what to change in each case and which settings show what the pool was doing.
  • A note that fair_unified is currently stricter than the paragraph describing it, pointing at fair_unified memory pool caps the pool total at pool_size / num_consumers instead of each consumer's share #5961. The existing description is left in place rather than rewritten, because fair_unified memory pool caps the pool total at pool_size / num_consumers instead of each consumer's share #5961 is a regression: when it is fixed the note is simply deleted.
  • A link from the user guide to the contributor guide's Memory Management page, which so far only linked one way.

The container sizing sum is Spark's own, from ResourceProfile.getResourcesForClusterManager, and matches the one already stated in the contributor guide.

One thing this PR deliberately does not touch: the TPC-H figures in Determining How Much Memory to Allocate ("Comet requires at least 5 GB of RAM") do not say which setting the number refers to, and I did not want to guess at a reading of numbers I cannot re-derive. Worth a separate look.

How are these changes tested?

Docs only, no code paths touched. No mermaid fences are added, so the #6021 pre-render step is unaffected.

npx prettier "docs/source/user-guide/latest/tuning.md" --check passes, which is what the Preflight job runs.

The two new in-page anchors follow the slug form MyST generates with myst_heading_anchors = 4, matching the existing #where-comets-budget-comes-from link in memory_management.md.

The tuning guide opened the memory section with "specify how much memory Comet can
use in addition to memory already allocated to Spark", which reads as though Comet's
budget sits outside what the container already accounts for. That is the misconception
apache#6014 and apache#6029 established the contributor-guide diagram to prevent, and users are the
audience that needs it.

Adds the container sizing sum, states the two facts against it (offHeap.size is inside
the limit; memoryOverhead is the only slack and sizes no budget), and compares how the
three settings that respond to an OOM kill move the container and Comet's budget in
different directions. Adds a table of what each of the three budgets looks like when it
is exceeded, notes that fair_unified is currently stricter than documented (apache#5961), and
links the contributor guide's Memory Management page.
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Sep 18, 2026
@andygrove andygrove closed this Sep 19, 2026
@andygrove
andygrove deleted the comet-memory-configuration-guide-eaa36a30 branch September 19, 2026 18:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tuning guide does not explain that Comet's memory comes out of the executor container

1 participant