The cgroup diagram in memory_management, added in #6014, groups everything the container counts by who accounts for it. It does not say which configuration value sizes each region.
That leaves the two facts the section goes on to state as prose only:
spark.memory.offHeap.size is part of the pod limit, not extra headroom on top of it.spark.executor.memoryOverhead is the only slack in the container.
Both are properties of how the configuration maps onto the regions, which is exactly the kind of
thing a diagram carries better than a sentence. The mapping is also uneven in a way that is easy to
get wrong: spark.memory.offHeap.size alone sizes three of the regions, including Comet's native
reservations, which are neither off-heap in Spark's sense nor allocated by the JVM, while
spark.executor.memoryOverhead sizes none of them.
Adding the three configuration values as inputs to the existing diagram would make that visible
without introducing a third diagram to the page.
The cgroup diagram in memory_management, added in #6014, groups everything the container counts by who accounts for it. It does not say which configuration value sizes each region.
That leaves the two facts the section goes on to state as prose only:
spark.memory.offHeap.sizeis part of the pod limit, not extra headroom on top of it.spark.executor.memoryOverheadis the only slack in the container.Both are properties of how the configuration maps onto the regions, which is exactly the kind of
thing a diagram carries better than a sentence. The mapping is also uneven in a way that is easy to
get wrong:
spark.memory.offHeap.sizealone sizes three of the regions, including Comet's nativereservations, which are neither off-heap in Spark's sense nor allocated by the JVM, while
spark.executor.memoryOverheadsizes none of them.Adding the three configuration values as inputs to the existing diagram would make that visible
without introducing a third diagram to the page.