What is the problem the feature request solves?
Since #5916 a map task spills every partition into one temp file. DiskManager::create_tmp_file picks a random local dir per file, and Comet hands it Spark's local dirs, so the per-partition files it replaced spread a task's spill across the local dirs while the single file pins it to one. With several local dirs on separate disks the per-disk peak becomes one task's whole spill, and a single file can grow to the task's total spilled bytes.
Rotation does not release space earlier: every file holds ranges from most partitions, so nothing can be unlinked before the merge finishes. What it can change is the spread across local dirs and the size of any one file.
Describe the potential solution
Compare the task-wide file against two alternatives that keep each partition's write order:
- One file per spill round, as DataFusion's own operators spill. The file count then equals the round count, which a pool under pressure can push into the thousands.
- A new file once the current one passes a byte threshold, rotating only between writes so no range straddles files. Ranges keep a task-wide offset space and stay 16 bytes; a small file table maps an offset to (file, local offset). The file count is bounded by spilled bytes / threshold whatever the round count.
Either way the merge should keep its open readers bounded (opened lazily, capped by the file count or an LRU) and read each partition's ranges in offset order, which preserves write order.
Measure on SSD/NVMe with many spill rounds and concurrent tasks, with two or more local dirs: spill write time and merge time, peak memory, open file counts, and per-disk peak space. write_time currently covers both the spill writes and the merge copies; splitting it into spill and merge time is a prerequisite for that report.
Additional context
Suggested in the review of #5916: #5916 (comment). #6009 covers the range metadata the merge keeps in memory.
What is the problem the feature request solves?
Since #5916 a map task spills every partition into one temp file.
DiskManager::create_tmp_filepicks a random local dir per file, and Comet hands it Spark's local dirs, so the per-partition files it replaced spread a task's spill across the local dirs while the single file pins it to one. With several local dirs on separate disks the per-disk peak becomes one task's whole spill, and a single file can grow to the task's total spilled bytes.Rotation does not release space earlier: every file holds ranges from most partitions, so nothing can be unlinked before the merge finishes. What it can change is the spread across local dirs and the size of any one file.
Describe the potential solution
Compare the task-wide file against two alternatives that keep each partition's write order:
Either way the merge should keep its open readers bounded (opened lazily, capped by the file count or an LRU) and read each partition's ranges in offset order, which preserves write order.
Measure on SSD/NVMe with many spill rounds and concurrent tasks, with two or more local dirs: spill write time and merge time, peak memory, open file counts, and per-disk peak space.
write_timecurrently covers both the spill writes and the merge copies; splitting it into spill and merge time is a prerequisite for that report.Additional context
Suggested in the review of #5916: #5916 (comment). #6009 covers the range metadata the merge keeps in memory.