The current behaviour
src/io/bam.rs:173-181: when --limitBAMsortRAM is set and the estimate (~400 bytes/record)
exceeds it, the sort aborts with
limitBAMsortRAM={} bytes exceeded: estimated {} bytes for {} records. Increase --limitBAMsortRAM or use --outSAMtype BAM Unsorted.
Native STAR treats the same flag as a budget, not a ceiling to fail at: it bins records to disk
and merges, so a small RAM limit means more spilling, not a failed run. So on this flag we are
currently divergent in kind, not just in numbers, and the escape hatch we offer ("use Unsorted")
discards the thing the user asked for.
Candidates
| Crate |
Note |
extsort |
On-disk sorting over arbitrarily sized iterators; optional rayon for the in-memory buffer |
ext-sort |
Explicit MemoryLimitedBufferBuilder |
| in-tree k-way merge |
Spill sorted runs to tempfile (already a dependency), merge on read |
The in-tree option deserves a fair hearing: tempfile is already in Cargo.toml, BySJout already
buffers to a NamedTempFile (src/io/sam.rs), so the spill machinery is not foreign to this
codebase, and a k-way merge over pre-sorted runs is a small, well-understood piece of code.
Constraints
- The output must not move. Sorted BAM ordering is
(refID, pos) with input read order as the
tie-break; whatever sorter is used must carry an explicit tie-break key rather than relying on
an unspecified stability guarantee.
- Byte-identical regardless of the RAM limit and the worker count. The limit may change how
much spills, never what comes out.
- Temp files must be cleaned up on abort, and must respect
--outTmpDir if that is honoured.
Checklist
The current behaviour
src/io/bam.rs:173-181: when--limitBAMsortRAMis set and the estimate (~400 bytes/record)exceeds it, the sort aborts with
Native STAR treats the same flag as a budget, not a ceiling to fail at: it bins records to disk
and merges, so a small RAM limit means more spilling, not a failed run. So on this flag we are
currently divergent in kind, not just in numbers, and the escape hatch we offer ("use Unsorted")
discards the thing the user asked for.
Candidates
extsortext-sortMemoryLimitedBufferBuildertempfile(already a dependency), merge on readThe in-tree option deserves a fair hearing:
tempfileis already inCargo.toml, BySJout alreadybuffers to a
NamedTempFile(src/io/sam.rs), so the spill machinery is not foreign to thiscodebase, and a k-way merge over pre-sorted runs is a small, well-understood piece of code.
Constraints
(refID, pos)with input read order as thetie-break; whatever sorter is used must carry an explicit tie-break key rather than relying on
an unspecified stability guarantee.
much spills, never what comes out.
--outTmpDirif that is honoured.Checklist
tempfilebefore adding a dependency--limitBAMsortRAMonce the behaviour changes