Skip to content

--limitBAMsortRAM aborts instead of spilling: evaluate an external sort for coordinate-sorted BAM #206

Description

@BenjaminDEMAILLE

The current behaviour

src/io/bam.rs:173-181: when --limitBAMsortRAM is set and the estimate (~400 bytes/record)
exceeds it, the sort aborts with

limitBAMsortRAM={} bytes exceeded: estimated {} bytes for {} records. Increase --limitBAMsortRAM or use --outSAMtype BAM Unsorted.

Native STAR treats the same flag as a budget, not a ceiling to fail at: it bins records to disk
and merges, so a small RAM limit means more spilling, not a failed run. So on this flag we are
currently divergent in kind, not just in numbers, and the escape hatch we offer ("use Unsorted")
discards the thing the user asked for.

Candidates

Crate Note
extsort On-disk sorting over arbitrarily sized iterators; optional rayon for the in-memory buffer
ext-sort Explicit MemoryLimitedBufferBuilder
in-tree k-way merge Spill sorted runs to tempfile (already a dependency), merge on read

The in-tree option deserves a fair hearing: tempfile is already in Cargo.toml, BySJout already
buffers to a NamedTempFile (src/io/sam.rs), so the spill machinery is not foreign to this
codebase, and a k-way merge over pre-sorted runs is a small, well-understood piece of code.

Constraints

  1. The output must not move. Sorted BAM ordering is (refID, pos) with input read order as the
    tie-break; whatever sorter is used must carry an explicit tie-break key rather than relying on
    an unspecified stability guarantee.
  2. Byte-identical regardless of the RAM limit and the worker count. The limit may change how
    much spills, never what comes out.
  3. Temp files must be cleaned up on abort, and must respect --outTmpDir if that is honoured.

Checklist

  • Decide the semantics: match STAR (spill) rather than abort
  • Measure peak RSS of a sorted-BAM run on a realistic library to size the buffer default
  • Compare a crate against an in-tree k-way merge on tempfile before adding a dependency
  • Byte-identical sorted BAM across RAM limits and thread counts
  • Update the docs for --limitBAMsortRAM once the behaviour changes

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions