Dependency discussion: HDF5, for CellRanger's raw/filtered_feature_bc_matrix.h5

[BenjaminDEMAILLE]

CONTRIBUTING.md asks for a new dependency, especially a non-Rust one, to be raised before the PR. This is that issue. The decision is yours; I am laying out the options rather than arguing for one, though I do say at the end which I would pick and why.

What it is for

cellranger count writes its matrices twice: once as MEX (raw_feature_bc_matrix/, three files) and once as HDF5 (raw_feature_bc_matrix.h5). #XXX gets the MEX side right, down to the directory names and the -1 barcode suffix, and against a real CellRanger 10.0.0 run the raw barcode sets now match exactly, 200 of 200. The .h5 files are the remaining piece.

They matter because a good deal of tooling reads only the .h5: scanpy.read_10x_h5, Seurat's Read10X_h5, and anything built on cellranger's own loaders. The file is small and its layout is documented by 10x:

/matrix/barcodes        [n_cells]   string
/matrix/data            [nnz]       int32
/matrix/indices         [nnz]       int64
/matrix/indptr          [n_cells+1] int64
/matrix/shape           [2]         int32
/matrix/features/{id, name, feature_type, genome, _all_tag_keys}

CSC, chunked, gzip-filtered. That is the whole file.

Option A — hdf5-metno, dynamically linked

Crate	hdf5-metno 0.14.0 over hdf5-metno-sys 0.12.1
Upstream	https://github.com/metno/hdf5-rust (maintained fork of the dormant hdf5 crate)
Licence	MIT OR Apache-2.0 on both
Build deps pulled in	pkg-config, regex, serde, serde_derive, libloading, winreg
Runtime deps pulled in	libc, parking_lot
System requirement	libhdf5 must already be installed

The build probes for a system libhdf5 with pkg-config (and the registry on Windows). That is the part I would not want to sign off casually: rustar-aligner is published to crates.io, so cargo install rustar-aligner would start failing on any machine without libhdf5 development headers. It also has to be installed on all five CI runners, including Windows MSVC, where HDF5 packaging is the least pleasant.

serde and regex as build dependencies of a -sys crate are also worth a look before agreeing to it.

Option B — hdf5-metno with features = ["static"]

Pulls in hdf5-metno-src 0.10.2, which vendors the HDF5 C source and builds it with cmake as a build dependency. Licence on that crate reads as non-standard on crates.io — it is HDF5's own BSD-style licence, which is permissive, but it is a third licence text to carry in LICENSES/.

No system library, so cargo install keeps working. The cost is cmake on the build path for every user and every CI runner, and a multi-minute C build on a cold cache for a feature that writes one small file.

Option C — write the file in-tree, no dependency

The 10x .h5 uses a small, fixed subset of HDF5: one group, nine datasets, one filter. Writing that subset is not out of reach, and it is the same reasoning that put the Smith-Waterman engine in-tree in #148 rather than binding a C library.

I want to be honest about the risk, because it cuts the other way here. HDF5 is a container format, not a serialisation: a conformant file needs a superblock, object headers with correctly laid-out messages, a B-tree and local heap for the group's links, and a chunk index for each chunked dataset. "It opens in h5py on my machine" is a much weaker guarantee than a passing test. I would want the acceptance criterion to be a round-trip against real readers (h5py, scanpy.read_10x_h5) in the differential harness, not a unit test asserting our own bytes.

Ballpark: 1000-1500 lines, most of it format plumbing, plus that external round-trip. Materially more work than Options A or B, and the failure mode is subtler.

Option D — do not write .h5 at all

The MEX output from #XXX is complete and loads in scanpy and Seurat. Anyone needing the .h5 can convert:

import scanpy as sc
sc.read_10x_mtx("outs/raw_feature_bc_matrix").write("raw_feature_bc_matrix.h5ad")

That is .h5ad, not 10x .h5 — near enough for scanpy users, not for a tool that specifically wants CellRanger's file. Note also that #1 asks for AnnData output, which would land in the same place and has the same dependency question underneath it.

My reading

I would not take Option A: a crates.io-published binary that fails to install without a system C library is a real cost paid by every user, in exchange for one optional output file.

Between B, C and D, I would ask what the .h5 is actually for. If it is "so scanpy users can load our output", D already works today and B is not worth cmake. If it is "so tools that read only CellRanger's .h5 work unchanged", then it has to be a real 10x .h5, and the question is whether the maintenance you prefer is a cmake build dependency (B) or ~1200 lines of format code that only we maintain (C).

If you want it, I would go with B, static only, so cargo install keeps working, and gated behind a Cargo feature that is off by default so the ordinary build is untouched. I will happily do C instead if you would rather keep the dependency tree pure — say which and I will build it. What I would not do is start either one before you have picked.

[Psy-Fer]

Absolutely not. Not adding build dependancies, especially this early in the project, and my previous experience with the FAST5 format (HDF5) (literally invented a new format, slow5, to kill it) has me never wanting to deal with it if I don't absolutely need to.

[BenjaminDEMAILLE]

CellBender is the right kind of example to test this against, so I went and read its loader rather than reasoning about it. It does not force HDF5: cellbender remove-background --input accepts a CellRanger MTX directory as well as a .h5, and #178 already produces exactly the directory it wants.

From cellbender/remove_background/data/io.py:

def choose_data_loader(input_file: str) -> Callable:
    file_ext = os.path.splitext(input_file)[1]
    if os.path.isdir(input_file):
        return get_matrix_from_cellranger_mtx
    elif file_ext == '.h5':
        return get_matrix_from_cellranger_h5

and the version detection inside that loader:

if os.path.isfile(os.path.join(filedir, 'features.tsv.gz')):
    return 3
else:
    return 2

with the v3 branch then reading

matrix_file  = os.path.join(filedir, 'matrix.mtx.gz')
gene_file    = os.path.join(filedir, 'features.tsv.gz')
barcode_file = os.path.join(filedir, 'barcodes.tsv.gz')

A bare 10x run on #178 writes:

outs/raw_feature_bc_matrix/
  matrix.mtx.gz
  features.tsv.gz
  barcodes.tsv.gz

features.tsv.gz is present, so CellBender takes its v3 path, and all three filenames are the ones it opens. It also does no barcode-suffix processing (np.genfromtxt, barcodes kept verbatim), so the -1 suffix from #178 passes through as it does from CellRanger. And CellBender wants the raw, unfiltered matrix, which is what raw_feature_bc_matrix/ is.

So for this workflow the answer is: point --input at the directory, and it works today. Nothing here requires us to write .h5.

Where the instinct is right anyway

The general point behind the question stands, and I do not want the CellBender result to make it look settled. .h5 is the default currency in the 10x ecosystem: CellBender writes its output as .h5, scanpy.read_10x_h5 and Seurat's Read10X_h5 are what most tutorials use, and any tool that only ever saw CellRanger output will reach for the single file rather than the directory. Every such tool we check individually may turn out to accept MTX; the cost of .h5 is that users should not have to check.

What that changes for this issue: it moves .h5 from "required for workflow X" to "removes friction across the ecosystem". That is a real benefit, but it is a weaker case than a hard blocker would have been, and it should be weighed against the costs in the four options above rather than treated as forced. If someone finds a tool that genuinely accepts only .h5 and matters to this project, that would change the balance and is worth adding to this issue.

My recommendation is unchanged: not Option A (a crates.io binary that will not install without a system libhdf5), and if we do it, Option B behind a default-off Cargo feature, or Option C if you would rather keep the tree pure. I am happy to build either once you pick.

[Psy-Fer]

or option "i used my brain and not an LLM", a tool to convert from one format to another as an extra step for tools that require hdf5 files, avoiding the issue completely.

No hdf5 please.

James

[BenjaminDEMAILLE]

Understood. Your call, and I'll drop it.

One factual point, not an argument to reopen. #180 adds no dependency: no libhdf5, no cmake, no -sys crate, nothing new in Cargo.lock. It is in-tree Rust writing the bytes directly, the same shape as the in-tree Smith-Waterman engine in #148. Your first comment gave build dependencies as the reason, so I want to be sure "No hdf5" means the format is out of scope for this codebase, rather than the dependency objection landing on a PR that doesn't carry one. Both are legitimate positions and I'll take either; they just have different follow-ups.

Say the word and I'll close #180. Nothing else in the stack depends on it — #173 through #179 stand on their own.

[Psy-Fer]

An in tree version just adds unnecessary maintenance surface to rustar. There should be a well defined and requested reason for it with no relatively simple solution outside of the codebase.

This doesn't satisfy any of those requirements.

Same reason I chose to build an updated opal in hyalite. Because I can ensure correctness and determinism with simd specifically withing that context. Rather than mixing it with rustar. Same reason STAR imports it as a library. Same reason hdf5 shouldn't be in tree.

So unless you do have a compelling reason, this should be closed.

[BenjaminDEMAILLE]

That's a clearer statement of the rule than the dependency one, and I think you're right: maintenance surface inside rustar is the cost, and hyalite is the shape that avoids it. Closing #180.

If it turns out to be wanted later, I'd rather offer the hyalite-shaped version than argue for the in-tree one: a standalone pure-Rust crate that writes the CellRanger matrix subset of HDF5, no C, no cmake, no system library, which I'd hand over to scverse rather than keep. Same reasoning you gave for hyalite, and it has one property in-tree code cannot: the crate's own CI can validate against libhdf5 — h5repack round-trip plus h5diff — without rustar ever depending on HDF5 tooling. The code already passes that; that is how the two bugs in it were found.

But I want to be straight about the other half of your point, because the shape being right doesn't make the reason compelling. I already conceded upthread that nothing is actually blocked: CellBender takes an MTX directory, scanpy has read_10x_mtx, Seurat has Read10X. The case for .h5 is convenience, not necessity, and "a tool wanted it and there was no simple alternative" is a bar it does not currently clear. So the offer stands if a real request turns up, and I am not going to keep it open on the strength of the one I have.

[BenjaminDEMAILLE]

Closing as not planned. The decision is James's and it is recorded above: HDF5 stays out of rustar, in tree or as a dependency, because the need is convenience rather than necessity — CellBender takes an MTX directory, scanpy has read_10x_mtx, Seurat has Read10X, and #178 writes exactly that directory.

If a tool turns up that genuinely accepts nothing else, the offer of a standalone pure-Rust crate, hyalite-shaped and transferable to scverse, stands. That would be a new issue with a real request behind it, not a reopening of this one.