feat(sparql-anything): split a file into the chunks a conversion can hold

[ddeboer]

convert() takes chunks it does not create, so every consumer was left to produce them – the shell script this package replaces does it with split(1) and a cat loop per chunk. chunk() streams a line-oriented file into fixed-row pieces and returns their paths in order, which is exactly what a job's chunks takes.

const chunks = await chunk('data/allCountries.txt', {
  rows: 1_000_000,
  into: 'data/chunks',
  header: 'geonameid\tname\tlatitude\tlongitude',
  extension: '.csv',
});
await converter.convert([{ queryFile: 'places.rq', chunks, load }], 'places.nt');

Why here, and not a package of its own

I started out proposing @lde/line-chunker, on the grounds that chunking is how an input is made to fit a bounded unit of work (ADR 12). That was wrong: in LDE, bounded memory comes from streaming, and the pipeline has no use for a file in pieces. Chunking exists because SPARQL Anything materialises a chunk's whole result graph before writing it – a property of this tool, not of the data or the pipeline. So it belongs in the package whose job is working around this tool, beside the one-JVM-per-chunk decision it is the other half of.

Nothing in it touches the converter, so if a second, streaming-averse tool ever appears, promoting it is a file move.

Why by line, and not by parsing

The tempting alternative was csv-parse, so that a quoted field containing a newline could not be cut in two. Two things against it:

• The input cannot produce that case. GeoNames is "tab-delimited text in utf8 encoding" with no quoting mechanism documented – without quoting there is no syntax for a newline inside a field. And the current pipeline does no parsing at chunk time either; split -l is pure line splitting, and the parsing happens downstream in SPARQL Anything. A parser here would be a new guarantee against an input this cannot receive.
• It costs about 6×. On a 220 MB / 2 M-row TSV: line splitting 1.2 s, csv-parse with raw 7.5 s.

So the contract says what it does – splits on lines, every record must be one line – rather than parsing every field to defend against something that cannot arrive. Widening header?: string to also mean "the input's first line" is a later change that breaks no caller, if a headered-CSV consumer ever turns up.

Measurements

Same 220 MB file, two runs each, chunks verified identical:

	time
cat copy (raw I/O floor, macOS)	0.18 s
GNU split -l (Linux, Docker)	0.30 s
this implementation	1.18 s
BSD split -l (macOS)	3.60 s
csv-parse with raw	7.50 s

GNU split is faster, and shelling out to it was the other candidate. It would have meant a POSIX dependency in the runner's environment, a dev machine 12× slower than CI (BSD split is pathological – 3.3 s of system time to move bytes a copy handles in 0.07 s), and a package that is five lines of shell behind an interface. Against a chunking step that is seconds inside a two-minute download.sh, that trade is not worth it.

extension

Defaults to the input's own, so chunking N-Triples for something that expects N-Triples needs nothing. It is there for tools that read the format from the file name – SPARQL Anything does, which is why the shell script renames its .txt chunks to .csv.