Split a large delimited file into row chunks with a repeated header

[ddeboer]

Part of #782 (port GeoNames RDF to LDE), which lists chunking as unhomed: #511 deliberately left it to the caller, so SparqlAnythingConverter consumes pre-split chunks that nothing in LDE produces.

Why chunking exists

It is not a convenience – it is what bounds the conversion’s memory. SPARQL Anything materialises a chunk’s whole result graph before writing it, and each chunk runs in its own JVM that frees it on exit. A single process over the full GeoNames dataset needs >14 GB and OOMs; at 1M rows per chunk a result graph is ~1.2 GB, which is why map.sh pairs CHUNK_SIZE with -Xmx2g per worker. Chunk size and heap are sized together, so whatever we build has to keep that pairing visible.

What geonames-rdf does today

download.sh’s chunk_with_header:

1. rm -rf $prefix* – drop a previous run’s chunks, so stale data cannot reach the output.
2. split -l $CHUNK_SIZE "$input" "$prefix".
3. For each part, write a header row in front of it (cat "$header" > "$f.csv"; cat "$f" >> "$f.csv"; rm "$f").
4. Fail loudly if split produced nothing, rather than letting an unmatched glob become the “filename”.

It is called twice: on the main table (~13.5M rows, after the awk pass that synthesises the adm1/adm2 foreign keys) and on the scoped alternate-names table.

What the LDE version needs

• Stream. allCountries.txt is ~1.9 GB and alternateNamesV2 is comparable; neither may be read into memory.
• A caller-supplied header. The GeoNames dumps carry no header row – it comes from config/headers-*.csv, and SPARQL Anything’s queries depend on those column names. So the header is a line the caller provides, not one lifted off the input. Taking the first line of the input instead is a different mode; if we support both, they must be explicit rather than inferred.
• Return the chunk paths. map.sh globs for them afterwards and needs a guard for the unmatched-glob case; returning an ordered array removes that class of bug outright. Order matters, because convert() concatenates in the order it is given.
• A fresh output directory per run, cleaned up by the caller – the same stale-output hazard as item 8 in Port GeoNames RDF to LDE #782, which we fixed in the converter by chunking into a per-run directory.
• Fail on an empty input, rather than emitting one header-only chunk. A run that produced no rows has failed upstream, and a header-only chunk hides that – the same reasoning as the empty-chunkPaths guard already in convert().
• Chunk on physical lines. That is only correct when no field contains a newline; the GeoNames dumps satisfy this, and the shell version already assumes it. Worth stating in the API docs rather than silently assuming, since it does not hold for CSV in general.

Where it lives – open question

Splitting a large delimited file is not SPARQL Anything-specific: an importer or a downloader could want it just as much. But a package for one function is heavy.

My inclination is to start it inside @lde/sparql-anything next to the converter that consumes its output, and to extract it only when a second consumer appears – the alternative is committing to a package boundary before we know whether anything else wants it. Worth a second opinion before implementation.

Not in scope

The GeoNames-specific preparation stays in geonames-rdf: the awk synthesis of the adm1/adm2 foreign keys, and dropping alternate names belonging to out-of-scope features. Those are domain policy, not plumbing.