Dataset tools

Tools for building training datasets:

• Book pipeline — pretraining text from PDF/EPUB books. Each book yields <Book>.md (structured intermediate), <Book>.txt (raw body text), and <Book>.jsonl (one {"text": ...} record per chapter).
• Tabular converters and JSONL utilities — CSV/Parquet to JSONL or chat-format messages, plus JSONL shuffling. See the last section.

Setup

python3 -m venv .venv
.venv/bin/pip install pymupdf   # book pipeline (PDF extraction)
.venv/bin/pip install pyarrow   # parquet converters

All scripts print their supported arguments when run with none.

Book pipeline

book.pdf/.epub --extract_pdf_text.py / extract_epub_text.py--> book.md
book.md  --clean_ocr_text.py----> book.md   (in place)
book.md  --build_pretrain_text.py--> book.txt
book.md  --build_pretrain_jsonl.py-> book.jsonl

Source preference: publisher EPUB > born-digital PDF. Reject PDFs with an OCR text layer (the tell: font sizes smear continuously, e.g. 9.6 / 9.8 / 9.9 / 10.1, instead of clustering at crisp values).

pdf_to_jsonl.sh <book.{pdf,epub}> <out.jsonl> [options] runs all four stages, picking the extractor by file extension; intermediates are kept. Options: --start/--end (body slice, both stages that slice), --pattern (only for tierless markdown, see below), --expect N (fail unless N chapters found).

extract_pdf_text.py (PDF -> md)

Uses text geometry, not content heuristics:

• Body size = modal font size. Headings are spans >= body + 0.7pt.
• Bold fallback: reflowed ebook-style PDFs set everything at one size and carry structure in bold weight. When almost no oversized text exists, fully-bold lines become ## headings instead — there is no chapter tier, so splitting needs --pattern.
• Chapter-tier headings (emitted #) are >= body + 4pt and the first text on their page; chapter-sized display text mid-page is demoted to ##. Chapter records split on the # tier downstream.
• Running heads / page numbers: dropped by top/bottom page bands, plus any single-line body-sized block that is purely digits/roman numerals (page sizes vary within a PDF, so bands alone are not sufficient).
• Superscript note markers (< 0.8x body) are dropped.
• Line-wrap hyphens are healed at join time; paragraphs split across pages are re-joined when the break is mid-sentence and the next part starts lowercase.

extract_epub_text.py (EPUB -> md)

Stdlib-only (no venv needed). Uses the EPUB's declared structure: spine order, <h1> -> # (chapter tier), <h2>..<h6> -> ##, <p> -> one line per paragraph; <sup> footnote markers dropped. Trailing footnote documents in the spine fall after the --end anchor.

clean_ocr_text.py (md -> md, in place)

Generic OCR/extraction repair passes plus exact-string fixes verified against "I Am a Strange Loop" (no-ops on other books). The junk-line pass drops non-heading lines with <= 50% wordlike tokens. It is deliberately aggressive — policy: dropping potential garbage beats keeping any garbage; all-numeric lines, initials, and separators are accepted casualties. Heading lines and lines naming a structural unit (chapter/part/book) are exempt.

build_pretrain_text.py (md -> txt)

Slices --start (first body heading, inclusive) to --end (first back-matter heading, exclusive) and strips heading markers. "" means start/end of file. Heading match is case-insensitive, marker-agnostic; duplicate headings (TOC + body) resolve to the last occurrence.

build_pretrain_jsonl.py (md -> jsonl)

Same --start/--end slice, split into records on chapter-tier # headings. Consecutive # lines (chapter number + title, wrapped titles) open one record, except a bare-number line always starts a new one (so a part page flowing straight into a chapter opener still splits). Markdown without tiers (docling-era conversions: every heading is ##) needs --pattern, a case-insensitive regex matched at line start against marker-stripped lines. Records are {"text": ...} only.

Per-book commands

PDF-extracted:

./pdf_to_jsonl.sh The_Beginning_of_Infinity_-_David_Deutsch.pdf \
  The_Beginning_of_Infinity.jsonl \
  --start Introduction --end Bibliography --expect 19

EPUB:

./pdf_to_jsonl.sh Consciousness_Explained_-_Dennett.epub \
  Consciousness_Explained.jsonl \
  --start PREFACE --end BIBLIOGRAPHY --expect 20

PDF-extracted, bold-mode (uniform font size, so chapters need --pattern; chapter and section headings share the "N—" prefix, hence the full chapter titles):

./pdf_to_jsonl.sh The_Conscious_Mind_-_David_Chalmers.pdf \
  The_Conscious_Mind.jsonl \
  --start 'INTRODUCTION TAKING CONSCIOUSNESS SERIOUSLY' --end NOTES \
  --pattern '(introduction taking consciousness seriously|part [ivx]+— .*|1— two concepts of mind|2— supervenience and explanation|3— can consciousness be reductively explained\?|4— naturalistic dualism|5— the paradox of phenomenal judgment|6— the coherence between consciousness and cognition|7— absent qualia, fading qualia, dancing qualia|8— consciousness and information: some speculation|9— strong artificial intelligence|10— the interpretation of quantum mechanics)$' \
  --expect 15

Docling-era markdown (already cleaned; txt via build_pretrain_text.py with the same --start/--end, jsonl as follows):

python3 build_pretrain_jsonl.py 'I_Am_a_Strange_Loop_-_Douglas_R_Hofstadter.md' \
  I_Am_a_Strange_Loop.jsonl --start PREFACE --end NOTES \
  --pattern '(preface|prologue|epilogue|chapter \d+)\b' --expect 27

python3 build_pretrain_jsonl.py 'The_River_of_Consciousness_-_Oliver_Sacks.md' \
  The_River_of_Consciousness.jsonl --start Foreword --end Bibliography \
  --pattern '(foreword|darwin and the meaning of flowers|speed|sentience:|the other road: freud as neurologist|the fallibility of memory|mishearings|the creative self|a general feeling of disorder|the river of consciousness|scotoma:)$' \
  --expect 12

python3 build_pretrain_jsonl.py 'The_Minds_I_-_Douglas_R_Hofstadter.md' \
  The_Minds_I.jsonl --start Preface --end 'Note From the Author' \
  --pattern "(preface|introduction|jorge luis borges|d\. e\. harding|harold j\. morowitz|a\.m\. turing|douglas r\. hofstadter|stanislaw lem|stanlisaw lem|terrel miedaner|allen wheelis|richard dawkins|arnold zuboff|daniel c\. dennett|david hawley sanford|justin leiber|rudy rucker|christopher cherniak|raymond m\. smullyan|john r\. searle|thomas nagel|robert nozick)\$" \
  --expect 29

The Mind's I is an anthology: records split on author-name headings (some authors recur; stanlisaw is an OCR typo present in the source). The River of Consciousness includes one 4-word part-title record.

Onboarding a new book

1. Run pdf_to_jsonl.sh without --expect; if the start/end headings are unknown, inspect rg -n '^#' <book>.md first.
2. Inspect the record count and sizes; then pin --expect N.
3. Verification invariant: concatenating the jsonl text fields must match <Book>.txt word for word.

Known limitations

• Dehyphenation joins a compound wrapped at its real hyphen ("human-level" -> "humanlevel"). Suspended compounds ("long- and short-styled") are preserved.
• Chapter-tier detection assumes chapters open a fresh page with oversized headings; verify with --expect on each new book.
• clean_ocr_text.py's junk pass can drop legitimate all-numeric content (accepted; see policy above).

Tabular converters and JSONL utilities

Standalone scripts, unrelated to the book pipeline:

• csv_to_jsonl.py <in.csv> <out.jsonl> [--skip-header] — two-column CSV to chat-format JSONL: each row becomes {"messages": [user(col 1), assistant(col 2)]}. Fails loudly on any other column count.
• parquet_to_jsonl.py <in.parquet|dir> [-o out.jsonl] — Parquet rows to JSONL, one object per row, columns as keys; stdout by default. Dates/decimals rendered as strings. Requires pyarrow.
• parquet_to_messages.py <in.parquet|dir> [-o out.json] — first two Parquet columns to chat-format pairs (col 1 = user, col 2 = assistant). Output is a single JSON array, not JSONL. Requires pyarrow.
• shuffle_jsonl.py <in.jsonl> <out.jsonl> [--seed N] — shuffle records; --seed for a reproducible order. Loads the whole file into memory.