Skip to content

Add parallel gzip decompression and shared scheduling #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jph00 merged 1 commit into from
Aug 24, 2026
Merged

Add parallel gzip decompression and shared scheduling #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ version = "0.1.5"
edition = "2024"
rust-version = "1.91"
license = "Apache-2.0"
description = "Fast parallel and indexed bzip2 decompression for Rust and Python"
description = "Compression-format research workbench with fast bzip2 and gzip decompression"
repository = "https://github.com/AnswerDotAI/fastbz2"
homepage = "https://github.com/AnswerDotAI/fastbz2"
documentation = "https://github.com/AnswerDotAI/fastbz2"
Expand All @@ -25,6 +25,7 @@ codegen-units = 1
[dependencies]
blake3 = "1.8.7"
clap = { version = "4.6.6", features = ["derive"] }
crc32fast = "1.5.1"
memmap2 = "0.9.11"
pyo3 = { version = ">=0.29.2", optional = true }
rayon = "1.12.0"
Expand All @@ -33,6 +34,8 @@ tempfile = "3.27.0"

[dev-dependencies]
crabz2 = { version = "0.4.0", features = ["parallel"] }
flate2 = { version = "1.1", default-features = false, features = ["rust_backend"] }
libc = "0.2.175"
libbz2-rs-sys = { version = "0.2.5", default-features = false, features = ["std"] }

[features]
Expand Down
47 changes: 34 additions & 13 deletions DEV.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ src/crc.rs bzip2 block and combined-stream CRC primitives
src/decode.rs serial/parallel decode scheduling and index construction
src/decoder.rs bzip2 block machinery and 12-bit Huffman fast tables
src/format.rs cheap structural scan for header and marker candidates
src/gzip.rs gzip framing, LSB-first DEFLATE, CRC32, and block reports
src/pipeline.rs shared ordered, byte-budgeted, staged worker scheduler
src/index.rs stable persistent index format
src/indexed.rs seekable decoded view and block cache
src/lib.rs public Rust API and private PyO3 binding
Expand All @@ -22,15 +24,17 @@ tests/ Rust CLI/corpus and Python API integration tests
tools/stage_binaries.py copy the release executable into Maturin wheel data
```

The current scanner deliberately does not treat 48-bit marker matches or later `BZh` headers as validated structure. Full decoding must establish the exact block chain and validate every block CRC plus the combined stream CRC before marker candidates can become trusted index entries. Python integration tests use standard-library `bz2`/libbz2 as an independent fixture generator.
The current bzip2 scanner deliberately does not treat 48-bit marker matches or later `BZh` headers as validated structure. Full decoding must establish the exact block chain and validate every block CRC plus the combined stream CRC before marker candidates can become trusted index entries. Python integration tests use standard-library `bz2`/libbz2 as an independent fixture generator.

The gzip decoder is an in-repo RFC 1952/RFC 1951 implementation rather than a wrapper around a production codec. It parses optional headers and concatenated members, decodes stored/fixed/dynamic blocks, maintains the 32 KiB LZ77 history, and validates FHCRC, CRC32, and ISIZE. For large inputs, independently discovered dynamic-block boundaries seed unknown history with compact markers. A marker-free history switches the same decoder to byte output; otherwise the coordinator resolves only the suffix needed by the successor and queues full resolution plus CRC on the shared staged scheduler. Reports retain member boundaries, DEFLATE block ranges, and accepted/fallback chunk counts. `crc32fast` is the sole production helper; `flate2` is dev-only.

The decoder remains independent of files, threads, Python, and the CLI. Parallel scanning/decoding and indexed seeking are layered over it. Native workers never call Python. Large offsets use explicit 64-bit bit/byte types, and speculative block-marker hits are accepted only when they form an exact stream chain with valid block and combined stream CRCs.

The core decode APIs can report completed compressed and decoded byte counts without knowing anything about terminals. The CLI layers delayed, rate-limited TTY progress rendering over those callbacks; redirected stderr and `--quiet` produce no progress output. Decoded files use same-directory temporary files and atomic persistence, then inherit the compressed input's modification time and permissions. `--rm` removes an input only after decode, persistence, and metadata copying all succeed. Output-size limits are enforced by a writer wrapper, so the decoder has one code path for files, stdout, validation, and indexing.
Both core decode APIs report completed compressed and decoded byte counts without knowing anything about terminals. The CLI selects bzip2 or gzip by a recognised extension and falls back to magic for stdin or unknown names. It layers delayed, rate-limited TTY progress rendering over the shared callbacks; redirected stderr and `--quiet` produce no progress output. Decoded files use same-directory temporary files and atomic persistence, then inherit the compressed input's modification time and permissions. `--rm` removes an input only after decode, persistence, and metadata copying all succeed. Output-size limits are enforced by a writer wrapper, so each decoder has one code path for files, stdout, validation, and listing.

Parallel decoding uses a rolling candidate queue rather than stopping at stream boundaries or waiting for fixed batches. Workers reserve the maximum possible decoded block size before starting; once a block finishes, that conservative reservation shrinks to its actual output size and is released when ordered validation consumes or rejects it. Thus the `memory_limit` bounds speculative decoded output while short multistream inputs can keep the worker pool busy. The 1 GiB default admits one worst-case block per worker on the primary 18-core machine.
The shared `pipeline.rs` scheduler provides ordered results, byte-budgeted admission, cancellation, and a staged priority queue. Bzip2 uses the rolling candidate path: workers reserve the maximum possible decoded block size, then shrink that reservation to actual retained output until ordered validation consumes or rejects it. Gzip uses the staged path: native workers alternate speculative DEFLATE decoding with higher-priority marker resolution, while the coordinator advances only the 32 KiB dependency windows and emits resolved chunks in order. Decode results and outstanding resolution results have separate bounded horizons, preventing either dependency stalls or unbounded memory.

The production decoder is safe scalar Rust designed for LLVM auto-vectorisation. Huffman decoding uses a 4096-entry direct table for codes up to 12 bits and canonical fallback for longer codes. Add narrowly scoped unsafe or architecture-specific SIMD only after profiling; `libbz2-rs-sys` remains the dev-only differential oracle.
The bzip2 decoder is safe scalar Rust designed for LLVM auto-vectorisation. Huffman decoding uses a 4096-entry direct table for codes up to 12 bits and canonical fallback for longer codes. Gzip uses full canonical lookup tables packed into `u16`, a branch-free 64 KiB marker-resolution lookup for large chunks, and `crc32fast::Hasher::combine` so CRC scanning runs with the resolution workers rather than serially in the coordinator. The only unsafe codec operation marks a just-initialized `Vec` result as initialized after writing every spare-capacity byte. Add architecture-specific SIMD or further cross-codec abstraction only after profiling; `libbz2-rs-sys` and `flate2` remain dev-only differential oracles.

## Commands

Expand All @@ -49,30 +53,45 @@ Run `cargo fmt --check` after Rust edits and `chkstyle` after Python edits once

## Correctness and performance acceptance

The normal release test path decodes selected valid and corrupt cases from the maintained upstream `bzip2-testfiles` collection. Generated byte distributions add differential coverage. Valid outputs are compared byte-for-byte with `libbz2-rs-sys`, which is a dev-only oracle and never part of production decoding.
The normal release test path decodes selected valid and corrupt cases from the maintained upstream `bzip2-testfiles` collection. Generated byte distributions add differential coverage. Valid bzip2 outputs are compared byte-for-byte with `libbz2-rs-sys`. Gzip tests cover stored, fixed-Huffman, and dynamic-Huffman blocks; optional headers and FHCRC; concatenated members; truncation; and trailer corruption across varied inputs and compression levels generated by `flate2`. Both oracles are dev-only and never part of production decoding.

The same test binary contains a warmed end-to-end performance regression gate capped at 1.3 times the oracle, allowing for noise on shared runners. Representative local acceptance remains 1.2 times the oracle. Keep the whole release test suite below five seconds on the primary development laptop; individual timed workloads should normally be about 0.1 seconds or less. Use the Simple English Wikipedia dump for heavier local throughput, scaling, memory, and time-to-first-output checks. `librapidarchive` was a one-time design comparison, not a retained baseline.
The normal release path contains warmed end-to-end performance regression gates capped at 1.3 times each oracle, allowing for noise on shared runners. The gzip gates independently exercise a highly compressible LZ77-heavy shape and an incompressible literal-heavy shape against `flate2`; the bzip2 gate uses `libbz2-rs-sys`. Representative local acceptance remains 1.2 times the corresponding oracle. The ignored full-wiki gzip test applies that threshold to rapidgzip-rust. Keep the whole release test suite below five seconds on the primary development laptop; individual timed workloads should normally be about 0.1 seconds or less.

Legacy randomized blocks produced by bzip2 versions before 0.9.5 are intentionally unsupported. Supporting that obsolete format would add complexity to the production decoder for data that is not realistically encountered today.

### Local Wikipedia benchmarks

`tests/wiki_perf.rs` contains a release-only local benchmark that is skipped by default. Its git-ignored fixture is a single bzip2 stream containing the first 84,423,012 decoded bytes (about 5%) of SimpleWiki, stored at `meta/simplewiki-first-5pct.xml.bz2`. Run it with:
`tests/wiki_perf.rs` contains release-mode local benchmarks that are skipped by default. The git-ignored SimpleWiki fixtures are a bzip2-compressed 5% prefix, the full bzip2 dump, the same 5% prefix recompressed as gzip, and the full XML recompressed with system `gzip -6`.

The quick bzip2 iteration test reads `meta/simplewiki-first-5pct.xml.bz2` before timing, then verifies decoded length, all CRCs, and BLAKE3:

```bash
cargo test --release --test wiki_perf simplewiki_first_five_percent -- --ignored --nocapture
cargo test --release --test wiki_perf simplewiki_first_five_percent -- --ignored --exact --nocapture
```

It uses all available CPUs by default. Set `FASTBZ2_THREADS` to compare an explicit thread count. The timed section includes allocation and decompression but excludes reading the compressed file and calculating its BLAKE3; decoded length, block/stream CRCs, and BLAKE3 are all checked.
The full bzip2 confirmation streams to a counting sink and validates every block and stream CRC:

```bash
cargo test --release --test wiki_perf simplewiki_full -- --ignored --exact --nocapture
```

For an occasional full-dump confirmation, create `meta/simplewiki-full.xml.bz2` as a symlink to the local dump and run:
The gzip acceptance test warms both executables with `meta/simplewiki-first-5pct.xml.gz`, performs exactly one measured full-dump validation with each, and fails above 1.2× the sibling rapidgzip-rust checkout:

```bash
cargo test --release --test wiki_perf simplewiki_full -- --ignored --nocapture
cargo test --release --test wiki_perf gzip_reference_ratio -- --ignored --exact --nocapture
```

The full test streams decoded bytes into a counting sink and validates every block and stream CRC. Stream headers found during the structural scan remain speculative: only the exact byte-aligned header following a validated end-of-stream marker starts another stream.
Set `FASTBZ2_THREADS` to use an explicit worker count. `RAPIDGZIP_BIN` can point at another reference executable. The warm-up is deliberately the small fixture, not an unreported repeat of the measured full workload.

Time/CPU/RSS and ungated macOS physical-footprint diagnostics are separate because process inspection can perturb sub-second parallel timings:

```bash
cargo test --release --test wiki_perf gzip_cli_process_metrics -- --ignored --exact --nocapture
cargo test --release --test wiki_perf rapidgzip_rust_process_metrics -- --ignored --exact --nocapture
cargo test --release --test wiki_perf system_gzip_process_metrics -- --ignored --exact --nocapture
```

The metrics helper uses `wait4` and, on macOS, `proc_pid_rusage` on its own child; it needs no task-inspection permission. Treat its wall time as diagnostic and use `gzip_reference_ratio` for the speed acceptance ratio.
The 1,000-stream enwiki comparison has a separate ignored test for each implementation and mode so a changed decoder can be measured without rerunning unchanged baselines. Each test reads the compressed fixture before starting its single timed decode, with no warmups or repeats:

```bash
Expand All @@ -91,14 +110,16 @@ The decoded lengths and the 5% BLAKE3 in `tests/wiki_perf.rs` are acceptance val
Set `wiki` to the checkout containing the Wikimedia dumps:

```bash
wiki=/path/to/wiki2md
wiki=/path/to/parse-wiki
```

Recreate the SimpleWiki fixtures from its validated compressed and decoded files:

```bash
head -c 84423012 "$wiki/data/simplewiki-latest-pages-articles.xml" | bzip2 -9c > meta/simplewiki-first-5pct.xml.bz2
ln -s "$wiki/data/simplewiki-latest-pages-articles.xml.bz2" meta/simplewiki-full.xml.bz2
head -c 84423012 "$wiki/data/simplewiki-latest-pages-articles.xml" | gzip -6c > meta/simplewiki-first-5pct.xml.gz
gzip -6c "$wiki/data/simplewiki-latest-pages-articles.xml" > meta/simplewiki-full.xml.gz
```

Run the 5% test to obtain and verify its decoded length and BLAKE3. Obtain the full decoded length with `stat`; update `FIVE_PERCENT_LEN`, `FIVE_PERCENT_BLAKE3`, or `FULL_LEN` only when intentionally changing a fixture.
Expand Down
Loading