Add streaming tar extraction and faster parallel gzip

Cargo.toml

@@ -31,6 +31,7 @@ pyo3 = { version = ">=0.29.2", optional = true }
 rayon = "1.12.0"
 serde_json = "1.0.151"
 tempfile = "3.27.0"
+tar = { version = "0.4.46", default-features = false }
 
 [dev-dependencies]
 crabz2 = { version = "0.4.0", features = ["parallel"] }

DEV.md

@@ -12,11 +12,13 @@ src/decode.rs         serial/parallel decode scheduling and index construction
 src/decoder.rs        bzip2 block machinery and 12-bit Huffman fast tables
 src/format.rs         cheap structural scan for header and marker candidates
 src/gzip.rs           gzip framing, LSB-first DEFLATE, CRC32, and block reports
+src/output.rs         owned/borrowed decoded-output sink abstraction
 src/pipeline.rs       shared ordered, byte-budgeted, staged worker scheduler
 src/index.rs           stable persistent index format
 src/indexed.rs         seekable decoded view and block cache
 src/lib.rs            public Rust API and private PyO3 binding
 src/source.rs         owned and memory-mapped compressed sources
+src/bin/fastbz2/tar_extract.rs  bounded decode-to-tar bridge, staging, and commit
 python/fastbz2/       thin Python I/O wrapper over fastbz2._core
 build_backend.py      stage the native CLI for PEP 517 wheel builds
 tests/corpus/         selected upstream conformance and corruption fixtures
@@ -26,11 +28,13 @@ tools/stage_binaries.py copy the release executable into Maturin wheel data
 
 The current bzip2 scanner deliberately does not treat 48-bit marker matches or later `BZh` headers as validated structure. Full decoding must establish the exact block chain and validate every block CRC plus the combined stream CRC before marker candidates can become trusted index entries. Python integration tests use standard-library `bz2`/libbz2 as an independent fixture generator.
 
-The gzip decoder is an in-repo RFC 1952/RFC 1951 implementation rather than a wrapper around a production codec. It parses optional headers and concatenated members, decodes stored/fixed/dynamic blocks, maintains the 32 KiB LZ77 history, and validates FHCRC, CRC32, and ISIZE. For large inputs, independently discovered dynamic-block boundaries seed unknown history with compact markers. A marker-free history switches the same decoder to byte output; otherwise the coordinator resolves only the suffix needed by the successor and queues full resolution plus CRC on the shared staged scheduler. Reports retain member boundaries, DEFLATE block ranges, and accepted/fallback chunk counts. `crc32fast` is the sole production helper; `flate2` is dev-only.
+The gzip decoder is an in-repo RFC 1952/RFC 1951 implementation rather than a wrapper around a production codec. It parses optional headers and concatenated members, decodes stored/fixed/dynamic blocks, maintains the 32 KiB LZ77 history, and validates FHCRC, CRC32, and ISIZE. For large inputs, independently discovered dynamic-block boundaries seed unknown history with compact markers. Primary jobs compute the CRC of each known clean suffix before ordered resolution. Resolution workers resolve the marker prefix, hash that prefix, and combine the two CRCs without rescanning the clean bytes. A marker-free history switches the same decoder to byte output. Reports retain member boundaries, DEFLATE block ranges, and accepted/fallback chunk counts. `crc32fast` is the sole production helper; `flate2` is dev-only.
 
 The decoder remains independent of files, threads, Python, and the CLI. Parallel scanning/decoding and indexed seeking are layered over it. Native workers never call Python. Large offsets use explicit 64-bit bit/byte types, and speculative block-marker hits are accepted only when they form an exact stream chain with valid block and combined stream CRCs.
 
-Both core decode APIs report completed compressed and decoded byte counts without knowing anything about terminals. The CLI selects bzip2 or gzip by a recognised extension and falls back to magic for stdin or unknown names. It layers delayed, rate-limited TTY progress rendering over the shared callbacks; redirected stderr and `--quiet` produce no progress output. Decoded files use same-directory temporary files and atomic persistence, then inherit the compressed input's modification time and permissions. `--rm` removes an input only after decode, persistence, and metadata copying all succeed. Output-size limits are enforced by a writer wrapper, so each decoder has one code path for files, stdout, validation, and listing.
+Both core decode APIs report completed compressed and decoded byte counts without knowing anything about terminals. The CLI selects bzip2 or gzip by a recognised extension and falls back to magic for stdin or unknown names. It layers delayed, rate-limited TTY progress rendering over the shared callbacks; redirected stderr and `--quiet` produce no progress output. Decoded files use same-directory temporary files and atomic persistence, then inherit the compressed input's modification time and permissions. `--rm` removes an input only after decode, persistence, and metadata copying all succeed. An `OutputSink` wrapper enforces output-size limits, so each decoder has one code path for files, stdout, validation, listing, and tar extraction.
+
+Tar format semantics use the mature `tar` crate, pinned from 0.4.46 and built without its optional xattr feature. It handles streaming GNU/PAX/long-name/link entries and confines extracted paths to the destination. A zero-capacity rendezvous channel transfers each owned decoder chunk and its live suffix offset to `tar::Archive`. The channel queues no chunks and applies backpressure. `tar::Archive` pulls data through `Read`, which copies once from the current chunk into its request buffer. Extraction writes immediately into a same-filesystem temporary directory, drains all trailing tar padding so codec validation completes, then preflights every destination conflict and moves entries into place with renames. Multiple inputs remain sequential so their per-codec worker pools cannot oversubscribe the global thread budget.
 
 The shared `pipeline.rs` scheduler provides ordered results, byte-budgeted admission, cancellation, and a staged priority queue. Bzip2 uses the rolling candidate path: workers reserve the maximum possible decoded block size, then shrink that reservation to actual retained output until ordered validation consumes or rejects it. Gzip uses the staged path: native workers alternate speculative DEFLATE decoding with higher-priority marker resolution, while the coordinator advances only the 32 KiB dependency windows and emits resolved chunks in order. Decode results and outstanding resolution results have separate bounded horizons, preventing either dependency stalls or unbounded memory.
 
@@ -53,10 +57,36 @@ Run `cargo fmt --check` after Rust edits and `chkstyle` after Python edits once
 
 ## Correctness and performance acceptance
 
-The normal release test path decodes selected valid and corrupt cases from the maintained upstream `bzip2-testfiles` collection. Generated byte distributions add differential coverage. Valid bzip2 outputs are compared byte-for-byte with `libbz2-rs-sys`. Gzip tests cover stored, fixed-Huffman, and dynamic-Huffman blocks; optional headers and FHCRC; concatenated members; truncation; and trailer corruption across varied inputs and compression levels generated by `flate2`. Both oracles are dev-only and never part of production decoding.
+The normal release test path decodes selected valid and corrupt cases from the maintained upstream `bzip2-testfiles` collection. Generated byte distributions add differential coverage. Valid bzip2 outputs are compared byte-for-byte with `libbz2-rs-sys`. Gzip tests cover stored, fixed-Huffman, and dynamic-Huffman blocks; optional headers and FHCRC; concatenated members; truncation; and trailer corruption across varied inputs and compression levels generated by `flate2`. Both oracles are dev-only and never part of production decoding. CLI tests generate tar archives and cover gzip/bzip2 wrappers, compound-extension dispatch, long names, stdin, raw-tar output, output limits, overwrite preflight, late checksum failure, and traversal confinement.
 
 The normal release path contains warmed end-to-end performance regression gates capped at 1.3 times each oracle, allowing for noise on shared runners. The gzip gates independently exercise a highly compressible LZ77-heavy shape and an incompressible literal-heavy shape against `flate2`; the bzip2 gate uses `libbz2-rs-sys`. Representative local acceptance remains 1.2 times the corresponding oracle. The ignored full-wiki gzip test applies that threshold to rapidgzip-rust. Keep the whole release test suite below five seconds on the primary development laptop; individual timed workloads should normally be about 0.1 seconds or less.
 
+### Local archive extraction benchmarks
+
+`tests/archive_perf.rs` measures the tar layer on the real `meta/simplewiki-first-5pct.xml.bz2` corpus. Fixture decoding and gzip/bzip2 recompression finish before timing. Each ignored test warms one target and measures it once. Run only the implementation changed:
+
+```bash
+cargo test --release --test archive_perf tgz_fastbz2_overhead -- --ignored --exact --nocapture
+cargo test --release --test archive_perf tgz_system_reference -- --ignored --exact --nocapture
+cargo test --release --test archive_perf tbz2_fastbz2_overhead -- --ignored --exact --nocapture
+cargo test --release --test archive_perf tbz2_system_reference -- --ignored --exact --nocapture
+cargo test --release --test archive_perf tar_crate_reference -- --ignored --exact --nocapture
+FASTBZ2_THREADS=18 cargo test --release --test archive_perf tgz_output_cadence -- --ignored --exact --nocapture
+```
+
+These are single runs after owned-suffix transfer and the 512 KiB gzip grid change:
+
+| Format | Raw decode | fastbz2 extraction | Extraction/raw | System `tar` | Extraction/system |
+|---|---:|---:|---:|---:|---:|
+| `.tgz` | 39.919 ms | 56.850 ms | 1.424x | 117.962 ms | 0.482x |
+| `.tar.bz2` | 148.411 ms | 151.783 ms | 1.023x | 1.168 s | 0.130x |
+
+Direct extraction of the uncompressed in-memory tar through the `tar` crate took 32.069 ms. Raw gzip decode plus direct tar extraction totals 71.988 ms. The combined pipeline takes 56.850 ms and hides 15.138 ms, or 47%, of the direct tar work.
+
+The cadence benchmark identified ordered gzip output as the main overlap limit. With a 1 MiB speculative grid, output began at 8.491 ms, reached 25% at 29.595 ms, and completed at 33.724 ms. A 512 KiB grid began at 4.798 ms, reached 25% at 23.993 ms, and completed at 32.915 ms. A 256 KiB grid emitted earlier but slowed raw decode to 38.073 ms and extraction to 57.792 ms. The 512 KiB grid gave the best measured balance. Computing each clean suffix CRC in its primary job moved 25% output to 19.818 ms, 75% to 30.355 ms, and completion to 30.678 ms. The corresponding extraction run was effectively flat at 56.850 ms. Tar cannot process later bytes while an earlier ordered gzip segment remains incomplete. A custom tar parser would not remove that dependency. A one-chunk channel buffer regressed extraction to 59.698 ms, so the bridge retains its zero-capacity rendezvous.
+
+System `tar` remains the external reference and 1.2x remains the research target. Raw-tar output is a lower bound rather than an extractor reference. The fastbz2 tests use a broad 3x raw-decode regression guard. Keep the measurements single-run; change an implementation before rerunning it.
+
 Legacy randomized blocks produced by bzip2 versions before 0.9.5 are intentionally unsupported. Supporting that obsolete format would add complexity to the production decoder for data that is not realistically encountered today.
 
 ### Local Wikipedia benchmarks
@@ -75,9 +105,10 @@ The full bzip2 confirmation streams to a counting sink and validates every block
 cargo test --release --test wiki_perf simplewiki_full -- --ignored --exact --nocapture
 ```
 
-The gzip acceptance test warms both executables with `meta/simplewiki-first-5pct.xml.gz`, performs exactly one measured full-dump validation with each, and fails above 1.2× the sibling rapidgzip-rust checkout:
+The fastbz2-only gzip test warms with `meta/simplewiki-first-5pct.xml.gz` and performs one full-dump validation. The ratio test warms both executables, measures each full dump once, and fails above 1.2x the sibling rapidgzip-rust checkout:
 
 ```bash
+cargo test --release --test wiki_perf gzip_fastbz2_validation -- --ignored --exact --nocapture
 cargo test --release --test wiki_perf gzip_reference_ratio -- --ignored --exact --nocapture
 ```
 

README.md

@@ -1,10 +1,10 @@
 # fastbz2
 
-An active compression-format research workbench with fast bzip2 and gzip decompression.
+An active compression-format research workbench with fast bzip2/gzip decompression and streaming tar extraction.
 
-`fastbz2` provides a native CLI, a Rust library, and a Python module. The CLI auto-selects an in-repo bzip2 or gzip decoder from the filename extension, falling back to stream magic when needed. Both decoders handle concatenated streams and fully validate their checksums. The bzip2 implementation also provides parallel decoding and persistent random-access indexes.
+`fastbz2` provides a native CLI, a Rust library, and a Python module. The CLI auto-selects an in-repo bzip2 or gzip decoder from the filename extension, falling back to stream magic when needed, and streams compressed tar variants through a bounded extractor. Both decoders handle concatenated streams and fully validate their checksums. The bzip2 implementation also provides parallel decoding and persistent random-access indexes.
 
-Compression and additional formats are planned, but the current implementation decompresses bzip2 and gzip.
+Compression is planned, but the current implementation decompresses bzip2 and gzip and extracts their tar-wrapped variants.
 
 ## Install
 
@@ -25,24 +25,28 @@ cargo add fastbz2 --git https://github.com/AnswerDotAI/fastbz2
 
 ## CLI
 
-Decoding is the default operation. `.bz2`, `.bzip2`, `.gz`, and `.gzip` select their corresponding decoder and are removed from the output name. `.tbz`, `.tbz2`, and `.tgz` produce a `.tar` filename; this currently decompresses the tar stream rather than extracting its entries. For stdin and unrecognised extensions, bzip2 or gzip magic selects the decoder. Other input names gain `.out`.
+Decoding is the default operation. `.bz2`, `.bzip2`, `.gz`, and `.gzip` select their corresponding decoder and are removed from the output name. Compressed tar names—`.tar.bz2`, `.tar.bzip2`, `.tbz`, `.tbz2`, `.tar.gz`, `.tar.gzip`, and `.tgz`—automatically extract into the current directory or `-C/--output-dir`. `-x/--extract` forces tar extraction for stdin or an unusual filename; an explicit `-o/--output` instead writes the decoded tar stream. For stdin and unrecognised extensions, bzip2 or gzip magic selects the decoder. Other non-archive input names gain `.out`.
 
 ```bash
-fastbz2 dump.xml.bz2                 # write dump.xml
-fastbz2 events.json.gz               # write events.json
-fastbz2 dump.xml.bz2 -o result.xml   # choose the output path
-fastbz2 dump.xml.bz2 -o -            # write plaintext to stdout
-fastbz2 -                             # read compressed data from stdin
+fastbz2 dump.xml.bz2                   # write dump.xml
+fastbz2 events.json.gz                 # write events.json
+fastbz2 source.tar.gz                  # extract into the current directory
+fastbz2 source.tbz2 -C unpacked        # extract into unpacked/
+fastbz2 --extract -C unpacked -        # extract gzip/bzip2 tar data from stdin
+fastbz2 source.tgz -o source.tar       # decode without extracting
+fastbz2 dump.xml.bz2 -o result.xml     # choose the decoded output path
+fastbz2 dump.xml.bz2 -o -              # write decoded bytes to stdout
 ```
 
-Multiple inputs are decoded in order, with parallelism applied inside each file. `-C/--output-dir` collects their outputs in one directory:
+Multiple inputs are processed in order, with parallelism applied inside each compressed stream. `-C/--output-dir` collects decoded files and is the extraction root for archives:
 
 ```bash
 fastbz2 data/*.bz2 logs/*.gz -C decoded
 fastbz2 data/*.bz2 logs/*.gz -C decoded --skip-existing
+fastbz2 backups/*.tgz -C restored
 ```
 
-The alternative modes are flags rather than subcommands:
+Validation and inspection remain flags rather than subcommands:
 
 ```bash
 fastbz2 --test dump.xml.bz2          # fully decode and validate, writing nothing
@@ -51,15 +55,16 @@ fastbz2 --list events.json.gz        # print the validated member/block layout
 fastbz2 --list --json dump.xml.bz2   # emit the complete layout as JSON
 ```
 
-`--test`, `--index`, and `--list` are mutually exclusive. Human-readable `--list` output labels each input when given multiple files; JSON output is one object for one input and an array for multiple inputs.
+`--test`, `--index`, `--list`, and explicit `--extract` are mutually exclusive. Human-readable `--list` output labels each input when given multiple files; JSON output is one object for one input and an array for multiple inputs.
 
 ### Output safety
 
-- Existing outputs are rejected by default. Use `--force` to replace them or `--skip-existing` to leave them untouched.
-- File outputs are written to a temporary file in the destination directory and persisted atomically only after successful CRC validation.
-- Extracted files inherit the compressed input's permissions and modification time.
-- `--rm` removes each compressed input only after its output has been persisted and its metadata copied successfully.
-- `--max-output SIZE` limits decoded bytes per input. Sizes accept binary suffixes such as `K`, `MiB`, and `G`.
+- Existing decoded files and archive entries are rejected by default. `--force` replaces them; `--skip-existing` applies to decoded files rather than archives.
+- Decoded-file outputs use a same-directory temporary file and become visible atomically only after successful checksum validation.
+- Tar entries stream into a same-filesystem staging directory through a bounded pipe. They are preflighted and moved into the destination only after both the compression stream and tar archive validate, so a late CRC failure leaves no extracted files.
+- Tar paths and link targets are confined to the destination; unsafe entries are skipped. New entries use the archive's permissions and modification times. Standalone decoded files inherit those values from the compressed input.
+- `--rm` removes each compressed input only after its decoded file or all archive entries have been committed successfully.
+- `--max-output SIZE` limits decoded bytes per input, including tar framing and padding. Sizes accept binary suffixes such as `K`, `MiB`, and `G`.
 
 Long interactive operations report completion, decoded throughput, compression ratio, and ETA on stderr. Progress is disabled automatically when stderr is redirected; `-q/--quiet` also suppresses progress and skip notices.
 
@@ -157,7 +162,7 @@ Full SimpleWiki recompressed with system `gzip -6` (`438,904,466` bytes compress
 | Decoder | Mode | Seconds | Peak physical footprint |
 |---|---|---:|---:|
 | rapidgzip-rust, local checkout | auto parallel, validation sink | 0.363 | 585 MiB |
-| fastbz2 | auto parallel, validation sink | 0.357 | 552 MiB |
+| fastbz2 | auto parallel, validation sink | 0.326 | 325 MiB |
 | Apple gzip | serial, stdout discarded | 1.371 | 1.2 MiB |
 
 The memory values use macOS physical footprint rather than `ru_maxrss`. The fastbz2 CLI memory-maps its 419 MiB input, so clean reclaimable file pages make RSS look roughly 419 MiB larger; `pread`-based tools leave the same cached pages outside process RSS. Physical footprint makes the comparison meaningful.