feat(writer): add ZLIB, Snappy, LZ4, and Zstd compression support

[Allegre7tto]

feat(writer): add ZLIB, Snappy, LZ4, and Zstd compression support

Background

While working on Apache Auron, I found that the native Rust ORC writer is still missing several capabilities required by downstream projects. In particular, it currently writes uncompressed ORC files even though the reader already supports multiple compression codecs.

I would like to contribute to improving the writer, and writer-side compression seems like a useful and reasonably self-contained place to start.

Writer development is already tracked in #15, and there are two related open pull requests, #82 and #84, which have not received further updates since early May. This implementation also addresses several points raised around #82, including avoiding unnecessary allocations in the uncompressed path, reusing the existing compression infrastructure, adding LZ4 support, and validating interoperability with PyORC.

Proposed scope

This PR adds configurable writer-side compression for:

• ZLIB
• Snappy
• LZ4
• Zstd

LZO remains unsupported for writing and produces an explicit error when selected.

The implementation follows ORC's block compression format:

• input streams are divided into configurable compression blocks;
• each block is prefixed with the ORC three-byte compression header;
• compressed data is used only when it is smaller than the original block;
• otherwise, the block is stored using ORC's original-block fallback;
• the block size must be greater than zero and smaller than the 23-bit ORC block-length limit.

Compression is applied independently to:

• column streams;
• stripe footers;
• file metadata;
• the file footer.

The PostScript remains uncompressed and records the selected compression kind and block size so that readers can decode the rest of the file correctly.

Compression is disabled by default, preserving the existing writer behavior.

Proposed API

Compression can be configured through ArrowWriterBuilder:

use orc_rust::{compression::CompressionType, ArrowWriterBuilder};

let mut writer = ArrowWriterBuilder::new(file, schema)
    .with_compression(CompressionType::Zstd)
    .with_compression_block_size(256 * 1024)
    .try_build()?;

writer.write(&batch)?;
writer.close()?;

When no compression codec is configured, the writer continues to produce an uncompressed ORC file.

The default compression block size is 256 KiB when compression is enabled and no custom block size is provided.

Implementation

The work is organized into three commits.

1. feat: add compressor infrastructure with four codec support

This commit extends the existing compression module so that the reader and writer share the same compression types and codec dependencies.

It adds:

• ORC compression-header encoding alongside the existing decoding logic;
• a writer-side Compressor;
• block splitting based on the configured compression block size;
• original-block fallback when compression does not reduce the payload size;
• compressor implementations for ZLIB, Snappy, LZ4, and Zstd;
• explicit rejection of LZO for writer use;
• reusable scratch buffers for codec output;
• unit tests that round-trip compressed blocks through the existing reader decompression path.

Keeping compression and decompression in the same module avoids creating parallel codec abstractions and helps ensure that the generated block format matches what the reader already expects.

2. feat: integrate ORC writer compression

This commit integrates the compressor with ArrowWriterBuilder, ArrowWriter, and StripeWriter.

It adds builder configuration for the codec and block size, then applies compression to each independently encoded ORC stream. Stripe footer lengths, data lengths, and physical stripe offsets are calculated using the compressed byte lengths.

At file close, the implementation also compresses the metadata and file footer before writing an uncompressed PostScript containing:

• the compressed footer length;
• the compressed metadata length;
• the selected CompressionKind;
• the configured compression block size.

When compression is disabled, Compressor::compress returns Cow::Borrowed rather than allocating and copying the input. This preserves the existing direct-write behavior for uncompressed files and avoids introducing an allocation regression into the default code path.

3. test: verify compressed ORC writer compatibility

This commit adds an external interoperability test using PyORC.

For each supported codec, the test:

1. writes an ORC file containing compressible string and integer data;
2. uses a small block size to ensure that the stream contains multiple compression blocks;
3. inspects the first block header to verify that the codec produced an actual compressed block rather than only exercising the original-block fallback path;
4. opens the generated file with PyORC;
5. verifies that all rows are decoded correctly.

The test is ignored by default because it requires a Python environment with PyORC installed.

Tests

The implementation includes tests covering the compression framing, writer integration, file metadata, physical offsets, and external interoperability.

Compression unit tests

• compression-header encoding and decoding round trips;
• known original and compressed header values;
• maximum valid 23-bit block length;
• rejection of zero and oversized block sizes;
• rejection of LZO for writer compression;
• empty input handling;
• block splitting with small configured block sizes;
• original-block fallback for payloads that do not compress;
• round trips for ZLIB, Snappy, LZ4, and Zstd through the existing Decompressor;
• borrowed output when compression is disabled.

Writer tests

• uncompressed writer round trip;
• compressed writer round trips for all four supported codecs;
• PostScript CompressionKind validation;
• custom compression block-size validation;
• default compression block-size behavior;
• empty compressed ORC files;
• invalid compression configuration;
• multi-stripe compressed files;
• stripe offset and compressed-length accounting;
• compressed stripe-footer decoding;
• complete RecordBatch equality after reading the generated file.

External interoperability test

• generates compressed files using ZLIB, Snappy, LZ4, and Zstd;
• verifies that genuinely compressed blocks were emitted;
• reads each file using PyORC;
• compares all decoded rows with the original input.

Run the optional interoperability test with a Python environment containing pyorc:

cargo test --test writer_compression compressed_files_are_readable_by_pyorc \
    -- --ignored --nocapture

Checklist

Tests

• Add compression-header unit tests
• Add codec round-trip tests
• Add invalid-configuration tests
• Add writer round-trip tests
• Add PostScript validation
• Add multi-stripe offset validation
• Add PyORC interoperability coverage
• Add writer compression benchmarks

Validation

• cargo build
• cargo fmt -- --check
• taplo format
• typos
• cargo test --all-features
• cargo test --no-default-features
• cargo clippy --all-targets --all-features -- -D warnings
• cargo clippy --all-targets --no-default-features -- -D warnings

[WenyXu]

@codex review

[Allegre7tto]

It looks like the CI failure may be unrelated to my changes. Is there anything I need to do about it?

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[WenyXu]

It looks like the CI failure may be unrelated to my changes. Is there anything I need to do about it?

fixed in #87