Pure-managed compression primitives, entropy coders, transforms, hashing, bit I/O, and reusable building blocks implemented clean-room in C# with no native compression dependency.
dotnet add package Hawkynt.Compression.CoreThe package also bundles Compression.Registry, so the common IBuildingBlock registry surface is available from the same NuGet installation.
- Clean-room implementations based on public specifications and papers rather than ports of native compression libraries.
- Composable dictionary coders, entropy coders, transforms, filters, integer codes, hashes, and bit-level I/O.
- Uniform registry surface for comparing and composing building blocks.
- Direct concrete APIs remain available when an algorithm exposes meaningful non-default parameters.
- Pure managed code for environments where native
zlib,liblzma,libarchive, or similar dependencies are undesirable. - Round-trip-oriented test coverage plus official/reference vectors where available.
This is a curated package-level map, not a hand-maintained claim to list every compiled building block. For the exact inventory in the version you reference, query BuildingBlockRegistry.All.
| Algorithm / primitive | Family | State | Notes | Reference |
|---|---|---|---|---|
| DEFLATE | Dictionary + entropy | R/W | Raw RFC 1951 building block | RFC 1951 |
| LZ77 | Dictionary | R/W | Sliding-window dictionary coding | Ziv & Lempel 1977 |
| LZ78 | Dictionary | R/W | Phrase-dictionary coding | Ziv & Lempel 1978 |
| LZW | Dictionary | R/W | Variable-width dictionary coding | Welch 1984 |
| LZ4 | Dictionary | R/W | Fast block compression | LZ4 block format |
| Snappy | Dictionary | R/W | Fast block compression | Snappy format |
| Brotli | Dictionary + entropy | R/W |
Decoder accepts more of the format than the encoder chooses to emit | RFC 7932 |
| LZMA | Dictionary + range coding | R/W | LZMA primitive | 7-Zip LZMA SDK |
| LZX | Dictionary + Huffman | R/W | Used by CAB/CHM/WIM families | Microsoft LZX |
| Zstandard entropy stages | FSE / Huffman | R/W | Reusable entropy components | RFC 8878 |
| Huffman coding | Entropy | R/W | Static/canonical Huffman primitives | Huffman 1952 |
| Arithmetic coding | Entropy | R/W | Adaptive arithmetic coder | Witten, Neal & Cleary 1987 |
| Range coding | Entropy | R/W | Arithmetic-coding family primitive | Martin 1979 |
| rANS | Entropy | R/W | Range Asymmetric Numeral Systems | Duda 2009 |
| FSE | Entropy | R/W | Finite State Entropy / tANS | FSE project |
| PPM | Context modelling | R/W | Prediction by Partial Matching | Cleary & Witten 1984 |
| Context Tree Weighting | Context modelling | R/W | Universal context weighting | Willems et al. |
| Burrows-Wheeler transform | Transform | R/W | Reversible block transform | Burrows & Wheeler 1994 |
| Move-to-front | Transform | R/W | Often paired with BWT | Bentley et al. 1986 |
| Run-length encoding | Transform | R/W | Generic RLE stages | Overview |
| CRC-32C | Hash/checksum | Compute | Hardware-assisted where available | RFC 3720 Appendix B |
| xxHash | Hash | Compute | Fast non-cryptographic hashing | xxHash |
| BLAKE2 | Hash | Compute | Cryptographic hash family | RFC 7693 |
R/W means the primitive exposes both compression/encoding and decompression/decoding paths. ⚠️ marks a deliberate subset whose limits matter for interoperability.
using Compression.Registry;
IBuildingBlock lzw = BuildingBlockRegistry.GetById("BB_Lzw")!;
byte[] compressed = lzw.Compress(originalBytes);
byte[] restored = lzw.Decompress(compressed);using Compression.Core.Dictionary.Lzw;
using var stream = new MemoryStream();
var encoder = new LzwEncoder(stream, minBits: 9, maxBits: 12);
encoder.Encode(originalBytes);
stream.Position = 0;
var decoder = new LzwDecoder(stream, minBits: 9, maxBits: 12);
byte[] restored = decoder.Decode(originalBytes.Length);using Compression.Core.BitIO;
using var stream = new MemoryStream();
var writer = new BitWriter(stream, BitOrder.MsbFirst);
writer.WriteBits(0b1011_0010, count: 8);
writer.WriteBits(0xF, count: 4);
writer.Flush();
stream.Position = 0;
var reader = new BitReader(stream, BitOrder.MsbFirst);
int byteValue = reader.ReadBits(8);
int nibble = reader.ReadBits(4);using Compression.Core.Hashing;
uint crc = Crc32C.Compute(data);
uint xx32 = XxHash32.Compute(data);
ulong fnv = Fnv1a64.Compute(data);
byte[] sha256 = Sha256.Compute(data);| Goal | Typical family to inspect |
|---|---|
| Fast dictionary compression | LZ4 / Snappy / LZO-style implementations present in the registry |
| General-purpose dictionary + entropy compression | DEFLATE / LZMA / Brotli-related implementations present in the registry |
| Transform pipelines | BWT → MTF → entropy coding |
| Adaptive entropy coding | Arithmetic / range / ANS-family implementations |
| Integer coding | Golomb/Rice, Exp-Golomb, Elias-family implementations |
| Research/experimental comparison | Whatever the current registry exposes for that build |
The source-repository CLI can benchmark the currently registered building blocks on representative input:
+cwb benchmark sample.binBuildingBlockRegistry.All is the authoritative list of what the referenced build actually contains.
Compression.Core/Hawkynt.Compression.Core.csproj is packable and uses the project filename as the NuGet package ID: Hawkynt.Compression.Core.
The project references Compression.Registry with PrivateAssets="all" and adds the resolved registry assembly to the package's lib/<tfm> output. Consumers therefore install one NuGet package while still getting the registry contracts used by Core.
The repository currently contains these other packable public package projects alongside Core:
| Package | Project | Purpose |
|---|---|---|
Hawkynt.FileFormats.Audio |
Hawkynt.FileFormats.Audio/Hawkynt.FileFormats.Audio.csproj |
Audio codecs and audio/container formats |
Hawkynt.FileFormats.Archives |
Hawkynt.FileFormats.Archives/Hawkynt.FileFormats.Archives.csproj |
Compression streams and archive/container formats |
Hawkynt.FileFormats.FileSystems |
Hawkynt.FileFormats.FileSystems/Hawkynt.FileFormats.FileSystems.csproj |
Filesystems and disk-image formats |
No additional package is named here unless a corresponding checked-in package surface actually exists.
Core is the reusable primitive layer. Its checked-in code is organised around concerns such as:
| Area | Examples | Role |
|---|---|---|
| Bit I/O | BitReader, BitWriter, BitOrder |
Bit-level parsing and emission used by variable-length coders |
| Dictionary coding | LZ-family implementations, DEFLATE-related primitives | Match-based compression building blocks |
| Entropy coding | Huffman, arithmetic/range coding, ANS-family primitives | Symbol coding and probability-model stages |
| Transforms | BWT, MTF, RLE, delta/BCJ-style transforms | Reversible preprocessing stages |
| Hashing/checksums | CRC-family, xxHash-family and cryptographic hashes present in Core | Integrity, lookup and format support |
| SIMD helpers | match-length/copy/histogram helpers | Accelerated hot-path primitives where supported |
| Streams | sub/concatenated stream helpers | Reusable format-reader plumbing |
| Disk-image helpers | MBR/GPT and partition-related primitives | Shared lower-level disk/container parsing |
The registry provides a common comparison surface; it does not erase algorithm-specific semantics. Concrete APIs remain appropriate when streaming, allocation, tuning parameters, or format-specific behavior matters.
A primitive used by a file format does not by itself imply full support for that format, and decode and encode coverage may legitimately be asymmetric. Documentation keeps those claims separate.
The public support matrix marks deliberate subsets with ⚠️. Brotli is one example: the decoder accepts more of the format than the encoder chooses to emit. Such distinctions belong in the support table and implementation discussion rather than being hidden behind a generic “supported” label.
For algorithm-specific investigations, inspect the implementation, nearby comments/tests, and dedicated repository documents where they exist. Examples include docs/LZMS-ON-DISK.md and Compression.Core/SqxFormat/README.md.
The repository test suite is the evidence source for implementation claims. Relevant test styles include:
- compress → decompress byte-identical round trips;
- official or specification-derived vectors where available;
- external-tool interoperability tests where the repository provides them;
- targeted regression tests for discovered edge cases.
An intended feature, TODO, experiment, issue, or roadmap item is not a support claim until the implementation and corresponding evidence exist.
The checked-in base version comes from the nearest MSBuild version declaration. At repository level, Directory.Build.props currently declares:
<Version>1.0.0</Version>The repository's .github/workflows/scripts/version.pl composes .NET package versions as X.Y.Z.BUILD, where BUILD is derived from the commit count for the directory that declares the effective version. Release workflows may pass that computed version into packing.
A consumer should reference the actual NuGet package ID:
<PackageReference Include="Hawkynt.Compression.Core" Version="1.0.0" />Use the concrete version you intend to consume; this document does not predict a future release number or stability milestone.
Every public and protected member of all 540 types, generated from the built assembly and its XML documentation, is in REFERENCE.md.
| Dependency | Packaging behaviour |
|---|---|
Compression.Registry |
Project dependency bundled into Core's package output |
| Native compression libraries | None required by the Core package design |
Audio, archive and filesystem functionality lives in the separate package projects listed above and is not pulled into Core transitively.
- A common API shape does not imply identical streaming, memory, or parameter semantics across every algorithm; use concrete APIs when those distinctions matter.
- Some encoders intentionally implement a standards-compliant subset while decoders accept a wider format.
- Pure managed code is the design goal, not an automatic speed claim. BCL/native implementations may be faster for common algorithms on some workloads.
- Do not infer a package, format, algorithm, profile, or release state from roadmap intent. Checked-in project files, the compiled registry/public API, and tests are the evidence sources.
- Do not state volatile algorithm counts unless they are generated from the registry/build.
- When code and prose disagree, the compiled registry/API and tests win; update the prose.
Open issues and pull requests in Hawkynt/CompressionWorkbench. Repository contribution and CI rules are documented in AGENTS.md and CONTRIBUTING.md.
If this project saves you time or money, consider supporting its development:
Licensed under LGPL-3.0-or-later — see the repository LICENSE.