Skip to content

+ implement greedy and exhaustive byte-pair compression - #102

Merged
Hawkynt merged 1 commit into
mainfrom
feat/bpe-building-block
Aug 29, 2026
Merged

+ implement greedy and exhaustive byte-pair compression#102
Hawkynt merged 1 commit into
mainfrom
feat/bpe-building-block

Conversation

@Hawkynt

@Hawkynt Hawkynt commented Aug 28, 2026

Copy link
Copy Markdown
Owner

What changed

  • replace the existing 16-bit BPE demonstrator with Philip Gage-style byte-pair compression
  • keep encoded symbols byte-sized by substituting pairs with byte values absent from each block
  • keep the parameterless/default building block on deterministic greedy construction with 65,535-byte blocks
  • add BpeConstructionStrategy.Exhaustive, which memoizes every profitable merge sequence within 64-byte exact-search blocks and chooses the smallest serialized grammar
  • keep greedy and exhaustive output on the same wire format; either instance can decode the other's stream
  • fall back to raw storage whenever a constructed block is not smaller
  • reject malformed block framing, duplicate codes, forward/cyclic grammar references, length mismatches, and trailing data
  • classify BPE as a dictionary/grammar compressor rather than an entropy coder

Verification

Added NUnit coverage for:

  • empty input
  • repetitive single-byte data and ratio
  • repeated multi-byte phrase and ratio
  • random incompressible input
  • all 256 byte values / raw fallback
  • multi-block boundary round-trip
  • deterministic greedy wire vector
  • a non-greedy 13-byte counterexample where exhaustive construction beats greedy/raw fallback
  • cross-decoding between greedy and exhaustive streams
  • exhaustive search-block boundary round-trip
  • malformed forward/cyclic grammar rejection
  • stable registry metadata and default strategy

The previous head completed the Ubuntu and Windows build/core/interop/OS/polyglot/performance jobs successfully. The local execution environment does not have the .NET SDK installed, so the new exhaustive-strategy commit is compiled and tested by repository CI.

@Hawkynt Hawkynt changed the title + implement byte-pair compression building block + implement greedy and exhaustive byte-pair compression Aug 28, 2026
@Hawkynt
Hawkynt force-pushed the feat/bpe-building-block branch from d8d00b1 to 8e1d0b5 Compare August 28, 2026 16:29
Replaces the 16-bit demonstrator with Gage-style byte-pair encoding: pairs are
substituted with byte values absent from the block, so symbols stay byte-sized.
The default stays deterministic greedy over 65,535-byte blocks; Exhaustive
memoizes every profitable merge sequence within 64-byte blocks and keeps the
smallest grammar. Both write the same wire format and can read each other, and
a block that did not get smaller is stored raw.

Malformed framing, duplicate codes, forward and cyclic grammar references,
length mismatches and trailing data are all rejected. BPE is filed as a
dictionary/grammar compressor, not an entropy coder.
@Hawkynt
Hawkynt force-pushed the feat/bpe-building-block branch from 8e1d0b5 to 382f62d Compare August 28, 2026 18:23
@Hawkynt
Hawkynt merged commit 9429f61 into main Aug 29, 2026
3 checks passed
@Hawkynt
Hawkynt deleted the feat/bpe-building-block branch August 29, 2026 03:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant