Conversation
Replace the byte-level AHashMap<u8, char> / AHashMap<char, u8> lookup
tables with fixed-size arrays: BYTES_CHAR_TABLE: [char; 256] and
CHAR_BYTES_TABLE: [Option<u8>; 512]. The byte-level alphabet is a fixed,
dense range, so an indexed array removes the per-lookup hashing and probe
on both the normalize/pre-tokenize and decode paths and keeps the tables
contiguous (cache-friendly).
This is the foundational primitive for the fused decode fast path (the
const reverse byte-level table in the decode design): the decoded byte
for a token char becomes a single array index instead of a hash lookup.
Pure refactor, behaviour is byte-for-byte identical.
cargo test --lib: 201 passed, 0 failed.
Perf (single-thread decode): pure refactor, no standalone change. It replaces
the per-byte AHashMap lookup with a single array index, which the fused decode
path ("decode: fused single-buffer zero-copy fast path") turns into the
byte-level decode speedups reported there.
…ecode
Add two ID-indexed structures to BPE, built at construction (BpeBuilder::build,
deserialize) and after training:
* vocab_r_vec: Vec<String> -- reverse vocab indexed by token ID, so
id_to_token() is an O(1) array index instead of an AHashMap<u32,String>
hash+probe. Empty string = no token for that ID.
* vocab_decoded: FlatVocabDecoded -- every token's ByteLevel-decoded bytes
concatenated into ONE contiguous buffer + an offset table, so a token's
raw bytes are a zero-copy &[u8] slice (id_to_decoded_bytes). Instead of
Vec<Vec<u8>> (one alloc + pointer chase per token) the flat layout is
~400KB for GPT-2 (fits in L2) vs ~2.2MB scattered on the heap. This is
the VocabStore/id_to_bytes idea from the decode design: contiguous beats
a HashMap<Rank,Vec<u8>> on cache behaviour.
build_vocab_decoded is byte_fallback-aware: "<0xHH>" tokens decode to their
single byte, and tokens with characters outside the ByteLevel range (e.g.
Metaspace) are left as empty entries so the fused decode path can return None
and fall back to the regular decoder -- keeping decode output correct for all
model/decoder combinations.
Expose id_to_decoded_bytes()/estimate_decode_capacity() on the Model trait
(default None/0) and dispatch them through ModelWrapper (BPE only). No decode
behaviour change yet; this is the storage/accessor layer for the fused decode
fast path.
cargo test --lib: 201 passed, 0 failed.
Perf (single-thread decode): adds the O(1) ID-indexed structures. On its own it
makes id_to_token a flat-array index instead of an AHashMap probe; the large win
lands when the fused decode path consumes vocab_decoded (next commit).
Add a fast path to TokenizerImpl::decode for models that expose pre-decoded token bytes (byte-level BPE). The regular path allocates an owned String per token (id_to_token), collects them into a Vec<String>, runs the decoder (often another Vec<String>), then joins -- O(N) allocations + copies per call. decode_fused does a single pass instead: for each ID, get the model's pre-decoded bytes as a zero-copy &[u8] slice into the contiguous vocab_decoded blob and extend ONE output buffer (pre-sized via estimate_decode_capacity), then validate UTF-8 once at the end with from_utf8 (reuses the buffer on success, from_utf8_lossy fallback for partial byte_fallback sequences). This mirrors tiktoken's id->bytes concat, but zero-copy from a contiguous store. Added/special tokens are handled with an id-range check: the min added-vocab ID is compared against the max input ID once, so normal text (whose IDs are below the added-vocab range) skips the per-token HashMap probe entirely. A cheap up-front gate (estimate_decode_capacity == 0) makes models without a pre-decoded byte table (Unigram, WordPiece, WordLevel) bail before any O(n) work, so their decode path is unchanged and not slowed down. The regular decoder path is otherwise untouched: same allocation behaviour as before when the fast path is not taken. Verified byte-for-byte identical decode output vs the previous implementation across byte-level BPE, byte_fallback (Llama-3), Unigram+Metaspace (ALBERT) and plain BPE, for skip_special_tokens both true and false, on round-trip decode and on decoding the entire id range. cargo test --lib: 201 passed, 0 failed. Perf (single-thread decode, taskset-pinned core, origin/main -> this series): byte-level decode llama3-en (English) llama3-ja (CJK) NVIDIA Vera (Olympus) 49.9 -> 109 MiB/s 48.5 -> 434 MiB/s (2.2x / 9.0x) Intel Granite Rapids 38.2 -> 75 MiB/s 41.5 -> 259 MiB/s (2.0x / 6.3x) AMD Turin 96c (9655P) 56.2 -> 102 MiB/s 57.5 -> 344 MiB/s (1.8x / 6.0x) AMD Turin 128c (9575F) 61.7 -> 113 MiB/s 63.3 -> 383 MiB/s (1.8x / 6.1x) Non-byte-level models (Unigram, WordPiece, WordLevel) are unaffected: the estimate_decode_capacity() == 0 gate returns before any work, leaving their decode path byte-for-byte identical to origin/main. Decode output verified identical to origin/main across BPE, byte_fallback (Llama-3), Unigram+Metaspace (ALBERT) and WordPiece (BERT), for skip_special_tokens both true and false. (Vera figures include the aarch64 prefetch of the following commit.)
In the fused decode loop, software-prefetch the pre-decoded bytes of a token a few positions ahead so the scattered reads into the contiguous vocab_decoded blob overlap with the current token's copy, hiding memory latency on the decode hot path. aarch64-only (prfm pldl1keep), gated behind #[cfg(target_arch = "aarch64")] and thus a no-op on every other target. This patch is self-contained so it can be dropped independently if inline asm is undesirable upstream. Decode output verified byte-for-byte identical to the pre-prefetch path. cargo test --lib: 201 passed, 0 failed. Perf (single-thread decode): aarch64-only software prefetch of the scattered vocab_decoded reads on the decode hot path; contributes the aarch64 portion of the NVIDIA Vera (Olympus) byte-level figures above (~1-2%). No-op on x86.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
ArthurZucker
reviewed
Jul 6, 2026
ArthurZucker
left a comment
ArthurZucker
left a comment
Collaborator
There was a problem hiding this comment.
Ty! We actually merged the first PR related to updated vocab for BPE, which is much much more friendly towards decode.
This PR is nice but bound to be superseed so not worth to have right now. If you want to work on decode you can of course, but from #2119 ! (See vocabStore stores the full decoded string as a single slab)
| #[derive(Clone, PartialEq)] | ||
| pub(crate) struct FlatVocabDecoded { | ||
| /// All tokens' decoded bytes concatenated end-to-end. | ||
| bytes: Vec<u8>, |
Collaborator
There was a problem hiding this comment.
Suggested change
| bytes: Vec<u8>, | |
| bytes: Box<[u8]>, |
once built it never changes no?
| bytes: Vec<u8>, | ||
| /// `offsets[id]..offsets[id+1]` is the byte range for token `id`. | ||
| /// Length = vocab_size + 1 (sentinel entry at the end). | ||
| offsets: Vec<u32>, |
Contributor
Author
|
Closing this PR, #2166 is the left-over rebased on top of huggingface:feat/train_encode_split |
sebpop
added a commit
to sebpop/tokenizers
that referenced
this pull request
Jul 8, 2026
TokenizerImpl::decode allocated an owned String per token (id_to_token), collected them into a Vec<String>, then ran the decoder -- O(N) allocations per call. Add a fused fast path that borrows each token's bytes straight from the model's VocabStore slab (zero-copy id_to_token_bytes -> &[u8]) and lets the decoder concatenate + transform them into one buffer in a single pass. Rather than introduce a parallel pre-decoded structure, this reuses the existing VocabStore/BucketVocabStore byte slab (per review feedback on huggingface#2144): * Model::id_to_token_bytes(id) -> Option<&[u8]> (default None; BPE returns a borrow into its VocabStore; dispatched through ModelWrapper). * Decoder::decode_fused_bytes(&[&[u8]]) + supports_fused_decode() (default None/false); ByteLevel maps each token's chars to bytes via the CHAR_BYTES_LOOKUP array into one Vec<u8>, then a single from_utf8_lossy -- output identical to decode_chain, with no per-token String. Added/special tokens are detected with an id-range check (min added-vocab id vs max input id, one scan): if any input id is in the added vocab the fast path bails to the regular decoder route, which keeps skip_special_tokens handling and added-token content exact. Non-byte-level decoders (Metaspace, WordPiece, Sequence, ...) report supports_fused_decode()==false and are untouched. Decode output verified byte-for-byte identical to the pre-change path across byte-level BPE, byte_fallback (Llama-3), Unigram+Metaspace (ALBERT) and plain BPE, for skip_special_tokens both true and false, on round-trip decode and on decoding the entire id range. tk-encode lib tests: 264 passed. Perf (single-thread decode, one pinned core, baseline -> this series): byte-level decode llama3-en (English) llama3-ja (CJK) Nvidia Vera 40.8 -> 64.5 MiB/s 41.1 -> 181 MiB/s (1.6x / 4.4x) Intel Granite Rapids 29.7 -> 44.6 MiB/s 35.6 -> 125 MiB/s (1.5x / 3.5x) AMD Turin 46.1 -> 66.2 MiB/s 51.1 -> 180 MiB/s (1.4x / 3.5x) Non-byte-level decoders (Unigram+Metaspace / ALBERT, WordPiece / BERT) stay at parity: decode_fused returns immediately when supports_fused_decode() is false, so their path is unchanged.
sebpop
added a commit
to sebpop/tokenizers
that referenced
this pull request
Jul 9, 2026
TokenizerImpl::decode allocated an owned String per token (id_to_token), collected them into a Vec<String>, then ran the decoder -- O(N) allocations per call. Add a fused fast path that borrows each token's bytes straight from the model's VocabStore slab (zero-copy id_to_token_bytes -> &[u8]) and lets the decoder concatenate + transform them into one buffer in a single pass. Rather than introduce a parallel pre-decoded structure, this reuses the existing VocabStore/BucketVocabStore byte slab (per review feedback on huggingface#2144): * Model::id_to_token_bytes(id) -> Option<&[u8]> (default None; BPE returns a borrow into its VocabStore; dispatched through ModelWrapper). * Decoder::decode_fused_bytes(&[&[u8]]) + supports_fused_decode() (default None/false); ByteLevel maps each token's chars to bytes via the CHAR_BYTES_LOOKUP array into one Vec<u8>, then a single from_utf8_lossy -- output identical to decode_chain, with no per-token String. Added/special tokens are detected with an id-range check (min added-vocab id vs max input id, one scan): if any input id is in the added vocab the fast path bails to the regular decoder route, which keeps skip_special_tokens handling and added-token content exact. Non-byte-level decoders (Metaspace, WordPiece, Sequence, ...) report supports_fused_decode()==false and are untouched. Decode output verified byte-for-byte identical to the pre-change path across byte-level BPE, byte_fallback (Llama-3), Unigram+Metaspace (ALBERT) and plain BPE, for skip_special_tokens both true and false, on round-trip decode and on decoding the entire id range. tk-encode lib tests: 264 passed. Perf (single-thread decode, one pinned core, baseline -> this series): byte-level decode llama3-en (English) llama3-ja (CJK) Nvidia Vera 40.8 -> 64.5 MiB/s 41.1 -> 181 MiB/s (1.6x / 4.4x) Intel Granite Rapids 29.7 -> 44.6 MiB/s 35.6 -> 125 MiB/s (1.5x / 3.5x) AMD Turin 46.1 -> 66.2 MiB/s 51.1 -> 180 MiB/s (1.4x / 3.5x) Non-byte-level decoders (Unigram+Metaspace / ALBERT, WordPiece / BERT) stay at parity: decode_fused returns immediately when supports_fused_decode() is false, so their path is unchanged.
sebpop
added a commit
to sebpop/tokenizers
that referenced
this pull request
Jul 10, 2026
TokenizerImpl::decode allocated an owned String per token (id_to_token), collected them into a Vec<String>, then ran the decoder -- O(N) allocations per call. Add a fused fast path that borrows each token's bytes straight from the model's VocabStore slab (zero-copy id_to_token_bytes -> &[u8]) and lets the decoder concatenate + transform them into one buffer in a single pass. Rather than introduce a parallel pre-decoded structure, this reuses the existing VocabStore/BucketVocabStore byte slab (per review feedback on huggingface#2144): * Model::id_to_token_bytes(id) -> Option<&[u8]> (default None; BPE returns a borrow into its VocabStore; dispatched through ModelWrapper). * Decoder::decode_fused_bytes(&[&[u8]]) + supports_fused_decode() (default None/false); ByteLevel maps each token's chars to bytes via the CHAR_BYTES_LOOKUP array into one Vec<u8>, then a single from_utf8_lossy -- output identical to decode_chain, with no per-token String. Added/special tokens are detected with an id-range check (min added-vocab id vs max input id, one scan): if any input id is in the added vocab the fast path bails to the regular decoder route, which keeps skip_special_tokens handling and added-token content exact. Non-byte-level decoders (Metaspace, WordPiece, Sequence, ...) report supports_fused_decode()==false and are untouched. Decode output verified byte-for-byte identical to the pre-change path across byte-level BPE, byte_fallback (Llama-3), Unigram+Metaspace (ALBERT) and plain BPE, for skip_special_tokens both true and false, on round-trip decode and on decoding the entire id range. tk-encode lib tests: 264 passed. Perf (single-thread decode, one pinned core, baseline -> this series): byte-level decode llama3-en (English) llama3-ja (CJK) Nvidia Vera 40.8 -> 64.5 MiB/s 41.1 -> 181 MiB/s (1.6x / 4.4x) Intel Granite Rapids 29.7 -> 44.6 MiB/s 35.6 -> 125 MiB/s (1.5x / 3.5x) AMD Turin 46.1 -> 66.2 MiB/s 51.1 -> 180 MiB/s (1.4x / 3.5x) Non-byte-level decoders (Unigram+Metaspace / ALBERT, WordPiece / BERT) stay at parity: decode_fused returns immediately when supports_fused_decode() is false, so their path is unchanged.
sebpop
added a commit
to sebpop/tokenizers
that referenced
this pull request
Jul 13, 2026
TokenizerImpl::decode allocated an owned String per token (id_to_token), collected them into a Vec<String>, then ran the decoder -- O(N) allocations per call. Add a fused fast path that borrows each token's bytes straight from the model's VocabStore slab (zero-copy id_to_token_bytes -> &[u8]) and lets the decoder concatenate + transform them into one buffer in a single pass. Rather than introduce a parallel pre-decoded structure, this reuses the existing VocabStore/BucketVocabStore byte slab (per review feedback on huggingface#2144): * Model::id_to_token_bytes(id) -> Option<&[u8]> (default None; BPE returns a borrow into its VocabStore; dispatched through ModelWrapper). * Decoder::decode_fused_bytes(&[&[u8]]) + supports_fused_decode() (default None/false); ByteLevel maps each token's chars to bytes via the CHAR_BYTES_LOOKUP array into one Vec<u8>, then a single from_utf8_lossy -- output identical to decode_chain, with no per-token String. Added/special tokens are detected with an id-range check (min added-vocab id vs max input id, one scan): if any input id is in the added vocab the fast path bails to the regular decoder route, which keeps skip_special_tokens handling and added-token content exact. Non-byte-level decoders (Metaspace, WordPiece, Sequence, ...) report supports_fused_decode()==false and are untouched. Decode output verified byte-for-byte identical to the pre-change path across byte-level BPE, byte_fallback (Llama-3), Unigram+Metaspace (ALBERT) and plain BPE, for skip_special_tokens both true and false, on round-trip decode and on decoding the entire id range. tk-encode lib tests: 264 passed. Perf (single-thread decode, one pinned core, baseline -> this series): byte-level decode llama3-en (English) llama3-ja (CJK) Nvidia Vera 40.8 -> 64.5 MiB/s 41.1 -> 181 MiB/s (1.6x / 4.4x) Intel Granite Rapids 29.7 -> 44.6 MiB/s 35.6 -> 125 MiB/s (1.5x / 3.5x) AMD Turin 46.1 -> 66.2 MiB/s 51.1 -> 180 MiB/s (1.4x / 3.5x) Non-byte-level decoders (Unigram+Metaspace / ALBERT, WordPiece / BERT) stay at parity: decode_fused returns immediately when supports_fused_decode() is false, so their path is unchanged.
sebpop
added a commit
to sebpop/tokenizers
that referenced
this pull request
Jul 22, 2026
TokenizerImpl::decode allocated an owned String per token (id_to_token), collected them into a Vec<String>, then ran the decoder -- O(N) allocations per call. Add a fused fast path that borrows each token's bytes straight from the model's VocabStore slab (zero-copy id_to_token_bytes -> &[u8]) and lets the decoder concatenate + transform them into one buffer in a single pass. Rather than introduce a parallel pre-decoded structure, this reuses the existing VocabStore/BucketVocabStore byte slab (per review feedback on huggingface#2144): * Model::id_to_token_bytes(id) -> Option<&[u8]> (default None; BPE returns a borrow into its VocabStore; dispatched through ModelWrapper). * Decoder::decode_fused_bytes(&[&[u8]]) + supports_fused_decode() (default None/false); ByteLevel maps each token's chars to bytes via the CHAR_BYTES_LOOKUP array into one Vec<u8>, then a single from_utf8_lossy -- output identical to decode_chain, with no per-token String. Added/special tokens are detected with an id-range check (min added-vocab id vs max input id, one scan): if any input id is in the added vocab the fast path bails to the regular decoder route, which keeps skip_special_tokens handling and added-token content exact. Non-byte-level decoders (Metaspace, WordPiece, Sequence, ...) report supports_fused_decode()==false and are untouched. Decode output verified byte-for-byte identical to the pre-change path across byte-level BPE, byte_fallback (Llama-3), Unigram+Metaspace (ALBERT) and plain BPE, for skip_special_tokens both true and false, on round-trip decode and on decoding the entire id range. tk-encode lib tests: 264 passed. Perf (single-thread decode, one pinned core, baseline -> this series): byte-level decode llama3-en (English) llama3-ja (CJK) Nvidia Vera 40.8 -> 64.5 MiB/s 41.1 -> 181 MiB/s (1.6x / 4.4x) Intel Granite Rapids 29.7 -> 44.6 MiB/s 35.6 -> 125 MiB/s (1.5x / 3.5x) AMD Turin 46.1 -> 66.2 MiB/s 51.1 -> 180 MiB/s (1.4x / 3.5x) Non-byte-level decoders (Unigram+Metaspace / ALBERT, WordPiece / BERT) stay at parity: decode_fused returns immediately when supports_fused_decode() is false, so their path is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings de-tokenization single thread performance up on arm64 and x86_64.
The change is the sum of 4 separate patches:
Perf (single-thread decode, taskset-pinned core, origin/main -> this series):