Tokenizers v1 - #2119
Tokenizers v1#2119McPatate wants to merge 108 commits into
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
6ca91df to
4404c11
Compare
* fmt * fix ci * re-generate readmes * readme check fix * restructure rust workflow * Audit & quality for node bindings * needs to generate cargo.lock * cleanup comments * cargo install fix * iwip * commit lock files + harmonize workflows * cache build artifacts and installed binaries
* factor byte_level constants in utils * also factor BYTES_CHAR * move tests * lint
* feat: impl new pretok for bert Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * refactor: change alphabet signature Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * feat: add new whitespace pretok Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * refactor: remove unneeded code & comments Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * feat: add lookup table for ascii fast path classification Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> --------- Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
* bucket draft commit * we need a mapp then fo check which token * we know we have to sort it at the end * buckets not vec buckets and then copy byte set values We do have a bit of work to do to mach some of the smallVec features. For now I am implementing, we'll see what I am doing is most probably not super optimal (espcially having to copy from slice into a newly allocated buffer?) * box dyn ... * boxed_slice * start flattening the datastructure * update * small updates to try and compile * compiles! * fix slice len copy * nits * use generic const * this was a bad idea * create vocab store * use ptr_hash * skip some tests for compilation * nit * add id to token * update * fix * well this was ai assisted :) * default and debug * learn to stop I have to go * add match_bytes func * clippy * fix index * clippy * get vocab bytes * nits * fix * omment * fix compilation * cleanup * fix compilation * nit * fix some indexing * fixes * more fixes esp en empty entries * fix ID value * simpler debug * first implem :) * handle many buckets case * details * fix * bunch of prints! * loop was stuck * for now build splits out of the split end and start * use VocabStore in BPE model and UnigramModel * match bytes needs to return token id and token length! * poc: fast, light, allocation-free BPE encode prototype + benches A from-scratch inference-only byte-level BPE encode path and the benchmarks behind the perf investigation. 14-18x over HF tokenizers (main) on byte-exact models, ~10x smaller binary, ~3x less RAM, 0 hot-path allocations, and a 56x multi-turn re-encode prefix cache. - poc/fast-encode: final encoder + benches (stage profile, model x task sweep, parallel scaling, multi-turn prefix cache, splitter shootout). NEON-DFA split, MPHF VocabStore, allocation-free hybrid merge, thread-local pretoken cache, IREE-style ring buffer. - poc/edge-minimal: C-free, allocation-free build for on-device (0.37 MB stripped). - poc/special-token-matcher: MPHF length-probe vs IREE-style scan vs daachorse. - poc/scripts: tokenizer downloader + apply_chat_template workload generators. - poc/ENCODE_PERF_CASE.md: write-up + the centralize-in-tokenizers case. Research prototype; byte-exact on the 12 GPT-2-byte-level models tested. * where i am at * updates * fi * fix byte match * fixes * fix * nits here and there * tedious fixes * nits types.rs * small update * loads of todo in constructing the struct ± * update * just fix warning for now * add a manual test :) * fix logic, add match bytes test * more tests (esp nibble case) * fis * better func * fix the test ! * add a small todo * memchr2,3 are slow actually * bench against daachorse * big update: faster than daachorse up to 90% density of special tokens This was fun to work on! The key is that the rejection was still very slow on startswith(). This was looping and we where potentially storing too long prefix -> vs now u64 & u64 which is efficiient and fast. This is the final nail for this splitting being fast. * renamed buckets<-types and remove POC * remove dummy * unused * add rstrip and lstrip * nits * naive matcher is IREE's style match * clean * add single word * updates * remove the bleuprint * fix CI: cross-arch compile, clippy, fmt - gate nibble_match_bytes + its test to aarch64 (x86/cross-compile build was failing on a missing method) - add Unigram::is_empty (clippy len_without_is_empty) - clippy --fix: needless_return, redundant_field_names, needless_borrow, len_zero, doc continuations - cargo fmt --all Normalized-matching / add_tokens tests still fail by design (extract_and_normalize WIP). * fix tests * nit * small todos * my comments * updates * Apply suggestions from code review Co-authored-by: Luc Georges <McPatate@users.noreply.github.com> * nits here and thre * fix * skip hand rolled byte checks .... * up * add "extract_next" API * simple is word, lstrip rstrip * nits * fix them up * use merge word char * isolate added vocab * clippy --------- Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>
* lint (cherry picked from commit 2d8adb2) * fix vocabstore partialeq (cherry picked from commit 927334c) * fix doctest (cherry picked from commit 2247647) * no fail-fast in CI workflow (cherry picked from commit 6371e9c) * fix export * refactor: park #2129 fast-encode path, restore legacy Tokenizer on the base Split #2129's token store in two: a legacy map-backed `VocabStore` (`crate::vocab_store`) that backs the models on this base, and the verbatim MPHF store renamed to `BucketVocabStore`. The models are unchanged — their `use crate::vocab_store::VocabStore` now resolves to the legacy twin, so the fast store can be swapped back in by the pipeline PR via a one-line alias. Restore the pre-#2129 PreTokenizedString `AddedVocabulary` as the active tokenizer AV, and park the #2129 fast path — `BucketVocabStore`, `Buckets`, and the bucket AV (renamed `bucket_added_vocabulary`) — unwired for the pipeline PR. The parked modules are not re-exported at the crate root so they don't collide with the legacy AV. This makes the base the genuine legacy Tokenizer, a fair A/B baseline against PipelineTokenizer. Models and the tk-train trainer stay byte-identical to #2129. The only #2129 code removed: its broken `extract_and_normalize` stub (replaced by a working two-pass), the `encode_special_tokens` default (true->false), and the legacy-tokenizer wiring of the new AV (reverted). BucketVocabStore is #2129 verbatim plus the on-branch PartialEq/doctest fixes, with expanded tests. tk-encode: 219 pass / 0 fail / 2 ignored, 20 doctests, clippy clean; workspace builds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor: park #2129 fast-encode path, restore legacy Tokenizer on the base Split #2129's token store in two: a legacy map-backed `VocabStore` (`crate::vocab_store`) that backs the models on this base, and the verbatim MPHF store renamed to `BucketVocabStore`. The models are unchanged — their `use crate::vocab_store::VocabStore` now resolves to the legacy twin, so the fast store can be swapped back in by the pipeline PR via a one-line alias. Restore the pre-#2129 PreTokenizedString `AddedVocabulary` as the active tokenizer AV, and park the #2129 fast path — `BucketVocabStore`, `Buckets`, and the bucket AV (renamed `bucket_added_vocabulary`) — unwired for the pipeline PR. The parked modules are not re-exported at the crate root so they don't collide with the legacy AV. This makes the base the genuine legacy Tokenizer, a fair A/B baseline against PipelineTokenizer. The parked bucket path stays #2129 verbatim, minus dead/broken bits: - `BucketVocabStore` = #2129's store + on-branch PartialEq/doctest fixes, with expanded unit tests. - the bucket AV's `extract_and_normalize` (and its stub-only tests) is dropped — the PipelineTokenizer drives the AV solely through `extract_next` (the `PipelinePatternMatcher` trait); `extract_and_normalize` is only the legacy TokenizerImpl's entry point and uses the legacy AV. - `extract_next` had an inverted vocab selection (#2129 used `self.vocab` for normalized text); fixed to search `normalized_vocab` when normalized and `vocab` otherwise, and covered with tests (routing, offsets, lstrip/rstrip, single_word). Models and the tk-train trainer stay byte-identical to #2129. The only other #2129 code removed is the legacy-tokenizer wiring of the new AV, reverted so the legacy AV is active again. tk-encode: 216 pass / 0 fail / 1 ignored, 20 doctests, clippy + fmt clean; workspace builds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* plumbing: full pipeline * lint * extract special tokens twice * lint * ai-generated: compare pipelines * ai-generated: pipeline stage analysis * wip: no vecs for splits * ai-generated: examples update * change special token matching api * iterator formulation * rm examples * lint * rm unused * cleanup * ai-generated: bench + correctness test * inline loop + comments * ai-assisted: docstring * lint * wire PipelineTokenizer onto the parked bucket AddedVocabulary Rebased onto feat/train_encode_split, which parked the #2129 fast path (BucketVocabStore / Buckets / bucket_added_vocabulary) and restored the legacy Tokenizer as the A/B baseline. This collapses the three stale fixup commits ("rebase", "attempt to fix", "apply normalizer in legacy tokenizer") from the previous rebase — they targeted the pre-park base — into one coherent step: - bucket_added_vocabulary.rs: the final fast AddedVocabulary (Buckets-backed, extract_next / extract_and_normalize, impl PipelinePatternMatcher). - PipelineTokenizer uses the bucket AddedVocabulary; the base Tokenizer stays legacy. TryFrom<&Tokenizer> rebuilds the bucket AV from the tokenizer's added tokens in id order, so ids are preserved (model-present tokens reuse their model id) and the pipeline emits the same ids as the reference tokenizer. pipeline_oracle passes: identical ids on big.txt (English) and wagahai (Japanese) at 1kB/10kB chunks. Full tk-encode/tk-train suites green; fmt + clippy -D warnings clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * remove unused extract_and_normalize from bucket AddedVocabulary The PipelineTokenizer drives special-token matching through `extract_next` (`PipelinePatternMatcher`) + `SpecialSegmentIterator`, so the bucket AV's `extract_and_normalize` / `split_on_matches` pair is dead code (only its own test used it). Drop them and the now-unused `Range`/`PreTokenizedString`/`Token` imports. The shared helpers (`is_ws`, `is_single_word`, `skip_whitespace_*`) stay — `extract_next` uses them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * remove extract_two_pass test harness and its tests from bucket AV Drop the test-only `extract_segments` / `extract_two_pass` / `owned` helpers and the 7 extraction tests built on them. `extract_two_pass` reimplemented `PipelineTokenizer::encode`'s two-pass loop inside the test module (a drift-prone duplicate); `extract_next`'s matching behavior is exercised end-to-end by pipeline_oracle. Kept the AV-level unit tests that don't go through the harness: can_add_tokens, can_add_special_tokens, normalized_tokens_are_stored_by_normalized_form. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test bucket AV extract_next through SpecialSegmentIterator Restore the extract_next coverage dropped with the extract_two_pass harness, but drive the real `SpecialSegmentIterator` instead of a parallel reimplementation. Five single-pass tests over a bucket `AddedVocabulary`: raw added-token carving, single_word, lstrip/rstrip span absorption, the encode_special_tokens toggle, and raw-vs-normalized matcher selection. `SpecialSegmentIterator::new` is now pub(crate) so the AV tests can construct it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* update makefile * ai-generated: comparative fixture bench + CI Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ai-generated: hardware + timestamp + revision --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ons (#2145) Move the Tokenizer-vs-PipelineTokenizer report out of a PR comment and into a marker-delimited section of the PR description, kept current in place (upsert_pr_section.py). Pushes to feat/train_encode_split now target PR #2119; /pipeline-bench dispatch targets its own PR. Add emoji feedback to the comment-triggered flow: 👀 when the trigger fires, 👍 from the bench workflow on success. Add a cancel-in-progress concurrency group keyed on github.ref so a newer commit/comment supersedes an in-flight run for the same target without cancelling other PRs' benches. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comparative Tokenizer-vs-PipelineTokenizer bench that maintains a
marker-delimited section (graph + table) in the target PR's description,
via .github/scripts/upsert_pr_section.py (replace-in-place, else append).
Triggering: issue_comment workflows only run from the default branch, so a
"/pipeline-bench" comment can't fire before this lands on main. Use a
pull_request:[labeled] trigger ("run-pipeline-bench") instead — it runs the
PR branch's own workflow, so it works pre-main. The run itself is the PR
check (no manual check-run API, no emoji reactions), the label is auto-removed
so re-adding re-runs, and the target PR is resolved from the event (labeled PR
/ dispatch input / #2119 for pushes to feat/train_encode_split). Same-repo PRs
only — fork PRs get no secrets and a read-only token.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comparative Tokenizer-vs-PipelineTokenizer bench that maintains a
marker-delimited section (graph + table) in the target PR's description,
via .github/scripts/upsert_pr_section.py (replace-in-place, else append).
Triggering: issue_comment workflows only run from the default branch, so a
"/pipeline-bench" comment can't fire before this lands on main. Use a
pull_request:[labeled] trigger ("run-pipeline-bench") instead — it runs the
PR branch's own workflow, so it works pre-main. The run itself is the PR
check (no manual check-run API, no emoji reactions), the label is auto-removed
so re-adding re-runs, and the target PR is resolved from the event (labeled PR
/ dispatch input / #2119 for pushes to feat/train_encode_split). Same-repo PRs
only — fork PRs get no secrets and a read-only token.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comparative Tokenizer-vs-PipelineTokenizer bench that maintains a
marker-delimited section (graph + table) in the target PR's description,
via .github/scripts/upsert_pr_section.py (replace-in-place, else append).
Triggering: issue_comment workflows only run from the default branch, so a
"/pipeline-bench" comment can't fire before this lands on main. Use a
pull_request:[labeled] trigger ("run-pipeline-bench") instead — it runs the
PR branch's own workflow, so it works pre-main. The run itself is the PR
check (no manual check-run API, no emoji reactions), the label is auto-removed
so re-adding re-runs, and the target PR is resolved from the event (labeled PR
/ dispatch input / #2119 for pushes to feat/train_encode_split). Same-repo PRs
only — fork PRs get no secrets and a read-only token.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ai-geenrated: render script grid * ai-geenrated: update GH workflow * ai-geenrated: run bench on several different tokenizers * ai-generated: restore full info + full tokenizers fixtures
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
* feat: `impl pipeline::PreTokenizer for FixedLength` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * feat: add `FixedLength` to `PipelinePreTokenizer` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> --------- Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
* feat: `impl pipeline::PreTokenizer for Digits` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * feat: add `Digits` to `PipelinePreTokenizer` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * feat: add extra test for a string of digits Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> --------- Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
* feat: `impl pipeline::PreTokenizer for CharDelimiterSplit` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * feat: add `Delimiter` to `PipelinePreTokenizer` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> --------- Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
* feat: `impl pipeline::PreTokenizer for UnicodeScripts` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * feat: extend ascii lut table to basic multilingual plane Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * feat: add `UnicodeScripts` to `PipelinePreTokenizer` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> --------- Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
* feat: `impl pipeline::PreTokenizer for WhitespaceSplit` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * feat: add `WhitespaceSplit` to PipelinePreTokenizer` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> --------- Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
* feat: handle `Merge*` variants of the `SplitDelimiterBehavior` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> * feat: add `Punctuation` to `PipelinePreTokenizer` Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com> --------- Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
* Implement Cow-based normalizers * ai-generated: bert normalizer * lint * implement precompiled * fallible + more nocopy * DRY * ai-assisted: rewrite bert hotpath
…ces (#2281) * feat(node): expose the pipeline encode path, and delete what it replaces Rebase of #2281 onto the current integration branch. The original branch carried 21 commits, but only one of them was the binding: the other twenty were the POC encode path -- bitsplit, the WordCache, the SWAR sidecar -- which has since landed by other routes. Replaying them would have been twenty conflicts to arrive back where the tree already is, so this takes the napi commit alone and ports it forward. Porting it forward meant most of this diff. `#2281` loaded through `tk::Tokenizer::from_file` and `Pipeline::try_from(&tok)`, and neither exists any more: since #2352 `tk-encode` cannot read a `tokenizer.json` at all -- the reader is `tk-serialize` and the legacy "1.0" -> canonical "2.0" upgrade is `tk-convert`. So `fromFile` now runs the file through `canonicalize_file` and hands the result to `from_json`, which is also what keeps the tokenizers already on disk loading. The rest of the binding had stopped compiling. `decoders`, `models`, `normalizers`, `pre_tokenizers`, `processors`, `tokenizer`, `trainers`, `encoding` and the async `tasks` wrapped the pre-v1 engine, and they import `DecoderWrapper`, `PostProcessorWrapper`, `BpeBuilder`, `TrainerWrapper`, `Trainable` and `NormalizedString` -- all removed with that engine. `cargo build` in `bindings/node` fails on the base commit with seven unresolved-import errors before it reaches anything else. Porting wrappers for an engine that is gone is not worth doing, so they are deleted along with the JS suite that tested them. That is what unblocks CI. `.github/workflows/node.yml` carried a `branches-ignore: feat/train_encode_split` whose comment says exactly this -- "195 references to wrapper types that no longer exist" -- so the filter goes with them. Dependencies, measured with `cargo tree` before and after: 107 unique crates -> 103. Six leave (`tokenizers`, and the `indicatif` / `console` / `unicode-width` / `unit-prefix` progress-bar chain the trainers pulled), `tk-serialize` joins. Modest, because the umbrella crate is already a thin shell over `tk-encode` + `tk-convert`; the real reduction is 4423 lines of binding that no longer had an engine underneath. `serde` stays -- `ptr_hash` depends on it unconditionally, and that is the last one. One bug caught while slimming: `default-features = false` on `tk-serialize` compiles out `deserialize`, which *is* the reader, and the component list has to mirror `tk-encode`'s or a config naming a component compiled out of one but not the other fails at load rather than at build. Both are spelled out in the manifest. `test/pipeline.test.ts` replaces the deleted suite: it checks `fromFile` reads a legacy config, that ids marshal as a `Uint32Array` rather than a boxed JS `Array` (the whole reason for the return type -- one napi value per token costs 13x the encode on token-dense input), that `encodeBytesInto` agrees with `encode` id for id, that it rejects an undersized buffer, and that `addSpecialTokens` is honoured. Run against the built dylib via `NAPI_RS_NATIVE_LIBRARY_PATH`: 5 passed. `cargo build --release` and `cargo clippy` are both clean. * chore(node): drop the unexposed VERSION constant `pub const VERSION` was not `#[napi]`, so it never reached JS -- it was a Rust constant nothing on either side of the boundary read. The binding is now the one class and its three methods, and nothing else.
* tk-encode: drop dead InputSequence/EncodeInput enums Neither type has any reference outside their own definitions in tokenizer/mod.rs; the encode path takes strings and slices directly. Confirmed with a workspace-wide grep and a full workspace build. * tk-encode: drop dead BertProcessing/RobertaProcessing structs Nothing in tk-encode, tk-serialize, tk-convert or tk-train constructs either type; tk-serialize only matches the "BertProcessing" and "RobertaProcessing" strings as JSON tags, it never builds these structs. Confirmed with a workspace-wide grep and a full workspace build.
…nizers into feat/train_encode_split
`panic_immediate_abort` is no longer a `-Z build-std-features` value. It is a
real panic strategy now, and the old spelling is a hard error:
error: panic_immediate_abort is now a real panic strategy! Enable it with
`panic = "immediate-abort"` in Cargo.toml, or with the compiler flags
`-Zunstable-options -Cpanic=immediate-abort`.
So `make slimest` only built on a nightly older than the rename. Pass the
strategy as a codegen flag through RUSTFLAGS instead.
Deliberately not `panic = "immediate-abort"` in `[profile.slimest]`: a profile
value stable cargo cannot parse would break every cargo command in the
workspace (`make build`, `make test`, `cargo metadata`, CI on stable), not
just this target.
Also makes the toolchain overridable (`NIGHTLY ?= nightly`), since the failure
this guards against is a stale default alias rather than a missing one.
BPE-only gzipped size (`--no-default-features --features bpe,deserialize`,
aarch64-apple-darwin):
slim minsize, stable 332,799 B
slimest old flag, nightly 1.89 201,047 B
slimest this change 190,690 B -42.7% vs slim
The extra 10,357 B over the old flag path is the newer compiler, not the flag.
* use makefile to get fixtures * chore(makefile): group fixture fetching, trim orphaned resources The fixture-fetch machinery (DATA_DIR/HF_REVISION/the download rule) sat at the bottom of the file while every target above it already depended on it, so reading top to bottom hit "how do I get this file" before "here's where files come from". Moved it to the top as one `---- fixtures ----` block, then grouped the rest into dev loop, aggregate checks, opt-in size profiling, and bench. ORACLE_RESOURCES built on SHARED_RESOURCES, so `make oracle` pulled in gpt2-vocab.json/big.txt/llama-3-tokenizer.json etc. that oracle.rs never reads. Named its four fixtures directly instead. Dropped gpt2-merges.txt, bert-base-uncased-vocab.txt, and small.txt from SHARED_RESOURCES: grepped every .rs file in the tree (workspace members and tk-train) and found no reader for any of the three. gpt2-vocab.json, unigram.json, roberta.json, tokenizer-wiki.json, and bert-wiki.json look similarly unreferenced by name, but stay: convert.rs's and tk-serialize's tests scan data/ for *.json opportunistically rather than naming files, so those are actually exercised, just not by grep. * chore(makefile): drop the unused multilingual fixtures corpus FIXTURE_LANGS/FIXTURE_MODALITIES/FIXTURES_RESOURCES and the `fixtures` target fetched 9 language + 6 modality corpora (~15 files) and the standalone agentic-traces.txt in TESTS_RESOURCES, but nothing in this repo reads any of them: their consumers, pipeline_benchmark.rs and pipeline_oracle.rs, are gone from the tree. Confirmed with a repo-wide grep (all crates, all file types) before removing. If a future oracle or bench needs multilingual corpora again, re-add them alongside the code that reads them. * chore(tk-convert): one oracle test per model, clear message on a missing fixture matches_the_released_crate looped over all four models in one test: a failure on any one of them hid whether the others passed, and cargo test couldn't target a single model. Split into one #[test] per model (gpt2, bert_base_uncased, t5_base, albert_base_v1) calling a shared assert_matches_released helper. canonicalize_file's IO error already distinguishes "no such file" from a real canonicalisation refusal (ConvertError::Io vs. everything else). Match on it to panic with a fixture-specific message pointing at `make oracle` and hf-internal-testing/tokenizers-test-data, instead of the generic "this pass refuses it" a missing file used to produce. * ci(rust): route the oracle step through make oracle The step ran `cargo test -p tk-convert --features bench-baseline --test oracle` directly, bypassing the Makefile. That was fine while oracle.rs fetched every model live from the Hub, but now it reads fixtures from data/ (see the Makefile/oracle.rs changes upstream of this commit), and `make test` above it only fetches TESTS_RESOURCES, which doesn't include bert-base-uncased.json or t5-base.json. Left as-is, this step would fail on a cold fixture cache. Route it through `make oracle` instead, matching the `make test` step right above it, so it always fetches exactly what oracle.rs currently needs. Also fixed the step's comment: it said "both oracles ... 16 tests", left over from before pipeline_oracle.rs/pipeline_decode_oracle.rs were consolidated into tests/oracle.rs's single (now four, one per model) test. * docs: fix the oracle test section in CONTRIBUTING.md Two things went stale here from earlier commits in this branch: - \`make fixtures\` no longer exists (the target and the unused multilingual corpus behind it were removed); the fixture-fetch step it was calling out is now just \`make oracle\`, which fetches its own fixtures and runs the tests in one command, like \`make test\`/ \`make bench\` already do. - The test is one per model now (gpt2, bert_base_uncased, t5_base, albert_base_v1), so added an example of filtering to one. Also: "the in-tree Tokenizer, which is being retired" was already inaccurate before this branch, independent of the oracle work -- rc0's own lib.rs doc says plainly that object model isn't shipped in this release, not that it's in progress. Pointed at REQUIRED_FOR_V1.md instead of asserting a status. * chore: add meta-llama/Llama-3.2-1B back to the oracle, bump HF_REVISION hf-internal-testing/tokenizers-test-data#10 merged (mirrors the gated model's tokenizer.json as fixtures/models/llama-3.2-1b.json, verified byte-identical to the real one). Bumped HF_REVISION to that merge commit (54d012f1) everywhere it's pinned: the Makefile, and the two hardcoded fixture-download steps in rust.yml (Windows lib-tests, feature-matrix) that don't go through the Makefile. Added the fifth oracle test, llama_3_2_1b, alongside the other four. Updated CONTRIBUTING.md's model list to match.
…2378) * perf(pipeline): decide the single-sequence serial fallback before planning `parallel::encode` gated on `inputs.size_bytes() < PARALLEL_MIN_BYTES`, then planned, then bailed to `encode_serial` when planning produced fewer than two tasks. For one document between `PARALLEL_MIN_BYTES` and twice it that bail is guaranteed: `plan_sequence` short-circuits any sequence under `2 * PARALLEL_MIN_BYTES` into a single chunk, one chunk makes one task, and one task fails the check. So the allocations were provably discarded, and the condition is knowable from the length alone. No measurable throughput effect, and it is worth recording why rather than implying one. Because `plan_sequence` short-circuits *before* its added-token scan, the only waste in that window is about six small `Vec` allocations, against ~22 us of encoding for a 9 kB document. Measured with single-document encodes at 4 kB / 9 kB / 20 kB (the first and last as controls, since the guard cannot fire there): every cell within +-1.5%, including the 9 kB window itself. Kept because it removes work that cannot ever be used, not because it is faster. The expensive version of this waste is at 16 kB and above, where `plan_sequence` does run the full `SpecialSegmentIterator` pass over the document, finds no special token, emits one chunk and discards it -- a duplicated added-token scan. This guard does not cover that, and it cannot cheaply: whether a sequence splits is only knowable from that scan. The real fix there is the intra-sequence splitting #2331 left as follow-up, which would make the planning useful instead of merely cheap to skip. * perf(pipeline): order chunks by size class with a counting sort `plan_work` sorted the chunk list with `sort_unstable_by_key(Reverse(len))` and then folded it a second time for the per-sequence `chunk_count`. Two things are wrong with that shape: - `SequenceChunk` is 40 bytes, so a comparison sort does O(n log n) moves of 40 bytes each. The container was never the problem -- `chunks` is a `Vec`, access was already O(1) -- what it sorts is. - It runs in the serial prologue, on the calling thread, before a single worker is spawned. It is pure Amdahl fraction, and worst exactly in the many-small-inputs case tasks were introduced for in #2338: one chunk per sequence means n is the batch size, so an 80k-document batch sorts 80k structs before any encoding starts. The grouping it feeds never needs a total order -- it walks the chunks accumulating bytes and cuts a task every `PARALLEL_MIN_BYTES`. Ordering by power-of-two size class is sufficient, and that is a counting sort: histogram, exclusive prefix sums largest-class-first, scatter. O(n), moving 4-byte indices instead of 40-byte structs, with `chunk_count` coming off the same histogram pass rather than a second fold. `tasks` now index `order` rather than `chunks`, so the worker pays one extra `u32` load per chunk -- against at least `PARALLEL_MIN_BYTES` of tokenizing per task. Batched `encode` of a 5 MB corpus, min of 5 passes with the corpus repeated to fill 250 ms per pass, 3 interleaved rounds, separately built binaries: 64 B documents n_docs base MB/s this commit gpt2/chinese 80457 362.7 +17.8% llama-3/chinese 80457 344.4 +11.0% llama-3/code 32691 709.0 +6.8% gpt2/english 81983 668.4 +4.0% gpt2/code 32691 683.9 +2.9% llama-3/english 81983 663.8 +1.5% 1024 B documents n_docs base MB/s this commit llama-3/chinese 5118 600.9 +2.1% gpt2/chinese 5118 846.5 +1.8% others -1.0% .. +0.9% Positive in every cell where n is large and gone by 1 kB documents, which is the shape an O(n log n) -> O(n) change in a serial prefix should have. The noise floor for this setup is about +-4%, measured from a commit that cannot affect the batch path at all, so the two chinese cells are the solid ones and the sub-3% entries are marginal. Token ids are unaffected: chunks are encoded independently and reassembled by `(seq, idx)`, so visiting order cannot change the result. Verified identical across all four builds in every cell. Only the task boundaries move. * perf(pipeline): partition the chunks rather than counting-sort them Replaces the counting sort from the previous commit with a single partition pass, on review against #2361 which reached the same conclusion independently and expressed it in eight lines. The task grouping never needed the ordering the counting sort provided. Tasks are byte-balanced -- each grown until it holds `PARALLEL_MIN_BYTES` -- so how the chunks below that threshold sit among themselves cannot change a task's size. Exactly one property matters: a chunk big enough to be a task on its own must not be picked last and become the straggler the batch waits on. Partitioning those to the front is O(n) in place, and drops the `Vec<u32>` permutation, the `order` field, the size-class table, and the extra load per chunk in `encode_task`. Same measurement as before -- batched `encode`, 64 B documents, corpus repeated to fill 250 ms per pass, 3 interleaved rounds -- shows the two are equivalent in throughput, so the precision the counting sort bought was not worth its code: counting sort partition gpt2/chinese +17.8% +16.9% llama-3/chinese +11.0% +12.9% llama-3/code +6.8% +6.3% llama-3/english +1.5% +5.4% gpt2/code +2.9% +3.0% gpt2/english +4.0% +2.6% Net effect on this PR: 152 insertions become 88, and the worker's inner loop goes back to indexing `chunks` directly. * Apply suggestion from @McPatate Co-authored-by: Luc Georges <McPatate@users.noreply.github.com> * Apply suggestion from @McPatate Co-authored-by: Luc Georges <McPatate@users.noreply.github.com> * Apply suggestion from @ArthurZucker --------- Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>
* implement padding * axe legacy pad() * padding can be a property of the tokenizer * read padding from json config * fmt * fix * to_json * allow disabling padding * test attention mask, type_ids, sentence_transformers configs * rm maybe_par_iter * test(convert): oracle the two sentence-transformers configs with truncation and padding all-MiniLM-L6-v2 and all-mpnet-base-v2 ship truncation (max_length 128) and padding (Fixed 128) in tokenizer.json, so they are where the padded attention mask and type ids are compared against the released crate. Their fixtures are pinned at hf-internal-testing/tokenizers-test-data 943303b (#11), one commit past the previous pin, so no json_oracle digest changes.
* perf(pipeline): reuse sequence A's buffer in post_process Extracted from #2361, which bundled it with the flat batch path; it is an independent change to the general path and reviews better on its own. The `batch_alloc` example the original commit added is left out: it imports `tk_encode::Tokenizer`, which #2352 removed, so it no longer compiles. #2361 deleted it two commits later for the same reason. * lol * remove bloat * Apply suggestion from @ArthurZucker * move post processor code where it belongs * more cleanup * Apply batched suggestions from code review Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * remove stuff that belongs in convert * update * nit * cleanup again * fix * fromat * nit * update * remove bloat shit * nit * Apply batched suggestions from code review Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * weave is ai * simple nit * fix(deps): bump chacha20 to 0.10.2, the yanked 0.10.1 fails cargo audit chacha20 0.10.1 and 0.10.0 are both yanked from crates.io. The audit jobs run `cargo audit -D warnings`, and a yanked crate is a denied warning, so the node audit failed. The `--ignore RUSTSEC-*` flags cannot suppress it: "yanked" is an index state, not an advisory ID. Pulled in transitively by rand 0.10.2. tokenizers/Cargo.lock was already on 0.10.2; the node and python lockfiles were missed. Verified with the exact CI command against all three lockfiles: node exit 0 python exit 0 (was 1: "1 denied warning found") tokenizers exit 0
* perf(pipeline): give each thread its own scratch sub-pool
`ScratchPool` is one `Mutex<Vec<EncodeScratch>>` and every `encode` locks it
twice: once to pop a scratch, once to push it back when `ScratchGuard` drops.
Every encoding thread therefore serialises on one lock, which is what the note
in the source predicted:
TODO @McPatate : The Mutex can create contention, to be replaced by a
better access pattern
Measured with N threads encoding a 12 kB prompt of 30k distinct words through
one shared `PipelineTokenizer`, throughput stops improving between 16 and 32
threads and then falls back (encodes/s):
threads Intel GNR 128c NVIDIA Vera 88c AMD Turin 96c
1 52,474 67,399 56,076
8 205,768 308,619 408,580
16 298,086 327,975 640,830
32 226,496 609,208 675,155
64 191,749 475,348 668,325
full 223,578 314,715 661,471
Scaling efficiency at full width is 3.3%, 5.3% and 12.3% of linear.
This gives each thread one of 64 sub-pools, picked once per thread from a
counter and kept in a thread-local. A scratch returns to the sub-pool it came
from, so a thread reuses its own buffers and the `WordCache` inside each scratch
stays warm per thread instead of being passed around. A thread whose own
sub-pool is empty takes from another before building -- `try_lock`, so it skips
a busy sub-pool rather than queueing behind it, which would reintroduce what the
sharding removes. That keeps the property the pool exists for: it hands out what
it has rather than allocating alongside it. `keeps_at_most_one_scratch_per_
concurrent_holder` is what makes this non-optional; without the borrow it fails.
Same machines, same inputs, with the sharded pool:
threads Intel GNR 128c NVIDIA Vera 88c AMD Turin 96c
1 52,557 67,927 56,054
8 401,004 538,663 438,338
16 798,554 1,074,139 871,056
32 1,568,314 2,135,057 1,698,626
64 2,858,405 4,265,264 2,832,353
full 4,623,304 4,904,611 3,719,211
20.7x, 15.6x and 5.6x at full width, at 69%, 82% and 69% efficiency. Single
thread is unchanged on all three (1.00x), so the serial path pays nothing.
Measured with the `scratch_scaling` example added in this series, 5 s per point,
on otherwise idle machines. `cargo test -p tk-encode` goes from 131 to 132
tests passing, the new one being the borrow path above.
Reproducibility differs by arm. Three sweeps on the same idle GNR node read
4.62M, 4.80M and 4.94M encodes/s at 128 threads with the sharded pool, against
223,578, 245,339 and 539,356 with the single mutex: a contended lock is sensitive
to where the scheduler puts the threads, and 128 threads have 512 of them to
choose from there. The full-width multiplier on that machine is therefore 9x to
21x depending on which baseline sample it is divided by. The sharded arm is the
reproducible one.
One caveat on scope: an HTTP serving frontend that encodes one request at a time
per connection was measured before and after this change and is unaffected
(0.88-0.98x). Its 60 kB prompts cost about five of the 12 kB encodes tabulated
above, so at 8,500 requests/s it asks the pool for roughly 43,000 encodes/s,
against the unsharded pool's peak of 609,208 on the same machine -- 7%, short of
where this mutex begins to bind. The win here is for callers driving `encode`
from many threads at once.
* example: measure how encode scales with thread count
N threads calling `encode` on one shared `PipelineTokenizer` for a fixed wall
time, reporting encodes/s, speedup and scaling efficiency per thread count. This
is what the preceding commit's numbers come from, and what makes a claim about
pool contention checkable rather than argued.
In tk-convert because loading a released `tokenizer.json` now needs both
`tk_convert::canonicalize_file` and `tk_serialize::from_json`, and this is the
crate that has both.
Vocabulary size is a knob for a reason. The scratch holds a `WordCache`, so a
prompt cycling through a handful of words is a near-100% cache hit: absolute
throughput is inflated, and so is the share of each encode spent on the pool
lock, which is the quantity being measured. `--vocab 30000` is nearer natural
text; `--vocab 15` reproduces the artefact deliberately.
* bench: move the scaling sweep to tk-serialize
Review feedback: a bench, not an example with its own arg parser. Thread counts
from available_parallelism, prompts from big.txt. Two rows: distinct text and a
repeated line, since a WordCache hit is a real regime, not a void measurement.
Drops the Dynamo notes and the TODO this PR fixes.
---------
Co-authored-by: Arthur <arthur.zucker@gmail.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* metaspace prepend first * serialization * fix + renames * fix + support mistral * fix(vocab): is_special_token checks the normalized bucket too is_special_token only looked up self.vocab (the raw-form matcher), so an added token with special: true and normalized: true (e.g. mpnet's <unk>, distinct from the model's own [UNK]) was never recognised as special. decode(ids, skip_special_tokens=true) then failed to drop it. Found while investigating an oracle.rs divergence surfaced by rebasing feat/metaspace-prepend-first: sentence-transformers/all-mpnet-base-v2 decoding "<unk>x" with skip=true diverged from the released crate. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * refactor(normalizers): pass the chunk's byte offset instead of a sequence-start flag Normalizer::normalize takes the byte offset of its input within the sequence. Metaspace with prepend: "first" acts when it is 0; the parallel encoder hands over chunk.range.start; every other normalizer ignores it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * bump revision in makefile * skip oracle tests that require truncatin --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* wrap encode options in a struct * lint * ai: update node bindings
#2178) `CHAR_BYTES_LOOKUP` was an `AHashMap<char, u8>` holding the 256-entry inverse of `BYTES_CHAR_LOOKUP`. `make_byte_char_lookup` only ever emits codepoints below 324 -- printable bytes map to themselves and the other 68 are shifted to `256..=323` -- so the inverse fits in a fixed `[Option<u8>; 512]` indexed by the codepoint, with no hashing and no probe. Read it through the new `char_to_byte`, which handles the out-of-range case, so the call sites stop repeating the lookup dance. All four are per-character over whole vocabularies or whole token streams: * `transform_vocab`, through `reverse_lookup`: every char of every vocab entry when a byte-level model is built. * `ByteLevelFold::fold` and `boundary_merge_ranks`: every char of every vocab entry again, also at build time. * `ByteLevelDecoder::decode_chain`: every char of every token, for a model whose vocab store does not already hold pre-decoded bytes. Three tests pin the invariant the unguarded indexing rests on. A future change to `make_byte_char_lookup` that moved the alphabet past 512 would otherwise panic on first use of the table, and guarding the write instead would silently drop the entry and decode wrongly, which is worse. Representation change only: decode output is byte-for-byte identical. Measured on three machines, each an exclusive node with the work pinned to one CPU. Both arms are built from the same tree and run round-robin rather than in sequence, so thermal or neighbor drift cannot land entirely on one of them. Change in `decode_chain` throughput, head vs base, median of 3 rounds: CPU arch English Japanese AMD EPYC 9655P, 96 core x86_64 +8.1% +36.6% Intel Xeon Platinum 8568CXL x86_64 +7.1% +33.4% NVIDIA Grace, Neoverse-V2 aarch64 +16.7% +29.3% `pipeline_decode` over the same corpora never reaches this code on these models, and it stayed within 2.1% on every machine; that is the run-to-run floor the gains above have to clear. Model load, which walks the table once per vocab entry, gained 0.6 to 3.6%. Japanese gains about twice what English does, which is what the change predicts: the removed probe is per character of the token string, and a Japanese codepoint is three UTF-8 bytes and therefore three byte-level characters to look up, against roughly one for English.
* 🐛 Fix inline vocab key packing on big-endian targets Signed-off-by: gkumbhat <Gaurav.Kumbhat@ibm.com> * :white_check_marks: Add unit test for key and hash function Signed-off-by: gkumbhat <Gaurav.Kumbhat@ibm.com> --------- Signed-off-by: gkumbhat <Gaurav.Kumbhat@ibm.com>
* wip: new python bindings * document copy on access * ai: return np.ndarray from encoding + mechanical improvements * partialeq + hash on padding arguments * improve * move python_v2 -> python * update Python CI * add free-threaded python back * multiprocess + ft tests * bump version + rm comments * lint * error out when the Padding Mutex is poisonned * fixes * skip pickle tests for now * Encoding: support both native python list and numpy array as output * wip: pickling * mutliprocess v2 * re-add stub-gen bin * Add from_pretrained * refactor tests * test padding left * materialize padding in the Rust side * lint * ignore ty * padding and options * lint * add a header in the generated python stubs * fix(ci): harden workflow files flagged on #2391 (#2406) Co-authored-by: hf-security-analysis[bot] <265538906+hf-security-analysis[bot]@users.noreply.github.com> * rust only, no python source code * clippy go * lint --------- Co-authored-by: hf-security-analysis[bot] <265538906+hf-security-analysis[bot]@users.noreply.github.com>
🚨 Workflow security review — 16 critical, 20 high, 35 medium, 17 low, 3 informationalScanned 10 workflow file(s) with zizmor, pinact, OSV/GHSA, Claude, in full.
|
6 participants
This PR introduces
tokenizers v1:TLDR: