feat: size chunks in tokens — exact counting via the model's own BPE tokenizer - #262
Conversation
The voyage provider had no tokenizer, so it guessed token cost as len(bytes)/2
and lived with the consequences: batches sized against a guess, a TPM throttle
metering phantom tokens, and over-long inputs cut into byte windows whose
vectors were then averaged. This adds a pure-Go, count-only implementation of
the model's actual pipeline and wires it in behind an interface, so the
guessing path becomes the fallback rather than the only option.
WHAT THE HEURISTIC ACTUALLY COST, measured against Voyage's own
usage.total_tokens (412 single-input requests) and against the HuggingFace
Rust tokenizer as oracle (20k real chunks from a 45-repo corpus):
- overestimates by 1.94x on average, so the TPM token bucket subtracts
roughly twice the tokens actually spent. At the observed indexing rate
(128 chunks/s, mean 221 tokens/chunk) real usage was ~1.7M TPM while the
limiter believed 3.3M and throttled against a 3M budget — the run was
self-limited to about half the tier's real capacity.
- AND still undercounts 0.5% of chunks, worst case -41%, which is the
direction that ships an over-limit POST. The safety the constant was
chosen for is not actually delivered.
- the "~1.4 bytes/token worst case" in the old comment is optimistic;
measured worst is ~1.18.
WHY A HAND-ROLLED TOKENIZER RATHER THAN A LIBRARY. Both pure-Go candidates
were evaluated against the same oracle on the same corpus:
- sugarme/tokenizer cannot load this tokenizer.json at all. It panics in
regexp.MustCompile: the Qwen2 pre-tokenizer pattern contains \s+(?!\S),
and Go's RE2 has no lookahead.
- the ollama tree's pure-Go tokenizer knows about that and rewrites the
pattern, but its compensation only covers space indentation. Tabs take a
different branch, so it mismatches HF on 8.19% of real chunks (30.2% of
chunks containing a tab), 1471 of them undercounts, worst -16.7% on
tab-indented JSON. It also skips the declared NFC normalizer, so
decomposed input costs an extra token.
bpecount instead implements the seven alternation branches by hand with Perl
leftmost-first semantics, which is what makes the whitespace precedence come
out right. Validated at 0 mismatches against the HF oracle across 70,412
inputs (412 Voyage-verified, 20k real chunks, 50k fuzz). It needs no new
module — golang.org/x/text was already an indirect dependency — and counts
27.5k chunks/s single-threaded, which is orders of magnitude above any
indexing rate on the 2-vCPU box this targets.
Only the merges array is read; the vocabulary is not needed for a count. New
pre-tokens are merged through a linked list plus a rank heap rather than the
naive pair rescan, because minified and generated files are real: a 30 KB run
of one character took 9s the naive way and takes ~16ms this way.
WHAT CHANGES FOR CALLERS. tokenizer.Budget is one interface, not a mandatory
counter plus an optional splitter, so the chunker takes a single constructor
argument. ExactCounts() is how a caller learns whether it may size to the
limit or must keep the old margin. The doc comment records the cost asymmetry
that makes two methods worth having: both do the same single pass over the
same memo, but CountTokens allocates nothing and runs per chunk (1.9M times on
the reference corpus) while SplitPoints builds an offsets slice and runs only
when an input does not fit (5 times on that same corpus).
SplitPoints is exact, not iterative. Because Split runs with Isolated
behaviour and BPE is applied per pre-token, merges never cross a pre-token
boundary, so token counts are additive over pre-tokens and a single
left-to-right pass yields cut points whose pieces provably sum to the whole.
Only a single pre-token larger than the budget needs a search, and it gets a
binary search on bytes inside that one pre-token.
Batch cap goes 80K -> 115K when counts are exact: the 40K of headroom existed
to absorb the heuristic's undercount against Voyage's 120K hard limit, not to
absorb anything on Voyage's side. An operator's explicit MaxTokensPerRequest
still wins over both.
MEASURED EFFECT, so nobody expects the wrong thing. Simulating both packers
over 200k real chunks: 1724 requests today vs 1568 with exact counts — 1.10x,
not the 2x the 1.94x inflation suggests. Batches are bound by the 128-input
cap of voyage-code-*, not by tokens (mean chunk is 221 tokens, so 128 inputs
is ~28K against an 80K cap). The throughput win is in the TPM throttle, not in
packing; the packing win is real but small.
DELIBERATELY NOT IN THIS CHANGE. The chunker does not yet consume Budget —
splitOversizeInput still byte-windows on the provider side, and only the
provider-side counting is switched over. That integration is the follow-up
this interface exists for, and it is where averaging over byte windows finally
goes away. On the reference corpus 5 chunks in 1.9M exceeded the old 30 KB
threshold, so the damage is bounded and self-healing: those files re-embed
correctly the next time they change, and can be found with
SELECT file_path FROM vector_contents WHERE LENGTH(content) > 30000.
TESTING. The golden counts need the real 7 MB tokenizer.json, which is not in
the repo, so those tests skip on a clean checkout — CI coverage comes from a
synthetic merge table that exercises the splitter, the merge loop and the
additivity property SplitPoints depends on. Packaging that file (embed vs
fetch-and-cache; only ~3.5 MB of it is load-bearing) is still open, which is
why the path is an operator-set config field for now and an unreadable path
degrades to the estimate with a warning rather than failing to start.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The chunker's size limit was 1500*3 bytes — a token target expressed in the only unit it had. That ratio holds for dense ASCII code and misses everywhere else: a comment block in Cyrillic or CJK costs two to three bytes per character, so a byte-capped chunk carries a third of the tokens intended, while minified JavaScript packs several times more. The chunker now asks the active embedding provider what a chunk actually costs, when that provider can answer. Wiring: embeddings.Service.TokenBudget() hands out the live provider when it implements tokenizer.Budget, nil otherwise. The indexer asks PER FILE rather than caching it, because a provider swap between files is legitimate while mixing two models' limits inside one file's chunk set is not. CIX_MAX_CHUNK_TOKENS finally does something: it was parsed into config and only ever used to size llama's context. The bound is applied ABOVE the chunking paths, not inside the tree-sitter one. That placement is the point: minified files and files with no grammar fall through to chunkFallback's sliding window, and those are precisely the inputs that blow the model's input limit. A budget that only covered the tree-sitter path would have missed exactly the cases it exists for — the first draft did sit inside that path, and the two tests that now cover minified JS and a grammarless file both failed against it. With a budget in hand the byte cap is deliberately raised out of the way (innerMax = MaxInt32) so semantic units arrive whole and are cut once, in tokens. Without one, nothing changes: nil budget, or a provider that reports ExactCounts() == false, keeps the byte path byte for byte. An estimating provider is routed to the byte path on purpose — its numbers are the same guess, and dressing them as a token budget would hide that from the caller. splitChunkTokens also closes a hole the byte splitter had. That loop requires len(currentLines) > 1, so a single line longer than the cap was never split: on the 45-repo reference corpus that produced a 65 KB chunk, which the voyage provider then cut into byte windows and averaged the vectors of — a vector representing neither half, with nothing in the logs to say so. A line that cannot fit alone is now cut on real token boundaries from Budget.SplitPoints, and the tail seeds the next chunk so short following lines can still join it. Attribution rule is unchanged: only the first piece keeps SymbolName and ChunkType, the rest become `block`, so one long function does not produce N symbol rows all claiming to be it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e counts The first token splitter counted each line, accumulated, and re-joined the lines it had collected. Joining reinserts the newlines, and a newline plus the next line's indentation is its own pre-token, so the joined text costs more than the sum the loop was tracking. On real files a 1500-token budget produced chunks of up to 1546 tokens — verified by counting the largest chunks of two freshly reindexed repos with the model's own tokenizer. Cut positions now come from Budget.SplitPoints over the whole content, which is exact by construction, and the pieces are SUBSTRINGS of that content rather than reassembled text. The class of error disappears instead of being compensated for. Two consequences worth spelling out, both learned from tests that failed: Snapping a cut back to a line start (so a piece never begins mid-line and its recorded line range does not lie) shrinks the piece BEFORE the cut and grows the one after it. Reusing the remaining offsets SplitPoints had returned would hand the next piece the tokens this one gave up and push it over budget. So each piece recomputes its cut from the actual new start; the loop is not an inefficiency, it is the correctness. The content-preservation invariant is asserted on splitChunkTokens directly rather than through ChunkFileTokens, because the sliding-window fallback deliberately overlaps its windows for recall — whole-pipeline output is not expected to concatenate back to the source, and a test that assumed otherwise was wrong about the pipeline, not about the splitter. Re-verified on the file that produced the reference corpus's 65 KB chunk: 48 pieces, largest exactly 1500 tokens, none over. The byte path leaves that file as 6 chunks whose largest is 65,553 tokens — twice voyage-code-3's 32K context, which with truncation enabled means the tail was silently dropped before embedding. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…n a real corpus Three bugs shipped into this branch before being caught, and the reason each one survived is the same: the only thing verifying the token layer was my own model of it. The tokenizer next door had zero bugs across 70,412 inputs because it could be checked against HuggingFace's implementation and against Voyage's billing. Chunk splitting has no such oracle — where to cut a chunk is our decision, not a spec — so the oracle has to be built out of invariants and inputs nobody wrote for the test. fakeBudget was the direct cause of the worst one. It counted whitespace-separated words and let newlines be free, which made the sum of per-line counts equal the count of the joined text — exactly the assumption the implementation got wrong. The double agreed with the bug, the tests passed, and real files came out 3% over budget. It now charges a token per newline, like the real tokenizer does, so summing parts no longer equals the whole unless the pieces are substrings — which is the property the splitter must have. The corpus tests run the chunker over real files from the local fixture: 396 files across 45 repositories, 5,199 chunks, largest exactly 1,500 against a 1,500 budget; 132 files large enough to need splitting, all reconstructing byte for byte with correct line numbers. Those files are what found the original bugs — a 65 KB single-line Zig literal, minified JavaScript that has no grammar and so takes the fallback path — and hand-written cases had not. The fixture is tens of gigabytes and is not in the repository, so the tests skip unless CIX_TEST_CORPUS_DIR and CIX_TEST_TOKENIZER are set. A clean checkout and CI stay green; anyone with a fixture gets the coverage. The file header says how to point them at one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dvcdsys
changed the title
Throughput, measured rather than reasonedThe PR body says the throughput win lives in the TPM throttle rather than in batch packing. That is now A/B tested on the same two repositories, same content, same code — the only difference being whether Run 1 — real tier limits (3M TPM):
No gain — 9% slower, inside noise. The reason is visible in the arithmetic: 22,390 chunks at ~221 tokens each is ~5M tokens over four minutes, i.e. ~1.25M TPM real and ~2.4M metered. Neither figure reaches the 3M limit, so the token bucket never made anyone wait, and the only thing left to measure was the cost of tokenising (plus clone/network variance). Run 2 — the throttle forced to bind (20K TPM), 15 files, 47,553 real tokens:
1.57×. Below the 1.94× the inflation implies, because chunking, SQLite writes and request latency are not throttle-bound and do not shrink. What this means for the changeThe win is conditional, not general: it appears exactly when an indexing run is large or concurrent enough to sit against the account's TPM ceiling, and it is absent on a small reindex. The original 45-repo run was in the first category — throughput plateaued at ~128 chunks/s, which is ~1.7M real TPM metered as ~3.3M against a 3M limit. Batch packing, separately, gains 1.10× (1724 → 1568 requests over 200k real chunks) because batches are bound by the 128-input cap of So the case for this change rests on correctness — no undercounts, no silent truncation of over-long inputs, chunk sizes in the model's own unit — with a throughput improvement that shows up only under throttle. Sizing expectations off the 1.94× figure alone would be wrong. |
…mits Nine findings from review, grouped by what they were actually about. CORRECTNESS — offsets into the wrong string (blocking). SplitPoints normalised its input and returned offsets into that NFC copy while callers slice the original. NFC changes lengths in both directions: decomposed input shrinks (e + U+0301 becomes one code point) so cuts landed mid-rune, and the composition exclusions at U+0958..U+095F DEcompose under NFC, making the normalised form longer, so an offset could exceed len(raw) and panic the indexer on the slice expression. Reproduced both ways before fixing. Input that is already NFC — nearly all source — takes a fast path where the two coincide; anything else is cut one piece at a time, each cut mapped back to a raw offset on a normalisation boundary, rounding down so the budget survives the mapping. Every previous fixture was ASCII, which is why nothing caught it. CORRECTNESS — an over-budget pre-token left its tail uncounted (blocking in spirit; latent for today's only caller). After splitInside the accumulator was reset to zero although the tail after the last inner cut opens the next piece, so following pre-tokens could stack a full budget on top of it and produce pieces of nearly twice the budget. The chunker escaped it by consuming only the first cut and recomputing, but the doc invites consuming the whole slice. CORRECTNESS — an unvalidated tokenizer could be confidently wrong (blocking). LoadBytes checked only model.type == "BPE" while the Qwen2 pre-tokenizer regex and the NFC normalizer are hardcoded here. A GPT-2 or o200k tokenizer.json parses fine, declares BPE, and would have produced plausible wrong counts with ExactCounts() reporting true — which then sizes chunks and packs batches. The declared normalizer and pre-tokenizer are now compared against what this package implements, and a mismatch fails the load so the caller stays on its estimate, which is wrong but knows it. CORRECTNESS — MaxInputTokens was a constant 32000, but the provider's own model enum offers voyage-code-2 at 16K. It is a per-model table now, with unknown models falling back to the smaller window: undershooting costs a needless split, overshooting costs silently truncated input. CORRECTNESS — token-sized chunks reopened the averaging path. The chunker lifts the byte cap when it has a budget, but the provider still cut any input over 30 KB into byte windows and AVERAGED their vectors — the invisible quality loss this branch exists to remove. At CIX_MAX_CHUNK_TOKENS=20000, legal and well inside the window, that would have been routine. With a tokenizer the provider now asks the real question (does this exceed the model's context) and cuts on token boundaries; byte windows remain only where there is no tokenizer and therefore no better answer. CORRECTNESS — the fallback path was still byte-biased. innerMax reached only the tree-sitter path, so files with no grammar kept the fixed 4000-byte sliding window, and boundTokens could not repair it: it splits chunks that are too big and cannot grow ones that are too small. Multi-byte text — which correlates with "no grammar" more than one would like — produced chunks worth a third of the budget on exactly the path the description claimed was covered. ROBUSTNESS — Service.TokenBudget was the only Service method without the nil/disabled guard, and a typed-nil *Service satisfies the capability interface the indexer asserts on, so the first indexed file would panic in RLock. The indexer's inline anonymous assertion is now the named TokenBudgetSource, hoisted out of the per-file loop (the per-file CALL stays — a provider swap between files is legitimate): renaming TokenBudget is now a compile error rather than a silent return to byte chunking. EFFICIENCY — splitChunkTokens called SplitPoints per piece and used only the first cut, rescanning the whole remainder each time: on the 66 KB single-line case that is ~990 suffix scans instead of ~44. The cut list is now reused while it stays valid and recomputed only when snapping to a line start actually moves a boundary — which on single-line content never happens. boundTokens also skips tokenising entirely when len(content) <= budget: a byte-level BPE token cannot cover less than a byte, so that comparison PROVES the chunk fits, and most chunks are small. Corpus property tests dropped from 5.4s/7.6s to 3.8s/2.2s. EFFICIENCY — the contraction branch lowercased the entire remaining suffix for every apostrophe, though only two bytes can match. A 512 KB file with 5,000 quotes moved over a gigabyte through the allocator on the indexing hot path. Also: the dead estimating branch of Provider.SplitPoints (unreachable, and it looped forever when the budget was smaller than one leading multi-byte rune) is gone; the batch log reported the Config cap while packing used the Provider cap; CIX_MAX_CHUNK_TOKENS's default now comes from chunker.DefaultMaxChunkTokens instead of a second copy of 1500; and the /loadtests/ gitignore entry ships here rather than sitting in a working tree, since committed tests reference that path. Tests: NFD and composition-exclusion cases (both previously absent — every fixture was ASCII), a foreign-pipeline rejection case, an uncounted-tail case, a fallback-fills-the-budget case, and TestSplitPointsOversizePreToken fixed to actually enter splitInside — "aB3" repeated pre-tokenizes into 2-byte pieces that never exceed any budget, so the path it named was never executed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Review addressed in c598cb5. Everything blocking is fixed and covered by a test that fails without the fix. Below is what I verified independently, and the two places I did something different from the suggestion. Blocking1. Offsets into the NFC copy. Confirmed by reproducing both directions before touching anything — decomposed input made cuts land mid-rune, and U+0958..U+095F (which decompose under NFC, so the normalised form is longer) produced 2. Trust chain. On "warn when Model changes while tokenizer_path stays": I did not add it. The pipeline check catches a different family of tokenizer, but voyage-code-3 and a hypothetical voyage-code-4 would share the pipeline and differ only in vocabulary, and nothing in tokenizer.json names the model. A warning keyed on nothing observable would be theatre. What would actually work is a fingerprint (merge count) pinned per model — worth doing when the packaging question is settled, since embedding the file removes the mismatch case entirely. 3. Averaging path reopened. Fixed at the provider rather than by teaching the chunker about bytes: with a tokenizer it now asks whether the input exceeds the model's context and cuts on token boundaries; byte windows remain only where there is no tokenizer and therefore no better answer. Tying the chunker to Should fix4. 5. 6. Fallback still byte-biased — this one was the most substantive. Efficiency7. Quadratic rescan — cut list is reused while valid, recomputed only when line-snapping actually moves a boundary (never on single-line content). 8. 8, second half — threading the count from the chunker into Tests10. MinorDead estimating branch of Left alone: Full suite and |
…byte runs Two blockers from the second review, both confirmed before fixing. DEADLOCK. splitInside binary-searched over byte offsets and aligned its midpoint to a rune start by DECREMENTING. When alignment pulled the midpoint below lo, the next lo = mid+1 did not advance, and the (lo, hi) pair repeated forever. Any run of multi-byte runes long enough to exceed the budget as one pre-token reached it — a box-drawing comment separator does, and so does an arrow run or a run of combining marks. The worker did not error or slow down, it stopped. Reproduced at budgets 5 and 50 on four inputs, including already-NFC ones on the fast path, so this was not confined to the new denormalised route. The search now runs over rune-start INDICES enumerated once, so every candidate is a valid boundary by construction and no alignment step exists to misbehave. A single rune that exceeds the budget is emitted rather than refused: not advancing is the failure being removed. All four inputs are tests now, at both budgets, asserting termination AND that no piece is cut mid-rune. Strictly this loop predates the previous commit, but the denormalised path added routes into it. UNREACHABLE FALLBACK. chunkFallbackTokens was dead code. chunkWithTreesitter never returns an error — all six of its decline paths called chunkFallback themselves and returned the byte windows as SUCCESS — so the caller's fallback branch could not run, and no-grammar, minified, parse-failure and empty-AST files kept getting 4000-byte windows regardless of any token budget. That is the finding the previous commit claimed to fix. Those six sites now return errUseFallback and the caller decides, because the caller is the one that knows whether a budget is in play. ChunkFile's byte path is unchanged: it already mapped err to chunkFallback. The test that was supposed to catch this passed by arithmetic. At budget 200 a 4000-byte window of the fixture is ~358 fake tokens, so boundTokens split every window into 200+158 and both halves cleared half-budget without the fallback being token-aware at all. Raised to 800, where a raw window is BELOW half the budget: verified it now fails against the old routing (7 of 8 chunks under half budget) and passes against the new. Also from the review: the split log reported max_input_bytes even when the split was token-bounded; the provider-level token-split branch had no direct test (added, covering both pass-through of a large-but-legal input and cutting one past the window); a doc comment still named the old lowercase constant. The token fallback drops the byte window's 500-byte overlap. That is deliberate and now documented at the site: overlap has to be expressed in tokens to coexist with a token budget — size pieces at budget minus overlap, then extend each start back into its predecessor — which is a change worth making on its own rather than smuggling into a correctness fix. The tree-sitter path has never overlapped, so the two paths are now consistent rather than the fallback being worse than its neighbours. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Numbers audit before merge — the body has been rewritten with a How these numbers were obtained section, and every figure now carries a marker pointing at what it was measured on. Three corrections came out of re-running things against the current tree rather than quoting earlier runs:
Also relabelled: the "~1.7M TPM real vs ~3.3M metered" line is arithmetic, not a measurement — it multiplies an observed rate by a measured mean and the measured inflation. The A/B under a forced 20K TPM ceiling (288 s → 183 s) is the part that actually demonstrates the throttle effect, and it is marked as such. Everything else was re-checked and holds: the 65,553-token Zig chunk, the 0-mismatch tokenizer validation, the live-reindex run with no 4xx/429/truncation, and the two throughput rows. |
1 participant
Chunk sizing and batch packing both ran on a byte guess. This replaces the guess with the model's actual tokenizer, and makes the chunker use it.
Every number below is followed by a superscript marker; the How these numbers were obtained section at the end says what each was measured on, and which of them are arithmetic rather than measurement.
Why
The voyage provider had no tokenizer, so token cost was
len(bytes)/2. Against Voyage's ownusage.total_tokensand the HuggingFace Rust tokenizer as oracle, that estimate:The chunker had the same problem one level up: its limit was
1500*3bytes, a token target expressed in bytes. That ratio holds for dense ASCII and fails elsewhere — Cyrillic or CJK comments cost 2–3 bytes per character, so a byte-capped chunk carried a fraction of the intended tokens, while minified JS packed several times more.Why not an off-the-shelf tokenizer
sugarme/tokenizerregexp.MustCompile: the Qwen2 pre-tokenizer contains\s+(?!\S)and RE2 has no lookaheadbpecountimplements the seven alternation branches by hand with Perl leftmost-first semantics: 0 mismatches against the HF oracle across 70,412 inputs ⟨3⟩. No new module dependency; 27.5k chunks/s single-threaded ⟨5⟩.What's here
internal/tokenizer— oneBudgetinterface (MaxInputTokens,ExactCounts,CountTokens,SplitPoints), so the chunker takes a single constructor argument and stays ignorant of which model is active.internal/tokenizer/bpecount— the counter and the splitter. Cuts are exact, not searched: merges never cross a pre-token boundary, so counts are additive and one pass yields cut points whose pieces provably sum to the whole.Budget; newtokenizer_pathconfig field; batch cap 80K → 115K when counts are exact; oversize inputs are cut on token boundaries instead of byte windows whose vectors were averaged. Operator override still wins.Measured
Correctness. The file behind the reference corpus's largest chunk (a 66 KB Zig integer literal): byte path → 6 chunks, largest 65,553 tokens against voyage-code-3's 32K context, so with truncation on the tail never reached the model. Token path → 48 chunks, largest exactly 1,500, none over ⟨6⟩.
Property tests over real files ⟨7⟩: 396 files from 45 repositories, 5,180 chunks, largest exactly 1,500 at a 1,500 budget; 132 files large enough to split, all reconstructing byte for byte with correct line numbers.
Live reindex of two repositories end to end ⟨8⟩: 0 errors, no 400s, no 429s, no
tokens after truncation. Vector count moved −1.0% onvuejs/core(15,783 → 15,630).Throughput gains are conditional. Batch packing: 1721 → 1568 requests over 200k real chunks, i.e. 1.10× ⟨1⟩ — batches are bound by the 128-input cap of
voyage-code-*, not by tokens. The rest lives in the TPM throttle, which meters the inflated estimate, and it only pays when the throttle actually binds ⟨9⟩:len/2A small reindex sees nothing; a full 70-repo run sits against the ceiling and sees most of it. Sizing expectations off the 1.93× inflation figure alone would be wrong.
Review notes
tokenizer_path: a nil budget, or a provider reportingExactCounts() == false, keeps the byte path byte for byte. An estimating provider is routed to the byte path deliberately — its numbers are the same guess, and dressing them as a token budget would hide that.SplitPointsreturned offsets into its NFC copy (panicked on composition exclusions), andsplitInsidedeadlocked on any run of multi-byte runes — a box-drawing comment separator was enough to stop an indexing worker with no error.How these numbers were obtained
Hardware: Apple M-series, 14 cores, 36 GB RAM, NVMe. Model config throughout:
voyage-code-3,output_dimension=2048,dtype=float, matching the production deployment this targets. "Reference corpus" means a local fixture of 45 large public GitHub repositories (kubernetes, grafana, django, rails, redis, deno, …), server-cloned and indexed by this build: 1.9M chunks, 21 GB of vectors. It is not in the repository.CIX_WORKER_CONCURRENCY=12.usage.total_tokens, 20,000 chunks sampled from the store, and 50,000 fuzz strings, each compared against the HuggingFace Rust tokenizer.go test ./internal/chunker/ -run TestRealFileAgainstRealTokenizeragainst that one file with the real tokenizer, byte path and token path side by side.CIX_TEST_CORPUS_DIR=… CIX_TEST_TOKENIZER=… go test ./internal/chunker/ -run Corpus— the property tests incorpus_property_test.go, sampling deterministically from the fixture's checkouts.vuejs/coreandaxios/axiosdeleted and re-added throughPOST /api/v1/git-repos, so the server cloned and indexed them from scratch with this build; logs then grepped for 4xx/429 and truncation warnings.tokenizer_path. The second row uses a 15-file / 47,553-token project so the forced 20K TPM ceiling binds without spending much of the API budget.⟨A⟩ is arithmetic, not a measurement: it multiplies the observed rate ⟨2⟩ by the measured mean chunk size and inflation ⟨1⟩. The A/B in row 2 of the throughput table is what actually demonstrates the throttle effect.
🤖 Generated with Claude Code