Skip to content

feat: size chunks in tokens — exact counting via the model's own BPE tokenizer - #262

Merged
dvcdsys merged 6 commits into
developfrom
feat/voyage-exact-token-counting
Aug 18, 2026
Merged

feat: size chunks in tokens — exact counting via the model's own BPE tokenizer#262
dvcdsys merged 6 commits into
developfrom
feat/voyage-exact-token-counting

fix(chunker,tokenizer): unreachable fallback and a deadlock on multi-…

0328c18
Select commit
Loading
Failed to load commit list.

Select a check to view from the sidebar