feat: size chunks in tokens — exact counting via the model's own BPE tokenizer - #262

Merged

dvcdsys merged 6 commits into

feat/voyage-exact-token-counting

Aug 18, 2026

Commits on Aug 18, 2026

feat(voyage): exact token counting via the model's own BPE tokenizer

dvcdsys
and
claude
committed
d859d24
feat(chunker): size chunks in tokens, not in a byte stand-in

dvcdsys
and
claude
committed
a3a8152
fix(chunker): take cut points from the tokenizer, not from summed line counts

dvcdsys
and
claude
committed
e343209
test(chunker): make the double adversarial and check the properties on a real corpus

dvcdsys
and
claude
committed
7906b4a
fix: review findings — offset mapping, tokenizer validation, model limits

dvcdsys
and
claude
committed
c598cb5
fix(chunker,tokenizer): unreachable fallback and a deadlock on multi-byte runs

dvcdsys
and
claude
committed
0328c18