feat: support structured special-token JSON inputs

[kexinoh]

To address #2225, I made the following changes.

#2225 mainly discusses a highly fragmented situation that exists today: although tokenizers already has a mature special-token mechanism, the encoding pipelines of the vast majority of models deployed for inference with vLLM/SGLang still ignore special tokens. In the agent era, this fragmentation can amplify security risks and potentially undermine existing safety-alignment strategies.

So far, the only approach we have observed is that a small number of models choose to bypass Jinja, but this requires them to reimplement their own parsers. Examples include K3 (https://github.com/vllm-project/vllm/blob/d3e2888c7588fe3a7be93606c7c626dbfd304d2e/rust/src/chat/src/renderer/kimi_k3/encoding.rs#L29) and DeepSeek (https://github.com/vllm-project/vllm/blob/main/vllm/tokenizers/deepseek_v32_encoding.py).

We are introducing this abstraction into the tokenizer itself (with follow-up integration needed in Jinja and Transformers). This should help reduce duplicated work across the community and improve the overall security of the ecosystem.

Summary

Adds opt-in structured special-token JSON support via structured_special_tokens=True across sync, async, batch,
and fast encoding APIs.

• Accepts Python dictionaries or JSON strings using the jinja_render_segments v1 schema.
• Encodes "special": "yes" segments only when they exactly match registered special tokens.
• Encodes "special": "no" segments as ordinary text without mutating tokenizer-wide policy.
• Preserves offsets, truncation, padding, post-processing, concurrency safety, and legacy behavior.
• Strictly rejects malformed schemas, unknown special tokens, pair inputs, and pretokenized inputs.

Validation

Passed Rust formatting, Clippy, 299 Rust tests, 159 Python binding tests, and dedicated sync/async structured-
input coverage.

[ArthurZucker]

Hey! I'll have a look I am in the middle of a big refactoring for RC0!

[ArthurZucker]

hey, will re-consider after #2119

[kexinoh]

okay

[kexinoh]

How’s it going?

[ArthurZucker]

rc0 should shop soon if you want to rebase to that branch! from the PR