Skip to content

fix: encode message content with split_special_tokens=True #116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
hubert-marek wants to merge 2 commits into
base: main
Choose a base branch
from
Open

fix: encode message content with split_special_tokens=True #116

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
83d20bc
fix: encode message content with split_special_tokens=True
hubert-marek Aug 5, 2026
23a7029
fix: exclude GLM-4.5-Air from split_special_tokens
hubert-marek Aug 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions renderers/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1819,6 +1819,7 @@ def attribute_text_segments(
segments: "list[tuple[str, bool]]",
*,
overlap_is_content: bool = False,
split_special_tokens: bool = True,

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve scaffold special tokens while splitting content

When attribute_text_segments is used for mixed scaffold/body runs, this new default also splits template-owned scaffold tokens, not just caller content. For example Qwen3 builds the tools system block through this helper and its scaffold contains literal <tool_call></tool_call> tags; with this default those no longer collapse to the tokenizer's tool-call special IDs, so any render with tools= diverges from the chat template/model wire format even though only user/body text needed protection.

Useful? React with 👍 / 👎.

) -> "list[tuple[int, bool]]":
"""Tokenize concatenated segments as a single BPE pass and return
``(token_id, is_content)`` pairs.
Expand Down Expand Up @@ -1867,6 +1868,7 @@ def attribute_text_segments(
encoding = offset_tokenizer(
full_text,
add_special_tokens=False,
split_special_tokens=split_special_tokens,

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Cover PrimeQwen3 pure text emits

This only closes the segmented helper path, but PrimeQwen3 is still registered for PrimeIntellect/Qwen3-0.6B and PrimeIntellect/Qwen3-1.7B, and its _TokenBuilder.emit_text still calls tokenizer.encode(text, add_special_tokens=False) directly in renderers/prime_qwen3.py. Any pure body flush for that renderer can still collapse a literal special-token spelling in caller content into a control ID, so the safety fix is incomplete for a model in the supported matrix.

Useful? React with 👍 / 👎.

return_offsets_mapping=True,
)
token_ids = list(encoding["input_ids"])
Expand Down
4 changes: 3 additions & 1 deletion renderers/deepseek_v3.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,9 @@ def _get_special_token(self, name: str) -> int:
def _encode(self, text: str) -> list[int]:
if not text:
return []
return self._tokenizer.encode(text, add_special_tokens=False)
return self._tokenizer.encode(
text, add_special_tokens=False, split_special_tokens=True
)

# ------------------------------------------------------------------
# Public API
Expand Down
8 changes: 6 additions & 2 deletions renderers/glm45.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,8 +158,10 @@ def emit_text_segments(
same way as the chat template, but attributed separately"
without splitting the encode call (which could shift BPE
merges at the boundary)."""
# split_special_tokens=False: GLM's template reads /nothink out of
# user content, so content must keep matching special tokens here.
for tok_id, is_content in attribute_text_segments(
self._tokenizer, segments
self._tokenizer, segments, split_special_tokens=False
):
tokens.append(tok_id)
indices.append(msg_idx)
Expand Down Expand Up @@ -381,8 +383,10 @@ def emit_text_segments(
*,
is_sampled: bool = False,
) -> None:
# split_special_tokens=False: GLM's template reads /nothink out of
# user content, so content must keep matching special tokens here.
for tok_id, is_content in attribute_text_segments(
self._tokenizer, segments
self._tokenizer, segments, split_special_tokens=False
):
ext.append(tok_id)
ext_indices.append(msg_idx)
Expand Down
4 changes: 3 additions & 1 deletion renderers/glm5.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,9 @@ def _token_id(self, token: str) -> int:
def _encode(self, text: str) -> list[int]:
if not text:
return []
return self._tokenizer.encode(text, add_special_tokens=False)
return self._tokenizer.encode(
text, add_special_tokens=False, split_special_tokens=True

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep tool-prompt scaffolds atomic

This helper is also used for renderer-owned scaffold strings, not just caller message text. In GLM5, for example, the tools= path appends _TOOLS_FOOTER and then calls emit_text(tool_text, ...); that footer contains literal <tool_call>, <arg_key>, and <arg_value> markers that the chat template tokenizes as special IDs, but this new unconditional split turns them into ordinary text pieces and breaks byte parity for any GLM5 tools prompt.

Useful? React with 👍 / 👎.

)

@staticmethod
def _visible_text(content: Any) -> str:
Expand Down
4 changes: 3 additions & 1 deletion renderers/gpt_oss.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,9 @@ def _token_id(self, token: str) -> int:
def _encode(self, text: str) -> list[int]:
if not text:
return []
return self._tokenizer.encode(text, add_special_tokens=False)
return self._tokenizer.encode(
text, add_special_tokens=False, split_special_tokens=True
)

def _prefix_content_mask(
self,
Expand Down
4 changes: 3 additions & 1 deletion renderers/hy3.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,9 @@ def _token_id(self, token: str) -> int:
def _encode(self, text: str) -> list[int]:
if not text:
return []
return self._tokenizer.encode(text, add_special_tokens=False)
return self._tokenizer.encode(
text, add_special_tokens=False, split_special_tokens=True
)

@cursor cursor Bot Aug 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hy3 segment encode path unprotected

Medium Severity

_attribute_segments encodes concatenated system (and tools) text without split_special_tokens=True, even though Hy3’s _encode was updated. System message bodies that go through this helper can still synthesize control tokens.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 23a7029. Configure here.


@staticmethod
def _visible_text(content: Any) -> str:
Expand Down
4 changes: 3 additions & 1 deletion renderers/kimi_k2.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,9 @@ def _token_id(self, token: str) -> int:
def _encode(self, text: str) -> list[int]:
if not text:
return []
return self._tokenizer.encode(text, add_special_tokens=False)
return self._tokenizer.encode(
text, add_special_tokens=False, split_special_tokens=True
)

def _ensure_system_message(
self, messages: list[Message]
Expand Down
4 changes: 3 additions & 1 deletion renderers/kimi_k25.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -721,7 +721,9 @@ def _try_token_id(self, token: str) -> int | None:
def _encode(self, text: str) -> list[int]:
if not text:
return []
return self._tokenizer.encode(text, add_special_tokens=False)
return self._tokenizer.encode(
text, add_special_tokens=False, split_special_tokens=True
)

# ------------------------------------------------------------------
# Core render
Expand Down
4 changes: 3 additions & 1 deletion renderers/laguna_xs2.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,9 @@ def _token_id(self, token: str) -> int:
def _encode(self, text: str) -> list[int]:
if not text:
return []
return self._tokenizer.encode(text, add_special_tokens=False)
return self._tokenizer.encode(
text, add_special_tokens=False, split_special_tokens=True

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve Laguna raw stop tokens

When LagunaXS2RendererConfig(render_assistant_messages_raw=True) is used with raw assistant content that already contains </think> or ends with </assistant>, this branch relies on emit_text(content, ...) to keep those added-vocab markers as their special IDs, and even skips appending an explicit close token when the content ends with </assistant>. With split_special_tokens=True, the raw content is emitted as ordinary text pieces instead, so the rendered history can lose the actual assistant stop token and no longer matches the raw chat-template path.

Useful? React with 👍 / 👎.

)

@staticmethod
def _visible_text(content: Content | None) -> str:
Expand Down
4 changes: 3 additions & 1 deletion renderers/llama_3.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,7 +130,9 @@ def _token_id(self, token: str) -> int:
def _encode(self, text: str) -> list[int]:
if not text:
return []
return self._tokenizer.encode(text, add_special_tokens=False)
return self._tokenizer.encode(
text, add_special_tokens=False, split_special_tokens=True
)

@staticmethod
def _content_str(content: Any) -> str:
Expand Down
4 changes: 3 additions & 1 deletion renderers/minimax_m2.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,9 @@ def _token_id(self, token: str) -> int:
def _encode(self, text: str) -> list[int]:
if not text:
return []
return self._tokenizer.encode(text, add_special_tokens=False)
return self._tokenizer.encode(
text, add_special_tokens=False, split_special_tokens=True
)

@cursor cursor Bot Aug 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minimax tool encode path unprotected

High Severity

emit_token_overlap_body still tokenizes tool-response body text without split_special_tokens=True, in both render and the bridge path. That is the content path this PR aims to harden, so literal special-token spellings in tool output can still collapse into reserved ids.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 23a7029. Configure here.


@staticmethod
def _visible_text(content: Any) -> str:
Expand Down
4 changes: 3 additions & 1 deletion renderers/nemotron3.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,9 @@ def _token_id(self, token: str, *, optional: bool = False) -> int | None:
def _encode(self, text: str) -> list[int]:
if not text:
return []
return self._tokenizer.encode(text, add_special_tokens=False)
return self._tokenizer.encode(
text, add_special_tokens=False, split_special_tokens=True
)

# ------------------------------------------------------------------
# Content rendering
Expand Down
4 changes: 3 additions & 1 deletion renderers/qwen3.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,9 @@ def _token_id(self, token: str) -> int:
def _encode(self, text: str) -> list[int]:
if not text:
return []
return self._tokenizer.encode(text, add_special_tokens=False)
return self._tokenizer.encode(
text, add_special_tokens=False, split_special_tokens=True
)

@staticmethod
def _query_boundary_text(content) -> str:
Expand Down
4 changes: 3 additions & 1 deletion renderers/qwen35.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -239,7 +239,9 @@ def _token_id(self, token: str) -> int:
def _encode(self, text: str) -> list[int]:
if not text:
return []
return self._tokenizer.encode(text, add_special_tokens=False)
return self._tokenizer.encode(
text, add_special_tokens=False, split_special_tokens=True
)

# ------------------------------------------------------------------
# Content rendering (mirrors the render_content Jinja macro)
Expand Down
4 changes: 3 additions & 1 deletion renderers/qwen3_vl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -370,7 +370,9 @@ def mm_token_type_id_map(self) -> dict[int, int]:
def _encode(self, text: str) -> list[int]:
if not text:
return []
return self._tokenizer.encode(text, add_special_tokens=False)
return self._tokenizer.encode(
text, add_special_tokens=False, split_special_tokens=True
)

def _get_processor(self):
if self._processor is not None:
Expand Down
Loading