Skip to content

Fix custom prefixes in BertWordPieceTokenizer - #2413

Open
HaokaiDing wants to merge 1 commit into
huggingface:mainfrom
HaokaiDing:fix/bert-wordpiece-custom-prefix
Open

HaokaiDing wants to merge 1 commit into
huggingface:mainfrom
HaokaiDing:fix/bert-wordpiece-custom-prefix

Conversation

@HaokaiDing

Copy link
Copy Markdown

Fixes #2412.

BertWordPieceTokenizer passes wordpieces_prefix to its decoder but leaves the WordPiece model using ##. Loading a vocabulary trained with a custom prefix such as @@ consequently turns known words into [UNK].

Forward the prefix to both WordPiece constructor branches. Add tests for default/custom prefixes that check tokens, IDs, offsets and decoding, plus a training/save/reload round trip.

Validation:

  • Before the fix: 2 failed, 3 passed. After: 5 passed, 2 network tests deselected in tests/implementations/test_bert_wordpiece.py.
  • Tested the current checkout's Python package with the released tokenizers 0.23.2 native extension; the local runner verifies the wrapper's source path and native module identity. Current Rust was not rebuilt and network-dependent tests were not run.
  • Ruff lint and format checks pass on both changed files.
  • Python 3.13.15 on macOS ARM64. No dependency changes.

Copilot AI lite review requested due to automatic review settings September 16, 2026 10:11

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BertWordPieceTokenizer ignores wordpieces_prefix when loading a vocabulary

2 participants