Skip to content

fix(train): drop unknown tokens during training and inference - #359

Open
serhiizghama wants to merge 3 commits into
MinishLab:mainfrom
serhiizghama:fix/train-drop-unk-tokens
Open

fix(train): drop unknown tokens during training and inference#359
serhiizghama wants to merge 3 commits into
MinishLab:mainfrom
serhiizghama:fix/train-drop-unk-tokens

test: cover the no-unk-token path in train tokenization

357b101
Select commit
Loading
Failed to load commit list.
Sign in for the full log view

Annotations

1 warning
Run tests with pytest (ubuntu-latest, 3.12)
succeeded Sep 4, 2026 in 2m 55s