feat(korean_pos): add companion stream 256-byte fallback, OOV benchmark suite, and evaluation improvements - #891
Open
kahye wants to merge 4 commits into
Open
Conversation
…, and resilience experiment report
…ility evaluation suite - Expand Korean POS dataset in get_dataset.sh with OPUS-100 and KLUE task splits (DP, NER, MRC, NLI, RE, STS, YNAT) and update lane_metadata.json - Add --pos_loss_weight argument in train_args.py and handle POS loss weighting and milestone checkpoint saving in train.py - Add --mc_ckpt and --base_ckpt path override support to benchmarks/run_phonetic_slang_eval.py and benchmarks/run_vocab_tail_perplexity.py - Add 4-capability evaluation benchmark suite (benchmarks/run_four_capability_evals.py) covering KLUE-NER, KLUE-DP, noisy text resilience (NSMC/UnSmile), and rare vocabulary/OOV (KorMedMCQA) - Add evaluation runner demos/run_all_epoch_evals.sh, Option 1 sweep runner run_option1_sweep.py, and 10-epoch experiment runner run_opt1_10ep_experiment.py
… POS tagsets, and benchmark support - Implement HangulFullPosFactorizedTokenizer (46 Sejong tags) and HangulCoarsePosFactorizedTokenizer (17 mapped macro tags) in hangul_factorizer.py - Add make_byte_fallback_meta() to support 256-byte companion character stream without OOV drop - Update POS lane metadata and unit tests in test_hangul_factorizer.py - Add 59.5M token milestone checkpoint saves (3ep: 10899, 5ep: 18165, 10ep: 36330) in train.py - Add prepare_pos_and_byte_lanes.py to prepare Full POS and 256-Byte Fallback companion stream - Add run_pos_byte_experiments.py automation runner for training and evaluating Full vs Coarse POS under Weighted and Unweighted loss - Update evaluation benchmarks (run_four_capability_evals.py, run_ko_hellaswag.py, run_vocab_tail_perplexity.py, run_phonetic_slang_eval.py) to support byte fallback and full/coarse POS models
… and tokenizer byte-fallback improvements - Add curated 20-prompt OOV and Unicode benchmark suite in benchmarks/prompts/ covering archaic Hangul, rare Hanja, ancient scripts, complex emojis, and mathematical notation - Add benchmarks/run_oov_evaluations.py to evaluate baseline and multicontext models on OOV prompts - Optimize CharBPETokenizerWithByteFallback encoding loop with length bucketing and interval progress updates - Update benchmark encoders across capability tests to properly use get_tokenizer_functions - Update milestone checkpoint save iterations in train.py - Update .gitignore to track benchmark prompt files
kahye
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
256-Byte Fallback & POS Tagsets:
HangulFullPosFactorizedTokenizer(46 Sejong tags) andHangulCoarsePosFactorizedTokenizer(17 mapped macro tags) inhangul_factorizer. py.make_byte_fallback_meta()companion character stream supporting 256-byte fallback without dropping OOV tokens.OOV & Unicode Stress-Test Suite:
- Added curated 20-prompt test suite in
benchmarks/prompts/across 6 categories (Middle Korean archaic Hangul, rare Hanja, ancient SMP scripts, complexZWJ/flag emojis, calculus notation, and internet slang).
- Created
benchmarks/run_oov_evaluations.pyto evaluate loss, perplexity, and generation across baselines and multicontext models.Tokenizer & Benchmark Optimizations:
- Optimized
CharBPETokenizerWithByteFallbackencoding loop with token length bucketing and batch progress updates.- Updated benchmark encoders (
run_four_capability_evals.py,run_ko_hellaswag.py,run_phonetic_slang_eval.py,run_vocab_tail_perplexity.py) toproperly resolve
get_tokenizer_functions.- Added milestone checkpoint save intervals in
train.py.