fix(reward): grade x=5, scientific notation, and empty \boxed{} on MATH-500 (#107)#113
Open
minion1227 wants to merge 1 commit into
Open
fix(reward): grade x=5, scientific notation, and empty \boxed{} on MATH-500 (#107)#113minion1227 wants to merge 1 commit into
minion1227 wants to merge 1 commit into
Conversation
…TH-500 (mini-router#107) The math grader marked several equivalent-but-unnormalized MATH-500 answers wrong, silently under-counting correct answers and biasing both the sep-CMA-ES training reward and reported eval accuracy downward (same false-negative class as the fixed thousands-comma bug mini-router#35): - Variable assignment: a boxed `x = 5` normalized to `x=5` and never matched the bare reference `5`. Grade the value, not the restated variable — drop a leading single-variable assignment in normalize_math_answer, guarded so `==` and inequalities (`x<=5`, `x>=5`) are left intact. - Scientific notation: after \times/\cdot become `*`, `5*10^{3}` parsed as neither float nor fraction, and the sympy fallback read `^` as XOR, so it never equaled `5000`. Rewrite `a*10^{b}` to the canonical `aeb` so the numeric path parses it. - Empty \boxed{}: extract_boxed returned "", which _check_math committed as the (blank) answer instead of falling back to the last number in the text the way a missing box already does. Treat an empty/whitespace box like a missing box. Adds tests/test_reward_math_forms.py: the three answer forms grade correct, while wrong values, inequalities, and an empty box with no number still score 0. Fixes mini-router#107
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #107
Problem
normalize_math_answer/_check_mathmark several correct MATH-500 answers wrong when the model writes an equivalent-but-unnormalized form. These are frequent answer shapes, so the grader silently under-counts, biasing both the sep-CMA-ES training reward and reported eval accuracy downward — the same false-negative class as the already-fixed thousands-comma bug (#35).Fix (three targeted spots)
x = 5normalized tox=5and never matched the bare reference5. Drop a leading single-variable assignment innormalize_math_answerso the value is graded. Guarded with a(?!=)lookahead so==stays intact, and because</>are not word characters, inequalitiesx<=5/x>=5are never touched.\times/\cdotbecome*,5*10^{3}parsed as neitherfloatnorFraction, and the sympy fallback reads^as XOR. Rewritea*10^{b}to the canonicalaebso the numeric path parses it (5*10^{3}->5e3->5000.0).\boxed{}—extract_boxedreturns"", which_check_mathcommitted as the blank answer instead of falling back to the last number in the text the way a missing box already does. Treat an empty/whitespace box like a missing box.Tests
Adds
tests/test_reward_math_forms.py(pure stdlib, no torch/GPU/network):\cdot)x<=5/x>=5inequalities, and an empty box with no number in the text still score 0normalize_math_answerdrops only a leading assignment (not==/<=) and rewrites scientific notationFull root suite: 206 passed;
ruff checkclean on the changed lines.