fix(reward): grade equation, scientific-notation, and empty-boxed math answers#108
Open
philluiz2323 wants to merge 5 commits into
Open
fix(reward): grade equation, scientific-notation, and empty-boxed math answers#108philluiz2323 wants to merge 5 commits into
philluiz2323 wants to merge 5 commits into
Conversation
…h answers
normalize_math_answer / _check_math scored several correct MATH-500 answers as
wrong because the model wrote them in an equivalent but unnormalized form:
- "x = 5" was never reduced to its value "5";
- "5*10^{3}" (after \times -> *) matched neither float() nor the fraction path,
and the sympy fallback reads "^" as XOR, so it never equaled "5000";
- an empty \boxed{} was committed as the (blank) answer instead of falling back
to the last number in the text.
Fixes: (1) drop a leading single-variable assignment (^[a-z]\w*=), guarded by a
(?=[^=]) lookahead and an identifier-only LHS so inequalities ("x<=5") and "=="
are left intact; (2) rewrite trailing scientific notation a*10^{b} -> aeb so it
parses as a float; (3) treat an empty/whitespace \boxed{} like a missing box and
fall back to extract_last_number.
Adds tests/test_reward_math_forms.py (22 cases): the wins plus guards that wrong
values, inequalities, and empty-with-no-number still score 0, and that existing
forms (fractions, commas, negatives, tuples) are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #107
normalize_math_answer/_check_mathscored several correct MATH-500 answers as wrong because the model wrote them in an equivalent but unnormalized form:�oxed{x=5}vs5— the variable assignment was never reduced to its value;�oxed{5 imes10^{3}}vs5000— afterimes->*,5*10^{3}matched neitherfloat()nor the fraction path, and the sympy fallback reads^as XOR;�oxed{}was committed as the (blank) answer instead of falling back to the last number in the text.Same class of false negative as the merged thousands-comma fix (#35).
What Changed
src/trinity/orchestration/reward.pynormalize_math_answer: drop a leading single-variable assignment^[a-z]\w*=(guarded by a(?=[^=])lookahead and an identifier-only LHS, so inequalities likex<=5and==are untouched); rewrite trailing scientific notationa*10^{b}->aebso it parses as a float._check_math: treat an empty/whitespace�oxed{}like a missing box and fall back toextract_last_number.tests/test_reward_math_forms.py: 22 offline cases — the fixes plus guards that wrong values, inequalities, and empty-with-no-number still score 0, and that existing forms (fractions, commas, negatives, tuples) are unchanged.Why This Matters
These answer shapes are common on MATH-500, so the grader silently under-counted correct answers in both the training reward and reported eval accuracy. The fixes are guarded to avoid introducing false positives.
Verification
Notes
A
docs/JOURNAL.mdentry was added per repository protocol (AGENTS.md §6). Unit-bearing answers (�oxed{5 ext{ cm}}vs5) are deliberately left for a follow-up — strippingext{...}risks false positives on textual answers likeext{Monday}.