Skip to content

fix(reward): grade equation, scientific-notation, and empty-boxed math answers#108

Open
philluiz2323 wants to merge 5 commits into
mini-router:mainfrom
philluiz2323:sn74-philluiz2323-math-normalize
Open

fix(reward): grade equation, scientific-notation, and empty-boxed math answers#108
philluiz2323 wants to merge 5 commits into
mini-router:mainfrom
philluiz2323:sn74-philluiz2323-math-normalize

Conversation

@philluiz2323

Copy link
Copy Markdown
Contributor

Summary

Closes #107

normalize_math_answer / _check_math scored several correct MATH-500 answers as wrong because the model wrote them in an equivalent but unnormalized form:

  • �oxed{x=5} vs 5 — the variable assignment was never reduced to its value;
  • �oxed{5 imes10^{3}} vs 5000 — after imes -> *, 5*10^{3} matched neither float() nor the fraction path, and the sympy fallback reads ^ as XOR;
  • an empty �oxed{} was committed as the (blank) answer instead of falling back to the last number in the text.

Same class of false negative as the merged thousands-comma fix (#35).

What Changed

  • src/trinity/orchestration/reward.py
    • normalize_math_answer: drop a leading single-variable assignment ^[a-z]\w*= (guarded by a (?=[^=]) lookahead and an identifier-only LHS, so inequalities like x<=5 and == are untouched); rewrite trailing scientific notation a*10^{b} -> aeb so it parses as a float.
    • _check_math: treat an empty/whitespace �oxed{} like a missing box and fall back to extract_last_number.
  • tests/test_reward_math_forms.py: 22 offline cases — the fixes plus guards that wrong values, inequalities, and empty-with-no-number still score 0, and that existing forms (fractions, commas, negatives, tuples) are unchanged.

Why This Matters

These answer shapes are common on MATH-500, so the grader silently under-counted correct answers in both the training reward and reported eval accuracy. The fixes are guarded to avoid introducing false positives.

Verification

$ PYTHONPATH=src python -m pytest tests/test_reward_math_forms.py -q     # 22 passed
$ PYTHONPATH=src python -m pytest tests/ --ignore=tests/smoke -q          # 198 passed

Notes

A docs/JOURNAL.md entry was added per repository protocol (AGENTS.md §6). Unit-bearing answers (�oxed{5 ext{ cm}} vs 5) are deliberately left for a follow-up — stripping ext{...} risks false positives on textual answers like ext{Monday}.

…h answers

normalize_math_answer / _check_math scored several correct MATH-500 answers as
wrong because the model wrote them in an equivalent but unnormalized form:

- "x = 5" was never reduced to its value "5";
- "5*10^{3}" (after \times -> *) matched neither float() nor the fraction path,
  and the sympy fallback reads "^" as XOR, so it never equaled "5000";
- an empty \boxed{} was committed as the (blank) answer instead of falling back
  to the last number in the text.

Fixes: (1) drop a leading single-variable assignment (^[a-z]\w*=), guarded by a
(?=[^=]) lookahead and an identifier-only LHS so inequalities ("x<=5") and "=="
are left intact; (2) rewrite trailing scientific notation a*10^{b} -> aeb so it
parses as a float; (3) treat an empty/whitespace \boxed{} like a missing box and
fall back to extract_last_number.

Adds tests/test_reward_math_forms.py (22 cases): the wins plus guards that wrong
values, inequalities, and empty-with-no-number still score 0, and that existing
forms (fractions, commas, negatives, tuples) are unchanged.
@github-actions github-actions Bot added docs Documentation changes miner Miner contribution PR train Training or routing changes labels Jul 11, 2026
@philluiz2323 philluiz2323 reopened this Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation changes miner Miner contribution PR train Training or routing changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reward: math grader misses common MATH-500 answer forms (x=5, scientific notation, empty oxed{})

1 participant