Skip to content

fix(reward): grade x=5, scientific notation, and empty \boxed{} on MATH-500 (#107)#113

Open
minion1227 wants to merge 1 commit into
mini-router:mainfrom
minion1227:sn74-minion1227-107
Open

fix(reward): grade x=5, scientific notation, and empty \boxed{} on MATH-500 (#107)#113
minion1227 wants to merge 1 commit into
mini-router:mainfrom
minion1227:sn74-minion1227-107

Conversation

@minion1227

Copy link
Copy Markdown
Contributor

Fixes #107

Problem

normalize_math_answer / _check_math mark several correct MATH-500 answers wrong when the model writes an equivalent-but-unnormalized form. These are frequent answer shapes, so the grader silently under-counts, biasing both the sep-CMA-ES training reward and reported eval accuracy downward — the same false-negative class as the already-fixed thousands-comma bug (#35).

R.score_text("math500", r"\boxed{x=5}", "5")               # was 0.0 -> now 1.0
R.score_text("math500", r"\boxed{5\times10^{3}}", "5000")  # was 0.0 -> now 1.0
R.score_text("math500", r"so the answer is \boxed{}. It is 42", "42")  # was 0.0 -> now 1.0

Fix (three targeted spots)

  • Variable assignment — a boxed x = 5 normalized to x=5 and never matched the bare reference 5. Drop a leading single-variable assignment in normalize_math_answer so the value is graded. Guarded with a (?!=) lookahead so == stays intact, and because </> are not word characters, inequalities x<=5/x>=5 are never touched.
  • Scientific notation — after \times/\cdot become *, 5*10^{3} parsed as neither float nor Fraction, and the sympy fallback reads ^ as XOR. Rewrite a*10^{b} to the canonical aeb so the numeric path parses it (5*10^{3} -> 5e3 -> 5000.0).
  • Empty \boxed{}extract_boxed returns "", which _check_math committed as the blank answer instead of falling back to the last number in the text the way a missing box already does. Treat an empty/whitespace box like a missing box.

Tests

Adds tests/test_reward_math_forms.py (pure stdlib, no torch/GPU/network):

  • the three answer forms grade correct (incl. negative exponents, multi-digit coefficients, \cdot)
  • guards: wrong values, x<=5/x>=5 inequalities, and an empty box with no number in the text still score 0
  • unit checks that normalize_math_answer drops only a leading assignment (not ==/<=) and rewrites scientific notation

Full root suite: 206 passed; ruff check clean on the changed lines.

…TH-500 (mini-router#107)

The math grader marked several equivalent-but-unnormalized MATH-500 answers
wrong, silently under-counting correct answers and biasing both the sep-CMA-ES
training reward and reported eval accuracy downward (same false-negative class
as the fixed thousands-comma bug mini-router#35):

- Variable assignment: a boxed `x = 5` normalized to `x=5` and never matched the
  bare reference `5`. Grade the value, not the restated variable — drop a leading
  single-variable assignment in normalize_math_answer, guarded so `==` and
  inequalities (`x<=5`, `x>=5`) are left intact.
- Scientific notation: after \times/\cdot become `*`, `5*10^{3}` parsed as
  neither float nor fraction, and the sympy fallback read `^` as XOR, so it never
  equaled `5000`. Rewrite `a*10^{b}` to the canonical `aeb` so the numeric path
  parses it.
- Empty \boxed{}: extract_boxed returned "", which _check_math committed as the
  (blank) answer instead of falling back to the last number in the text the way a
  missing box already does. Treat an empty/whitespace box like a missing box.

Adds tests/test_reward_math_forms.py: the three answer forms grade correct, while
wrong values, inequalities, and an empty box with no number still score 0.

Fixes mini-router#107
@github-actions github-actions Bot added miner Miner contribution PR train Training or routing changes labels Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

miner Miner contribution PR train Training or routing changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reward: math grader misses common MATH-500 answer forms (x=5, scientific notation, empty oxed{})

1 participant