Skip to content

fix: harden corpus math normalization - #14

Merged
developer0hye merged 2 commits into
mainfrom
feat/wasm-corpus-60-sync
Mar 9, 2026
Merged

fix: harden corpus math normalization#14
developer0hye merged 2 commits into
mainfrom
feat/wasm-corpus-60-sync

Conversation

@developer0hye

Copy link
Copy Markdown
Owner

What changed

  • harden corpus text normalization so underscore-bearing identifiers stay intact during source snapshot extraction
  • normalize math-heavy headings against extracted Unicode math glyphs
  • make LaTeX math stripping line-aware so literal prose delimiters like $$ no longer swallow later content
  • add targeted regression tests for identifier preservation, heading normalization, inline/block math stripping, and the texify delimiter edge case
  • refresh the 60-entry corpus baselines after re-running validation with the new rules

Why

The validator was producing false negatives on math-heavy READMEs, especially around underscore-bearing identifiers and Markdown lines that mention literal math delimiters. That kept several corpus entries below threshold even when the rendered PDFs were fine.

Verification

  • node --test validation/text_metrics.test.mjs validation/readme_corpus.test.mjs
  • node validation/readme_corpus.mjs bless --tier all --force
  • Full corpus result: 60 total, 0 hard failures, 5 advisory-only remote asset warnings

Related: #13

Signed-off-by: Yonghye Kwon <developer.0hye@gmail.com>
Signed-off-by: Yonghye Kwon <developer.0hye@gmail.com>
@developer0hye
developer0hye merged commit ae4ad99 into main Mar 9, 2026
2 checks passed
@developer0hye
developer0hye deleted the feat/wasm-corpus-60-sync branch March 9, 2026 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant