fix(flint-py): point run_full_eval at the moved fixture corpus - #81
Open
zl190 wants to merge 1 commit into
Open
Conversation
`tools/run_full_eval.py` exits with "manifest.json not found" and evaluates nothing. `FIXTURES` still resolves to `packages/flint-py/tests/fixtures/`, but the corpus moved to `shared/test-data/` in 1d1ee39 and this constant did not follow. The report path is derived from `FIXTURES.parent`, so repointing the corpus alone would have carried the report out of `tests/` along with it. Giving the report its own `REPORT_DIR` keeps both outputs on the locations where the committed copies already live: `results.json` next to the corpus, `FULL_GALLERY_REPORT.md` under `tests/`. With this applied the tool completes and reports PASS 476 / MISMATCH 180 / PY_ERROR 0 / JS_ERROR 1 / FIXTURE_MISSING 0. The mismatch count agrees with `pytest -q`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
|
this is great, the python library is quite outdated, I think it's good that we should use the new test cases to keep them in sync. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tools/run_full_eval.pyexits withERROR: manifest.json not foundand evaluates nothing:FIXTURESstill points atpackages/flint-py/tests/fixtures/, but the corpus moved toshared/test-data/in1d1ee39.FIXTURESon its own would have dragged the report along with it, since line 345 derived that path asFIXTURES.parent— hence the separateREPORT_DIR.results.jsonandFULL_GALLERY_REPORT.mdto the two locations where the committed copies already live.Context
Found while investigating #80. The eval tool is the instrument that measures
flint-pyagainst the JS reference, so having it run again is the prerequisite for acting on anything in that issue.Changes
packages/flint-py/tools/run_full_eval.py(+8 / −4):FIXTURES→shared/test-data/, with a comment naming the commit that moved it.REPORT_DIR = ROOT / "tests"soFULL_GALLERY_REPORT.mdkeeps landing next to its committed copy rather than following the corpus.Review focus
results.jsongoes toshared/test-data/results.json(which is tracked there already) and the report topackages/flint-py/tests/FULL_GALLERY_REPORT.md(likewise). I inferred the intent from where the committed copies sit — say the word if either belongs elsewhere.Test plan
uv run python tools/run_full_eval.pyfrompackages/flint-py— completes and prints:PASS: 476,MISMATCH: 180,PY_ERROR: 0,JS_ERROR: 1,FIXTURE_MISSING: 0.git statusshows exactlypackages/flint-py/tests/FULL_GALLERY_REPORT.mdandshared/test-data/results.jsonas modified, nothing untracked).MISMATCH: 180againstuv run pytest -q, which reports 180 failures. Two independent instruments, same number.Notes for reviewers
658 / 658 (100.0%)into476 / 656 (72.6%). That drop is what Golden fixtures in shared/test-data no longer match the JS reference (705/705) #80 is about; it is not caused by this change, which touches only where the tool reads and writes.N > 0it contradicts itself. Left alone to keep this diff to the paths; happy to send a follow-up.