Add LLM translation eval suite and page cost tooling - #465
Draft
Raghaddahi wants to merge 2 commits into
Draft
Conversation
Raghaddahi
marked this pull request as draft
August 5, 2026 11:41
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relates to #446
Description
Adds tooling to evaluate and price LLM translation for the Wagtail user guide, supporting the model/provider decision in #446.
Why this approach:
translate_text(sameSYSTEM_PROMPT, HTML sanitisation and validation as the admin "Translate with AI" button) without writing anything to the DB, so it runs safely against production data.EVAL_TARGET_LANGUAGE/EVAL_LANG_SLUGenv vars, with outputs namespaced underout/<lang>/, so the same pipeline covers Arabic and French.scripts/page_cost.pyis a read-only cost report straight from savedTranslationLogrows (no LLM calls); page selection is anchored on the maxStringTranslation.updated_atto match a page's translation run.out/, per-model cost CSVs) are gitignored; scorecards from the runs are kept in the report on this branch for the Automated translations via wagtail-localize #446 discussion.Areas needing careful review:
run_judge.py) pairwise scoring in one call (candidates A/B/C shuffled per segment) — judge reliability is the crux of the eval.check_rules.pydeterministic checks (HTML tag/id preservation,<b>/<i>kept English, truncation) — these feed the scorecard.ruff.tomlgainsper-file-ignoresforscripts/*.py(T20prints,E402Django-shell imports) since these are intentional CLI patterns.Testing
poetry run ruff check scripts/— all checks passed.poetry run ruff format --check scripts/— clean afterruff format.summarize.pyagainst existing French candidate data inside the container; output matches the pre-refactor scorecard exactly (rank logic intact after lint cleanup).TRANSLATION_EVAL_REPORT.mdon this branch.QA left to do: full 392-segment eval once a model decision is needed; no unit tests for the scripts (they are run inside the Docker web container via
manage.py shell).AI usage