Skip to content

Add LLM translation eval suite and page cost tooling - #465

Draft
Raghaddahi wants to merge 2 commits into
mainfrom
feature/llm-transaltions
Draft

Add LLM translation eval suite and page cost tooling#465
Raghaddahi wants to merge 2 commits into
mainfrom
feature/llm-transaltions

Conversation

@Raghaddahi

Copy link
Copy Markdown
Member

Relates to #446

Description

Adds tooling to evaluate and price LLM translation for the Wagtail user guide, supporting the model/provider decision in #446.

Why this approach:

  • Standalone scripts instead of fork changes — the eval suite calls the real fork translate_text (same SYSTEM_PROMPT, HTML sanitisation and validation as the admin "Translate with AI" button) without writing anything to the DB, so it runs safely against production data.
  • Scripts are now language-parameterized via EVAL_TARGET_LANGUAGE/EVAL_LANG_SLUG env vars, with outputs namespaced under out/<lang>/, so the same pipeline covers Arabic and French.
  • scripts/page_cost.py is a read-only cost report straight from saved TranslationLog rows (no LLM calls); page selection is anchored on the max StringTranslation.updated_at to match a page's translation run.
  • Generated eval outputs (out/, per-model cost CSVs) are gitignored; scorecards from the runs are kept in the report on this branch for the Automated translations via wagtail-localize #446 discussion.

Areas needing careful review:

  • The LLM-as-judge (run_judge.py) pairwise scoring in one call (candidates A/B/C shuffled per segment) — judge reliability is the crux of the eval.
  • The check_rules.py deterministic checks (HTML tag/id preservation, <b>/<i> kept English, truncation) — these feed the scorecard.
  • ruff.toml gains per-file-ignores for scripts/*.py (T20 prints, E402 Django-shell imports) since these are intentional CLI patterns.

Testing

  • poetry run ruff check scripts/ — all checks passed.
  • poetry run ruff format --check scripts/ — clean after ruff format.
  • Re-ran summarize.py against existing French candidate data inside the container; output matches the pre-refactor scorecard exactly (rank logic intact after lint cleanup).
  • Full eval runs (collect → rules → judge → summarize) for both Arabic and French completed successfully against the current prompt; scorecards are in TRANSLATION_EVAL_REPORT.md on this branch.

QA left to do: full 392-segment eval once a model decision is needed; no unit tests for the scripts (they are run inside the Docker web container via manage.py shell).

AI usage

  • Agent: Claude, via opencode
  • AI generated the code and this PR description; human-reviewed by the contributor.

@Raghaddahi
Raghaddahi marked this pull request as draft August 5, 2026 11:41
@RealOrangeOne
RealOrangeOne temporarily deployed to wagtail-guid-feature-ll-ieqhbn August 5, 2026 11:42 Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants