[mmcky]

Encodes the third inline native review of the Malayalam edition — 44 suggestion blocks from @adisankarmt on QuantEcon/lecture-python-programming.ml#13 (matplotlib) — and adds the mechanism that round pointed at: editor-approved sentence pairs shown to the translator as style examples. Analysis and per-flag dispositions are in QuantEcon/project-translation reports/2026-09-18-ml-matplotlib-review-disposition.md; the editor's open questions are QuantEcon/lecture-python-programming.ml#22.

What the review showed

Every class encoded from round 2 came through clean (terminal punctuation 26 → 0 flags, capitalisation 14 → 0, fixed pairs 10 → 0, connectives 7 → 0; the ml#12 pins held), and he left 22 of 66 Malayalam lines untouched against 5 of 123 in round 2, in the same reply channel. What remains is style: clause-boundary commas (16 flags, six of them nothing but a comma), verb aspect and mood (10), clause order (10). Rules state those badly; his own sentences show them well.

What changes

Area	Change
src/types.ts, src/translator.ts	Glossary.style_examples and formatStyleExamples: pairs render after the glossary terms under STYLE EXAMPLES:, with an instruction to match register, rhythm, punctuation and clause order — never content — and to follow a numbered rule where the two differ. They ride the glossary object, so they reach sync, init, forward and resync with no new input or loader, and they sit in the prompt-cached stable block (a fixed set per glossary version; no per-call retrieval, which would break the cache). A glossary without the key renders byte-identically to before. The reviewer is untouched.
src/language-config.ts	ml rules 23 → 27: clause-boundary commas; verb form by meaning (-തിരിക്കുന്ന, -ഇട്ടുണ്ട്, conditional -ആൽ, -ഏക്കാം); idiom and coined jargon by plain meaning; Further Reading lists stay English. Four rules extended (hortative കരുതാം; fragment bullets, instrument-first, മറ്റൊരു; fixed renderings multiple, explicit, X-like, simple; light verbs remove, label). The hortative rule no longer ends on a dangling colon (the second residual on #301; the lint-regex residual there is untouched).
glossary/ml.json	v0.6.0: +4 terms and 32 style_examples drawn only from the three lectures he has reviewed. No example contains a banned rendering or touches a question still open with him (a test enforces both).
.dev/decisions/	D-2026-09-18-ml-further-reading-lists-stay-english — accepted by @mmcky; prompt rule for now, to move into code once he tells us where the boundary is.
experiments/ml-benchmark/	The regeneration test below, plus five round-3 banned renderings in ml_metrics.py.

New admission test for rules, agreed this round: a pattern is encoded only if it recurs (twice in a round, or across rounds). Two draws of this lecture at near-identical engines shared 9 of 66 lines, and at least six of his 44 edits simply restored the other draw's wording — a singleton flag is as likely sampling noise as engine habit. Held for his answers: the rule-2 boundary (which ordinary words stay English — the round-2 pendulum reversed this round), provide, prefer, draw, the plural of retained nouns, the comma between single-word -ഉം pairs. None blocks a release; they ride the next one, as the ml#12 answers did.

Regeneration test — and what it does not show

experiments/ml-benchmark/arms/2026-09-18-round3-rules-exemplars-sonnet5/: a 2 × 2 of rules × exemplars, three draws per cell, on matplotlib and — new this round — on a held-out lecture (functions, scored against his round-2 text, with no functions pairs among the exemplars). Score is paragraph-level character similarity to his reviewed text; means of three draws.

Arm	matplotlib median	matplotlib commas / para	functions median	functions commas / para
editor's text	1.000	0.71	1.000	0.82
baseline (main)	0.845	0.44	0.814	0.56
+ exemplars only	0.854	0.55	—	—
+ round-3 rules	0.928	0.67	0.809	0.72
+ rules + exemplars	0.934	0.71	0.805	0.70

Read plainly: the in-sample jump is mostly the rules quoting his sentences — on the held-out lecture overall closeness does not move. What transfers is what the rules state generically (comma density 0.56 → 0.72 against his 0.82; every round-3 signature lands in the six in-sample draws, bar one ലളിതമായ slip; Further Reading verbatim 6/6). The exemplars show no measurable closeness effect in any pairing; without the rules they pull the style signatures part-way, with the rules they add nothing this metric can see. They are cheap (cached) and did no measurable harm, so they ship — but as unproven, and dropping the style_examples key from ml.json reverts them without touching code. Two method findings are recorded in the arm README: the seed an editor edited is anchored to his text (0.951 against 0.845 for fresh draws) and is not a baseline, and character similarity is near its ceiling as an instrument — the editor-side series (touch rate, accepted lines, edit magnitude) is the better round-over-round measure.

One of 22 runs was rejected by the structural-parity guard (a dropped {raw} directive) and re-run; it was an exemplar arm, n = 1, recorded in the README so a pattern would be visible.

Verification

Full suite green (1,584), lint and format:check clean, check-dev-refs clean, dist-action/ rebuilt. New tests cover example rendering on every prompt builder, the omission cases, glossary shape, and the rule set. Not yet done: a harness run on the .ml lane with the local bundle — planned before release, per the harness-first policy. Targets v0.29.0; round 4 (numpy) is generated at @v0 after it ships.

🤖 Generated with Claude Code

[Copilot]

🔵 Needs a closer look

The new prompt-rendering code in formatGlossary / formatStyleExamples introduces unintended leading/trailing whitespace, undermining prompt stability and the “byte-identical” omission claim.

Pull request overview

Introduces “style examples” as a new glossary capability to better teach translation style (especially Malayalam), and encodes the third Malayalam native review round by expanding ml prompt rules, updating the ml glossary, and archiving supporting benchmark artifacts.

Changes:

• Add Glossary.style_examples / StyleExample and render editor-approved EN→target sentence pairs in prompts under STYLE EXAMPLES:.
• Update Malayalam rules (23→27) and glossary ml.json to v0.6.0 (new terms + style examples) based on round-3 review outcomes.
• Add/extend tests and archive a regeneration/benchmark arm (scripts + draw outputs + scores) documenting the evaluation.

File summaries

File	Description
src/types.ts	Adds StyleExample type and optional Glossary.style_examples.
src/translator.ts	Renders glossary style examples in the prompt stable block via formatStyleExamples.
src/language-config.ts	Expands Malayalam additional rules (round-3: commas/aspect/idiom/further-reading scope + rule extensions).
src/tests/translator-prompts.test.ts	Tests style-examples rendering/omission across prompt builders and preserves rule ordinal contiguity.
src/tests/language-config.test.ts	Validates ml glossary v0.6.0 contents and ml rule count/constraints.
glossary/README.md	Documents optional style_examples in glossary schema.
glossary/ml.json	Bumps to v0.6.0 with new terms and style_examples set.
experiments/ml-benchmark/scripts/ml_metrics.py	Adds round-3 banned renderings to metrics tokens list.
experiments/ml-benchmark/arms/2026-09-18-round3-rules-exemplars-sonnet5/select_exemplars.py	New script to select eligible style exemplars from reviewed lecture pairs.
experiments/ml-benchmark/arms/2026-09-18-round3-rules-exemplars-sonnet5/scores-matplotlib.json	Archived in-sample scoring output for the benchmark arm.
experiments/ml-benchmark/arms/2026-09-18-round3-rules-exemplars-sonnet5/scores-functions.json	Archived held-out lecture scoring output for the benchmark arm.
experiments/ml-benchmark/arms/2026-09-18-round3-rules-exemplars-sonnet5/score.py	New paragraph-alignment scoring driver to produce the score JSON.
experiments/ml-benchmark/arms/2026-09-18-round3-rules-exemplars-sonnet5/README.md	Documents the benchmark design, results, and incidents.
experiments/ml-benchmark/arms/2026-09-18-round3-rules-exemplars-sonnet5/glossaries/glossary-v0.6.0-terms.json	Archived glossary snapshot used by benchmark runs (terms-only variant).
experiments/ml-benchmark/arms/2026-09-18-round3-rules-exemplars-sonnet5/draws/matplotlib-Aex-draw3.md	Archived generated output for baseline+exemplars arm.
experiments/ml-benchmark/arms/2026-09-18-round3-rules-exemplars-sonnet5/draws/matplotlib-Aex-draw2.md	Archived generated output for baseline+exemplars arm.
experiments/ml-benchmark/arms/2026-09-18-round3-rules-exemplars-sonnet5/draws/matplotlib-Aex-draw1.md	Archived generated output for baseline+exemplars arm.
experiments/ml-benchmark/arms/2026-09-18-round3-rules-exemplars-sonnet5/draws/matplotlib-A-draw3.md	Archived generated output for baseline arm.
experiments/ml-benchmark/arms/2026-09-18-round3-rules-exemplars-sonnet5/draws/matplotlib-A-draw2.md	Archived generated output for baseline arm.
experiments/ml-benchmark/arms/2026-09-18-round3-rules-exemplars-sonnet5/draws/matplotlib-A-draw1.md	Archived generated output for baseline arm.
experiments/ml-benchmark/arms/2026-09-18-round3-rules-exemplars-sonnet5/corpus/matplotlib.en.md	Archived English source corpus used for benchmark alignment/scoring.
experiments/ml-benchmark/arms/2026-09-18-round3-rules-exemplars-sonnet5/align.py	New alignment utility pairing EN↔ML prose paragraphs by structural anchors.
docs/user/glossary.md	Adds user-facing documentation for style_examples behavior and constraints.
CHANGELOG.md	Records the new glossary style examples feature and Malayalam round-3 updates under Unreleased.
.dev/STATE.md	Updates dev state to include ml round-3 in-flight summary.
.dev/log/2026-09-18-ml-round3.md	Adds a dev log entry for ml round-3 encoding and benchmark findings.
.dev/decisions/D-2026-09-18-ml-further-reading-lists-stay-english.md	New decision record capturing Further Reading “stay English” rule for ml.

Review details

• Files reviewed: 46/52 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[mmcky]

Held for the full test programme, now complete — ready for review. Since the PR opened, the arm gained a second instrument and three exemplar variants (d569742), and the "not yet shown" caveat in the body has an answer.

Instrument. Paragraph similarity was flat on the held-out lecture for every arm (0.80–0.81), so judge.mjs was built: Opus 5 (not the Sonnet 5 generator), the English paragraph plus the editor's reviewed Malayalam as reference, two arms' renderings in randomised order, style only, TIE allowed. 2,570 judgements; the first-shown candidate won 0.48–0.54 of decisions, so no position bias. It detects the in-sample rule effect decisively on matplotlib (rules 76 : 24), which similarity also saw.

Held-out verdict (functions, decisions only, two-sided sign test): rules over baseline 55 : 45 (p = 0.19); rules + the shipped 16-pair set over baseline 57 : 43 (p = 0.054); rules vs rules + pairs 53 : 47. Then the variants, nine more draws: rules vs rules + 20 contrastive triples (EN / engine draft / editor's correction) 51 : 49; rules vs rules + 56 pairs 50 : 50; baseline vs baseline + 56 pairs (examples instead of rules) 50 : 50 — that arm reproduced the rules' comma density (0.71 vs 0.72) but not their judged style.

What this PR ships, accordingly: the four rules and the rule extensions (a real, small, consistent edge on unseen text — a preference in roughly 5–7 of every 100 paragraphs); the style_examples mechanism with the small set (the only example form with any positive signal, not significant, cheap, reversible with one key); nothing contrastive and nothing larger — the NOT: rendering one arm used was a local build and is not in this PR. The exemplar line is closed pending round-4 editor data. The ceiling is visible in the numbers: 25–30% of held-out paragraphs come out byte-identical across arms and 30 of 100 sit below 0.70 similarity in every arm — generator sampling variance, not missing instructions — so the next lever on the ml side is lint-and-repair of the rules already present (#260, #301), not more prompt.

Copilot's overview claim about whitespace / byte-identity was checked and is wrong; 9fea195 pins the byte-identical rendering with a test. Full tables and reproduction in the arm README.

[mmcky]

Applied the finding to the PR (0150c6f): the ml style-example set is out; the mechanism stays. By the bar set for rules — nothing ships without a measured improvement — the example-specific increment on unseen text (53 : 47, p = 0.40) does not qualify, so glossary/ml.json v0.6.0 now carries the four round-3 terms only. style_examples remains as a capability (rendered on every write path, byte-identical when absent, tested, documented) for a measured case later; the 32-pair set and every variant stay in the arm archive. Tests, docs, CHANGELOG and the language-config comment say the same. Suite 1,584 green, lint/format clean, dist-action/ rebuilt.

What the PR now ships: the 27 ml rules (preferred 55 : 45 over baseline on a held-out lecture), glossary v0.6.0 (+4 terms), the Further Reading decision record, the style_examples mechanism with no data, the byte-identity test, and the full 31-draw / 2,570-judgement archive. Ready for review.

[Copilot]

🔵 Needs a closer look

It modifies shared prompt-building and prompt-cache behavior affecting every language and write path, states a byte-identity invariant the code does not fully uphold, and the author notes harness verification on the .ml lane is still pending.

Review details

• Files reviewed: 36/74 changed files
• Comments generated: 0 new
• Review effort level: Balanced