Skip to content

The page for teachers was still showing a corpus of ninety and a rate of zero - #85

Merged
peopleworks merged 1 commit into
mainfrom
why-page-recalibrated
Sep 7, 2026
Merged

peopleworks merged 1 commit into
mainfrom
why-page-recalibrated

Conversation

@peopleworks

Copy link
Copy Markdown
Owner

README.md:15 is the first link in the file — "Are you a teacher? Start here →" — and it points at why.html, which is live.

The page said The project publishes
ninety texts 296, since 1 September
zero flagged — a green 0, 46px 2 of 296
less than 4.1% 2.4%
engine 0.4.0 0.7.0
uniform rhythm in 27.8% 32.1%

Nothing on it was a lie: the footer dates every figure to 5 August. But the headline argument a teacher reads is a green zero, on the page whose entire purpose is that this project publishes how often it is wrong. The 1 September work existed to say the rate had stopped being zero. This page never heard.

It carried its own trigger, and the trigger did nothing

From a comment written in August, still in the file:

Every figure comes from published-calibration.json… If that file changes, this page is stale and must be regenerated by hand — there is no build step behind it, deliberately.

That file changed twice. Asking the next person to remember is not a guarantee.

WhyPageTests now reads the same published-calibration.json the engine ships and requires the page to agree — in both languages, since one half going stale alone is the likelier failure. Seven of its eight guards fail against the page exactly as it is live right now. The eighth is the "asks nothing of any server" rule, which was already true and is guarded because a redraw is when it would slip.

Redrawn, not retyped

  • The dot chart is 296 circles with two filled, not ninety with none.
  • The interval band runs 0.2–2.4% around a measured 0.7%, not 4.1% around zero. Rendered and inspected in a browser; a label collision found that way and fixed.
  • The English bar shows 2.7% — the bound at the recommended threshold, not the 1.4% best bound. That substitution is the exact defect the committee caught in the evidence report on 1 September, and a guard here refuses it explicitly.

The best change is a paragraph that inverted

It used to concede that the corpus was "published articles, not student work" and that the difference "is not measured here". It is measured now: 206 of the 296 are classroom essays by adults learning English — the group that fares worst in every column, and the reason the boundary is 30 rather than 25. What is still unmeasured is the student essay in Spanish, and the page says that instead.

For the audience this page exists to reach, that is the strongest thing the project can currently say, and it was sitting unsaid.

Also #78, everywhere outside Docs/Blog

"before generative models existed" overclaims — GPT-3 is 2020. The worst instance was in Docs/Calibration/corpus.json, where the note sat on fifty Wikipedia revisions from 2021: the exact case the sentence does not cover. Fixed in Fetch.cs so it stays fixed, plus the MCP report tool, four Core comments, PARAPHRASE.md and a test.

The blog articles keep their wording. They are dated studies, per the standing policy.

432 tests.

🤖 Generated with Claude Code

https://claude.ai/code/session_015PEbbiYSNPw7jE3LrPNhyF

… of zero

README line 15 is the first link in the file: "Are you a teacher? Start here".
It points at why.html, which is live, and which said:

  ninety texts        the corpus has been 296 since 1 September
  zero flagged        the published figure is 2 of 296
  less than 4.1%      the published bound is 2.4%
  engine 0.4.0        the engine is 0.7.0

Nothing on it was a lie — the footer dates every figure to 5 August. But the
headline argument a teacher reads is a green zero, on the page whose whole
purpose is that this project publishes how often it is wrong. The 1 September
work existed to say the rate had stopped being zero. This page never heard.

The page carried its own trigger, in a comment written in August: "If that file
changes, this page is stale and must be regenerated by hand." The file changed
twice. Asking the next person to remember is not a guarantee, which is why
WhyPageTests now reads published-calibration.json and requires the page to
agree with it — in both languages, since one going stale alone is the likelier
failure. Seven of its eight guards fail against the page as it was live today.

Redrawn, not retyped: the dot chart is 296 circles with two filled, the interval
band runs 0.2–2.4% around a measured 0.7% instead of 4.1% around zero, and the
English bar shows 2.7% — the bound at the recommended threshold, not the 1.4%
best bound, which is the trap the committee caught in the report on 1 September.

The best change is a paragraph that inverted. It used to concede that the corpus
was "published articles, not student work" and that the difference was not
measured. It is now measured: 206 of the 296 are classroom essays by adults
learning English, the group that fares worst, and the reason the boundary is 30
rather than 25. What is still unmeasured is the student essay in Spanish, and
the page now says that instead.

Also #78, everywhere outside Docs/Blog: "before generative models existed"
overclaims, because GPT-3 is 2020. The worst instance was in the corpus manifest,
where the note sat on fifty Wikipedia revisions from 2021 — the exact case the
sentence does not cover. Fixed in Fetch.cs so it stays fixed, and in the MCP
report tool, four Core comments, PARAPHRASE.md and a test. The blog articles keep
their wording: they are dated studies.

432 tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015PEbbiYSNPw7jE3LrPNhyF
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant