diff --git a/.okf/build/test-gates.md b/.okf/build/test-gates.md index e3a1f34f1..ac86e6d5d 100644 --- a/.okf/build/test-gates.md +++ b/.okf/build/test-gates.md @@ -120,7 +120,9 @@ the gate was blind to it by 50x - and passed, which is worse than failing page is ~1e-6, not 0 (three runs of `services/_technologies`: 0.013838252, 0.013838252, 0.013837770 - about 2 px of 2,073,600), so the floor has to clear that; 0.0001 (~207 px) sits 100x above it, fails that copy edit, and goes green -again when it is reverted. The noise claim is +again when it is reverted. **That 1e-6 is the macOS-local figure and does NOT +generalise - see the per-platform noise section below before you lower any +tolerance.** The noise claim is checkable without re-running the copy edit: two independent runs of the same test reported difference_levels identical to 16 decimal places (0.6893909143518518 for `services/_footer` every time), and 2 of the 7 screenshots stayed GREEN at @@ -759,3 +761,31 @@ and confirm the page loads its own fingerprinted `.min..css` before trusting any computed value (caught 2026-08-21 during the pilot column-fix verification; the earlier "inconclusive" production check had exactly this cause). + +# The noise floor is per-PAGE, not per-repo (measured 2026-08-22) + +The ~1e-6 above was measured on macOS, locally, on a static page. CI amd64 is +different, and the difference decides whether a tolerance may be lowered. + +**Method**: two `update-baselines` dispatches of the SAME commit, ~12 hours +apart, compared file by file. Anything that moves between two recordings of +identical content is render noise by definition. + +**Result**: **135 of 147 Linux baselines were byte-identical.** CI is +deterministic for ~92% of pages. The remaining 12 split cleanly: + +| Pages | difference_level | What they are | +|---|---|---| +| 11 | 0.0003 - 0.002 | mermaid diagrams, syntax-highlighted code, course + about + clients pages - font and SVG rasterisation | +| 1 | 0.0425 (max_delta 238) | `mobile/blog/index/_pagination` - a real content difference, not noise | + +**The consequence that matters: every one of those 11 noisy pages already pins +`tolerance: 0.03` at its call site.** Those pins are LOAD-BEARING on CI, not +legacy padding left over from the blind era. Lowering them toward the 0.0001 +default - which an earlier plan proposed as a tidy-up - would redden CI +permanently on pages that have no defect. + +**Rule.** Before lowering any pinned tolerance, measure that page's noise on +the platform that will judge it, by recording it twice on the same commit. The +floor for a page is its measured noise, not the repo default. A tolerance that +looks like slack may be the only thing absorbing a rasteriser. diff --git a/.okf/log.md b/.okf/log.md index d3c3d611a..ab7abd0d4 100644 --- a/.okf/log.md +++ b/.okf/log.md @@ -51,6 +51,34 @@ make it green: restructure same-day entries under one heading, and add `timestamp` to the 23 concepts missing it (anchored to each file's last commit time, which is verifiable - never invented). +## 2026-08-22 - the noise floor is per-page, and the pinned tolerances are load-bearing + +Chasing the last red Linux key produced a better finding than the fix. Four +hypotheses died on the way - arm64-vs-amd64 drift (the files measured +pixel-identical against a control), #560's tolerance drop (that test pins its +own 0.03, so the default never applied), the PR-merge-commit checkout, and a +date-gated post - and each one cost a cycle because it was reasoned rather +than measured. + +What the measurement says: two `update-baselines` dispatches of the SAME commit +twelve hours apart left **135 of 147 Linux baselines byte-identical**. CI is +deterministic for ~92% of pages. Eleven font/SVG-heavy pages (mermaid, syntax +highlighting, course, about, clients) moved by 0.0003-0.002, and the twelfth, +`mobile/blog/index/_pagination`, moved by 0.0425 with a max channel delta of +238 - a real content difference, which is why it was the only failure. + +**Every one of those eleven pins `tolerance: 0.03` at its call site.** Those +pins are not legacy padding from the blind era; they are the only thing +absorbing a rasteriser on CI. An earlier plan to measure and lower the section +tolerances toward the new 0.0001 default would have reddened CI permanently on +pages with no defect. That plan is now scoped out, and the rule is written +where the tolerance lives: measure a page's noise on the platform that judges +it - by recording it twice on one commit - before lowering anything. + +Also corrected: the ~1e-6 noise figure in test-gates and in the code comment +was macOS-local and read as universal. Same overclaim shape the #566 reviewer +caught one layer down. + ## 2026-08-22 - the site sells one thing, and title claims are now a canon rule Paul changed the positioning: the site sells an embedded team of senior diff --git a/test/application_system_test_case.rb b/test/application_system_test_case.rb index 6e4bea31e..445a0c8b8 100644 --- a/test/application_system_test_case.rb +++ b/test/application_system_test_case.rb @@ -78,7 +78,12 @@ class ApplicationSystemTestCase < Minitest::Test # the baseline fossilized. Measured run-to-run noise is ~1e-6 (three runs of # services/_technologies: 0.013838252 twice, then 0.013837770 - about 2 px of # 2,073,600), so the floor only has to clear that: 0.0001 is ~207 px. Calls - # needing real slack (font-swap, animation) still pin their own tolerance. + # needing real slack (font-swap, animation) still pin their own tolerance - + # and on CI those pins are LOAD-BEARING, not padding. That 1e-6 is macOS + # local; two CI recordings of the same commit left 135 of 147 baselines + # byte-identical but moved 11 font/SVG-heavy pages by 0.0003-0.002, every one + # of which pins 0.03 here. Do not lower a pinned tolerance without measuring + # that page's noise on the platform that judges it (.okf/build/test-gates.md). DEFAULT_SCREENSHOT_CONFIG = {tolerance: 0.0001}.freeze private