Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 31 additions & 1 deletion .okf/build/test-gates.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,9 @@ the gate was blind to it by 50x - and passed, which is worse than failing
page is ~1e-6, not 0 (three runs of `services/_technologies`: 0.013838252,
0.013838252, 0.013837770 - about 2 px of 2,073,600), so the floor has to clear
that; 0.0001 (~207 px) sits 100x above it, fails that copy edit, and goes green
again when it is reverted. The noise claim is
again when it is reverted. **That 1e-6 is the macOS-local figure and does NOT
generalise - see the per-platform noise section below before you lower any
tolerance.** The noise claim is
checkable without re-running the copy edit: two independent runs of the same
test reported difference_levels identical to 16 decimal places (0.6893909143518518
for `services/_footer` every time), and 2 of the 7 screenshots stayed GREEN at
Expand Down Expand Up @@ -759,3 +761,31 @@ and confirm the page loads its own fingerprinted `.min.<hash>.css` before
trusting any computed value (caught 2026-08-21 during the pilot column-fix
verification; the earlier "inconclusive" production check had exactly this
cause).

# The noise floor is per-PAGE, not per-repo (measured 2026-08-22)

The ~1e-6 above was measured on macOS, locally, on a static page. CI amd64 is
different, and the difference decides whether a tolerance may be lowered.

**Method**: two `update-baselines` dispatches of the SAME commit, ~12 hours
apart, compared file by file. Anything that moves between two recordings of
identical content is render noise by definition.

**Result**: **135 of 147 Linux baselines were byte-identical.** CI is
deterministic for ~92% of pages. The remaining 12 split cleanly:

| Pages | difference_level | What they are |
|---|---|---|
| 11 | 0.0003 - 0.002 | mermaid diagrams, syntax-highlighted code, course + about + clients pages - font and SVG rasterisation |
| 1 | 0.0425 (max_delta 238) | `mobile/blog/index/_pagination` - a real content difference, not noise |

**The consequence that matters: every one of those 11 noisy pages already pins
`tolerance: 0.03` at its call site.** Those pins are LOAD-BEARING on CI, not
legacy padding left over from the blind era. Lowering them toward the 0.0001
default - which an earlier plan proposed as a tidy-up - would redden CI
permanently on pages that have no defect.

**Rule.** Before lowering any pinned tolerance, measure that page's noise on
the platform that will judge it, by recording it twice on the same commit. The
floor for a page is its measured noise, not the repo default. A tolerance that
looks like slack may be the only thing absorbing a rasteriser.
28 changes: 28 additions & 0 deletions .okf/log.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,34 @@ make it green: restructure same-day entries under one heading, and add
`timestamp` to the 23 concepts missing it (anchored to each file's last commit
time, which is verifiable - never invented).

## 2026-08-22 - the noise floor is per-page, and the pinned tolerances are load-bearing

Chasing the last red Linux key produced a better finding than the fix. Four
hypotheses died on the way - arm64-vs-amd64 drift (the files measured
pixel-identical against a control), #560's tolerance drop (that test pins its
own 0.03, so the default never applied), the PR-merge-commit checkout, and a
date-gated post - and each one cost a cycle because it was reasoned rather
than measured.

What the measurement says: two `update-baselines` dispatches of the SAME commit
twelve hours apart left **135 of 147 Linux baselines byte-identical**. CI is
deterministic for ~92% of pages. Eleven font/SVG-heavy pages (mermaid, syntax
highlighting, course, about, clients) moved by 0.0003-0.002, and the twelfth,
`mobile/blog/index/_pagination`, moved by 0.0425 with a max channel delta of
238 - a real content difference, which is why it was the only failure.

**Every one of those eleven pins `tolerance: 0.03` at its call site.** Those
pins are not legacy padding from the blind era; they are the only thing
absorbing a rasteriser on CI. An earlier plan to measure and lower the section
tolerances toward the new 0.0001 default would have reddened CI permanently on
pages with no defect. That plan is now scoped out, and the rule is written
where the tolerance lives: measure a page's noise on the platform that judges
it - by recording it twice on one commit - before lowering anything.

Also corrected: the ~1e-6 noise figure in test-gates and in the code comment
was macOS-local and read as universal. Same overclaim shape the #566 reviewer
caught one layer down.

## 2026-08-22 - the site sells one thing, and title claims are now a canon rule

Paul changed the positioning: the site sells an embedded team of senior
Expand Down
7 changes: 6 additions & 1 deletion test/application_system_test_case.rb
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,12 @@ class ApplicationSystemTestCase < Minitest::Test
# the baseline fossilized. Measured run-to-run noise is ~1e-6 (three runs of
# services/_technologies: 0.013838252 twice, then 0.013837770 - about 2 px of
# 2,073,600), so the floor only has to clear that: 0.0001 is ~207 px. Calls
# needing real slack (font-swap, animation) still pin their own tolerance.
# needing real slack (font-swap, animation) still pin their own tolerance -
# and on CI those pins are LOAD-BEARING, not padding. That 1e-6 is macOS
# local; two CI recordings of the same commit left 135 of 147 baselines
# byte-identical but moved 11 font/SVG-heavy pages by 0.0003-0.002, every one
# of which pins 0.03 here. Do not lower a pinned tolerance without measuring
# that page's noise on the platform that judges it (.okf/build/test-gates.md).
DEFAULT_SCREENSHOT_CONFIG = {tolerance: 0.0001}.freeze

private
Expand Down
Loading