Re-record 127 stale baselines and delete the tolerance shield - #566
Merged
Conversation
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…pting The macOS visual suite has been red on master since #540. `bin/rake test:system` at clean HEAD: 79 runs, 26 failures, 134 screenshots compared, 48 failures. All 48 are correctly-detected staleness, not regressions - nothing here was verified by difference_level alone. Why they went stale: a green run restores the git-HEAD image over the fresh capture, so any change small enough to pass fossilizes the baseline. #540 was large enough to fail and simply was not re-recorded; the rest had been accumulating under the old 0.02 default since #520 and #528. Classification (baseline-vs-candidate pixel transitions, not diff levels): 42 #540 dark-surface recolour. rgb(0,0,0) -> rgb(20,17,15) accounts for 94-100% of every changed pixel in each file (--surface-ink #14110f). Footers, CTA bands, contact_us, both pagination frames, codeblock frames - the same surfaces #540's own commit message lists as failing. 3 Post `description` backfilled by #520, which the post template renders as a grey lead above the body, pushing content down. Verified the rendered text is the frontmatter string verbatim, e.g. stimulus-keyboard-event-filter: "In the last update of Stimulus v3.2.2, the ability to filter the event by PageDown and PageUp keys was added." (desktop+mobile code_highlight_post, inline_style_post) 2 Course module eyebrow added by #552 - "MODULE 1 - HYPOTHESIS & SMOKE TEST" now renders in ruby above the H1, shifting the page down. (desktop+mobile course/chapter) 1 services/_testimonials-header: a 1px vertical offset from #528's eyebrow consolidation. #528 touched services-critical.css and content-block.css and re-recorded only the two sections that exceeded 0.02; the sections below the eyebrow shifted 1px, passed under the old default, and fossilized. Measured, not inferred: realigning the baseline by dy=+1 drops the residual to exactly 0.0000, i.e. the content is pixel-identical and only its position moved. Evidence that these are stale baselines rather than flaky rendering: two consecutive runs of DesktopSiteTest#test_services reported difference_levels identical to 17 decimal places (_technologies 0.013838252314814815, _use-cases 0.01749807098765432, _testimonials-header 0.005922550154320988), and a second full-suite run produced the same 48-file set. One instrument error worth recording, since it nearly became a finding: the first transition script loaded each baseline from a reused temp filename, and libvips caches operations by filename - so runs 2..n silently re-compared image 1. It reported services/_footer's baseline as a pure-black frame. A brightness audit over all 155 tracked macOS baselines refuted it (darkest mean is 69/255, none below 1.0), and reloading from git buffers fixed it. The control that keeps the rewritten script honest: an unchanged baseline (desktop/vibe_code_rescue) reports exactly 0 changed pixels. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ
…easured SECTION_CONFIGS was introduced one day ago as an explicitly TEMPORARY shield holding ~22 section screenshots at the old 0.02 tolerance while their drift was unmeasured, with the exit condition written into its own comment: "delete it once they are measured and re-recorded." Measured both ways on the full macOS system suite, same environment, only this variable changed: shield in place 134 screenshots compared, 48 failures shield emptied 134 screenshots compared, 50 failures Cost of deleting it: +2 screenshots, both on /services/ - desktop/services/_use-cases (0.017498) and desktop/services/_technologies (0.013838). Both are the same 1px vertical offset from #528's eyebrow consolidation already classified in the previous commit: realigning baseline to candidate by one pixel drops the residual to exactly 0.0000 in several bands, so the content is pixel-identical and only its position moved. Both are re-recorded here, which is the last thing the exit condition asked for. Deleted: SECTION_CONFIGS, screenshot_config_for, extract_section_key, test/support/screenshot_section_config.rb and its unit test. That file existed so the table could be unit-tested without booting Hugo or Capybara; with the table gone there is no logic left to test, only a constant. DEFAULT_SCREENSHOT_CONFIG moves onto ApplicationSystemTestCase unchanged at 0.0001. Per-call tolerances are untouched - ~30 calls pin their own, mostly 0.03, and those still win over the default. Gates, all on this exact tree: bin/rake test:unit 287 runs, 6143 assertions, 0 failures bin/hugo-build clean, 8 validators passed bin/rake test:system 79 runs, 218 assertions, 0 failures [snap_diff] 134 screenshots compared, no failures The suite was run with the shield deleted BEFORE these two baselines were accepted, and reported exactly the 2 predicted failures and no others - so the deletion is measured, not assumed. .okf/build/test-gates.md is updated: the shield paragraph is replaced by the measurement, and a misattribution is corrected. That file claimed all five failing /services/ screenshots were the #540 dark-surface recolour; only the two large ones are. The three small ones carry 0.13% or less of the black->ink transition and are #528's 1px shift. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ
The re-record was dispatched to CI on this branch (run 32526977795, native
amd64 - the only trusted way to record linux/ baselines; local ARM Docker
plants false drift) and succeeded. FORCE_SCREENSHOT_UPDATE re-records
EVERYTHING, so the bot commit touched 83 files and needed screening.
Screened with the gate's OWN instrument rather than by eye or by file size:
capybara-screenshot-diff counts a pixel as differing only above
perceptual_threshold = 2.0 dE00, and the default tolerance is now 0.0001 of
the frame. Computing that same libvips dE00 fraction for every file in the bot
commit splits it cleanly:
keep 77 would FAIL the gate, so the rewrite is real
drop 6 difference_level exactly 0.00000000 - would have PASSED
The 6 dropped are pure encoder churn with zero perceptual difference:
desktop+mobile course/landing, desktop+mobile blog/special/codeblocks/text,
clients/agent-inbox/_overview, clients/single-full.
Of the 77 kept, 8 are NEW keys with no linux baseline at all - friday_report
(desktop+mobile) and the three register pilots next/{editorial,precision,
rescue-room} (desktop+mobile) - and the rest span 0.000139 to 0.973.
Why the Linux set is broader than the macOS one (77 vs 50), which looks wrong
until you check it: earlier PRs re-recorded only the macOS side. #528 committed
macos/mobile/services.png and left its linux/ twin stale, so linux/mobile/services
carries a visible vertical shift that macOS no longer has - confirmed by
painting the dE00 mask over the capture: identical content, offset position.
The reverse also holds - linux/desktop/blog/special/inline_style_post barely
moves (0.000139) because the Linux blog baselines were already re-recorded
after #520, while macOS was 25% stale. Neither OS is a subset of the other.
Two kept files were inspected directly rather than accepted on their number:
linux/desktop/404.png (0.001235) is a 4px-wide strip at the right edge - a
scrollbar artifact, page content identical, and 4 x 640 px is exactly the
reported difference_level; linux/mobile/services.png (0.1389) is the #528
shift above. Both render correctly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ
… a subset of the other Records the Linux leg of the re-record in .okf/log.md and lifts two rules into test-gates.md that were learned doing it: - Screen a FORCE_SCREENSHOT_UPDATE record with the gate's own instrument (the libvips dE00 fraction above perceptual_threshold 2.0, compared to the 0.0001 default), not by eye or file size. 83 files split 77 keep / 6 drop with no judgement call. - Neither OS baseline set is a subset of the other, because earlier PRs re-recorded only the OS they ran on. Do not reason about one from the other. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ
Independent review measured what the evidence actually supports: - run-to-run noise is ~1e-6, not 0 (test-gates said 0.0 while the code comment in the same PR said 1e-6 - one PR, two numbers) - realigning 1px zeroes SEVERAL BANDS, not the page: a sub-pixel shift re-rasterizes glyphs, so _use-cases still differs in 22,895 px after the best alignment. Content identical on inspection; the conclusion holds, the sentence did not - "nothing unexplained" is really "every dominant cause explained" - three files carry a secondary 1px shift and two carry #540's sibling --ruby-on-ink token - 20.10 still pointed at test/support/screenshot_section_config.rb, which this PR deletes Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ
pftg
force-pushed
the
baseline-rerecord
branch
from
August 21, 2026 21:39
3d5ea96 to
bbe61c1
Compare
Contributor
Screenshot diffs detected
|
pftg
added a commit
that referenced
this pull request
Aug 22, 2026
…d the one red (#567) The repositioned pilot copy lived only in a scratchpad that dies with the session, so it is now 20.09 with its apply checklist and the three claims still awaiting Paul. Ledger carries the positioning change, #566's result, and the residual Linux failure with the evidence that it is a stale baseline rather than flake - plus the two wrong diagnoses it defeated, so the next session does not repeat them. Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
pftg
added a commit
that referenced
this pull request
Aug 22, 2026
Reflection on #566: I read a run-level conclusion of success while the job had concluded failure, and told Paul a re-run had passed when it had not. The same session also had a monitor fire a false 'settled' because an empty check list contains no 'pending' to wait on. Both are the same mistake - trusting a summary signal over the specific one, and reading 'no data yet' as 'done'. Costly here because the screenshot job is routinely expected-red and every merge decision depends on reading it precisely. Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
pftg
added a commit
that referenced
this pull request
Aug 22, 2026
… it justified (#593) The codeblocks drift was stale baselines; close it and cancel the work it justified Re-recorded linux/ on master (dc11791, all 55) and ran local bin/dtest against it: 55 screenshots compared, no failures. Debian ARM and Ubuntu ARM are pixel-identical. The drift that produced five explanations over three weeks was old amd64/Chrome-152 recordings compared against current renders. Because a green gate and a blind gate look the same, the instrument was checked before the result was believed: forcing background-color #ff0000 on .post-article .highlight pre produced 16 failures - all 8 desktop plus all 8 mobile codeblocks keys and nothing else - and reverting returned it to 0. Clean 0 -> injected 16 -> reverted 0. The consequence worth the most is a deletion. The GHCR container-publishing plan - publish this 2.54 GB image, run CI inside it - existed solely to close a Debian-vs-Ubuntu gap by construction. There is no gap, so it is cancelled rather than left in the doc as a tempting someday. The rejected alternative, a split where CI owned pixel truth and dtest was merely behavioural with divergent keys screened, is dead too: an expected-red list is what rotted into "everything is expected red" before #566. What replaces both: bin/dtest is authoritative for DETECTING visual defects, so a local red is a real defect and there is no expected-red list. CI keeps RECORDING, because the agreement is measured rather than structural - Debian and Ubuntu upgrade freetype independently, so it can end with nothing in this repo changing. Reopen only on a dtest red that a master re-record then makes green. Two rules added, both about the shape rather than this drift: - Suspect the baselines BEFORE the mechanism. All five theories were stories about a mechanism; none was a check on the instrument. "Are these current?" is the cheapest question available and would have ended it in a day. - Two observers agreeing proves nothing when they share the input under test. Local and CI failing the same 8 keys read as corroboration and was an artifact - both were reading the same stale baselines, so every hypothesis predicted red on both. Also records why this family and not another: dense monospace text is where any rendering difference lands first, so the codeblocks screenshots are the suite's canary, not an arbitrary victim - which is why they kept looking like evidence for whichever theory was current. And docs 20.10's O3 entry is corrected; it asserted "the 10 red Linux baselines are NOT stale", which was exactly backwards and was repeated for three weeks. Gates: OKF 0 ERRORs, bash -n clean, unit 289 runs / 6185 assertions / 0 failures, hugo-build green. Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First green macOS visual suite since #540.
79 runs, 218 assertions, 0 failures/[snap_diff] 134 screenshots compared, no failures.Why they were stale
#560 established the mechanism: the gate's default tolerance was ~50x blind AND on every PASS the gem restores the git-HEAD image over the fresh capture, so baselines fossilized. The reds were never flake and never a bad machine — they were real, correctly-detected changes nobody could re-record while the gate kept restoring the old bytes.
What was accepted, and on what evidence
Every one of the 50 macOS re-records was classified by pixel-transition histogram, never by difference_level:
rgb(0,0,0)→rgb(20,17,15)= 94–100% of changed pxdescriptionbackfillZero unexplained diffs — nothing was accepted as "probably drift".
Shield deleted, on a measurement
#560 introduced
SECTION_CONFIGSas a documented temporary shield holding ~22 section screenshots at the old 0.02 "until their drift is measured and re-recorded". That exit condition is now met, and the cost was measured rather than assumed: shield in place = 48 failures, shield emptied = 50 — cost +2, both the #528 1px offset. The suite was re-run with the shield deleted before those two baselines were accepted, and reported exactly the two predicted failures and no others.DEFAULT_SCREENSHOT_CONFIGsurvives at 0.0001.Linux
CI
update-baselinesdispatch on this branch (run 32526977795, native amd64), then screened with the gate's own instrument (libvips dE00 vs the 0.0001 default): 83 recorded → 77 kept, 6 dropped (difference_level exactly 0 = encoder churn). The 8 new keys includefriday_reportand all threenext/*register pilots.Two corrections to prior documented beliefs
test-gates.mdmisattributed all five failing/services/screenshots to Phase 1a.4: dark surfaces onto --surface-ink, --ruby-on-ink, and two live AA fixes #540; only two are — the other three are Phase 1a.4: one eyebrow style (4 variants → 1) #528's 1px shift (≤0.13% black→ink transition). Corrected in the bundle.Instrument error caught before it became a finding
The first transition script reused a temp filename; libvips caches operations by filename, so runs 2..n silently re-compared image 1 and reported a baseline as a pure-black frame. Refuted by a brightness audit of all 155 tracked macOS baselines (darkest mean 69/255). The rewritten script carries a permanent control: an unchanged baseline must report exactly 0 changed pixels. Also measured: run-to-run noise is ~1e-6, not 0 — still 100x under the floor, and the comment now says what was observed.
🤖 Generated with Claude Code
https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ