Visual gate: default tolerance 0.02 → 0.0001 (the gate was blind by 50x) - #560
Merged
Conversation
DEFAULT_SCREENSHOT_CONFIG demanded that ~41,472 pixels of a 1920x1080 frame differ before an assertion failed, so real changes passed green. services/_testimonials-header sat at difference_level 0.005923 through an entire dark-surface recolour and reported no change. Measured on DesktopSiteTest#test_services at clean HEAD, 7 screenshots compared per run: 0.02 gives 2 failures, this commit gives 3. Run-to-run noise is 0.0 - repeat runs return difference_levels identical to 16 decimal places, and 2 of the 7 stay green at the new floor - so 0.0001 (~207 px) clears jitter without tripping on it. The ~30 calls that pin their own tolerance are untouched; only the floor for calls pinning nothing moved. Passing quietly is worse than failing, because it fossilizes the baseline: the gem writes the fresh capture over the baseline PNG, then a PASS restores the git-HEAD image back over it. A green run therefore never refreshes anything, and every sub-tolerance change leaves the committed PNG showing the old render until something large enough to fail arrives and the accumulated drift gets accepted in one lump nobody can attribute. Corollary now documented: only a RED run leaves a committable candidate, and deleting the PNG first does nothing since the base is read from git HEAD. SECTION_CONFIGS is KEPT at 0.02 on purpose. Its 7 keys equalled the old default, making it a silent no-op; lowering the default inverts that into a load-bearing shield holding ~22 section screenshots at the old tolerance until their drift is measured. Deleting it here would drop those to 0.0001 unmeasured and would also corrupt the headline number: with the shield removed the same test fails 5 instead of 3, which is +3 rather than the +1 this change actually costs. The two extra are exactly the screenshots whose names hit a SECTION_CONFIGS key (services/_use-cases 0.017498, services/_technologies 0.013838), since keys match the tail after "/_" - /_technologies matches technologies, /_testimonials-header does not match testimonials. That mapping went from inert to safety-critical with no coverage, so it moves to test/support/screenshot_section_config.rb and is pinned by test/unit/screenshot_section_config_test.rb. It had to move: requiring application_system_test_case.rb from a unit test boots Hugo, Capybara and that file's dirty-fixtures abort. The guard was verified by mutation - splitting the key on "-" so _testimonials-header matches the testimonials shield turns the suite red, reverting turns it green - and the system path re-measured identical (same 3 failures, same difference_levels) after the extraction. test-gates.md also documents two blind spots tolerance cannot fix. Captures are viewport-only at scroll top (1920x1080 desktop, 360x800 mobile, test/support/setup_capybara.rb:85-86), so below-fold content is invisible at ANY tolerance - a table-cell edit measures difference_level exactly 0 on mobile even at tolerance 0, and every failure region here caps at the viewport height. And perceptual_threshold = 2.0 (test/support/setup_snap_diff.rb:25) means vips only counts a pixel as differing above CIE dE00 2.0, so a sub-threshold recolour contributes zero differing pixels at any tolerance. Coverage fixes are out of scope. Live doctrine swept truthful: the tutorial documented a SCREENSHOT_TOLERANCE env var that exists nowhere in the codebase (and claimed a 1% default the code never had), a helper named assert_stable_problematic_screenshot with its own section and three call sites, and a per-OS tolerance-multiplier table - none of which exist in test/, bin/ or lib/; visual-qa-criteria pinned 0.02. A doctrine file names its own mechanisms, so its fiction looks executable and gets copied. Gates: bin/rake test:unit 289 runs, 6090 assertions, 0 failures. bin/hugo-build clean. okf_validate: 0 errors, 111 -> 112 warnings (exactly the one section-7 date heading this log entry adds). Screenshot suites are red on the stale #540 baselines by design - that is the gate biting: Screenshot does not match for 'desktop/services/_testimonials-header': ({"difference_level":0.005922550154320988}) which passed silently before this commit. Those baselines are NOT re-recorded here; that is the follow-up's job. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The list taught the 2% blindness in present tense with 'left as-is deliberately' - three claims made false by the previous commit. Edit prescribed verbatim by the delta re-clear. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Screenshot diffs detected
|
This was referenced Aug 21, 2026
pftg
added a commit
that referenced
this pull request
Aug 21, 2026
…nal-cto/ (#562) Pilot B of the three-register design comparison (10.01): Paul's in-progress Claude Design artifact ("Fractional CTO - Artifact Editorial", DESKTOP_1440 + MOBILE_390) translated from inline styles onto the /next/ rail. Paper #fbf9f8 with #f3f0ed tint bands, white cards on 1px #d8d3cd, sections divided by 1px #e5e1dd hairlines, deep red #a91918 (NOT the standard ruby - the artifact uses the darker cut and keeps #cc342d as its hover only), ZERO border-radius anywhere, Source Serif 4 headlines over Hanken Grotesk body with JetBrains Mono for every eyebrow and label. COPY IS UNCHANGED FROM PILOT A - that is the point. The stub was copied, not rewritten: `diff` between the two stubs is six lines (title, description, `register`, `font`). The desktop and mobile screenshot tests assert the same strings as pilot A's, so a copy edit that reaches only one pilot fails. The comparison varies the register and nothing else, or it measures nothing. Register fidelity, and the two places it could not be literal: - The artifact has NO dark band; the settled anatomy does, and the anatomy is held constant across pilots, so the band is inverted rather than dropped. This is where pilot A shipped an AA miss and it is not repeatable here: the accent is 7.05:1 on paper but 2.31:1 on ink #1b1c1c. On the band, text takes a light neutral (#c9c2ba, 9.69:1 / #b9b2aa, 8.14:1), the accent survives only as a lightened non-text rule (#e07a6e, 5.84:1), and the filled CTA inverts to paper - an accent fill on the #232424 card is ~2.2:1, a button with no visible edge (WCAG 1.4.11). - Devices the anatomy has no slot for were translated, not invented: the artifact's screen-label chip becomes the brand mark, its boxed STATUS pill becomes the step numeral, its 32x2 accent rule opens every band (suppressed in the hero, where the artifact starts with the eyebrow alone), its square ink list marker replaces the disc, and the tech tags become square hairline chips because the register has no rounded corner anywhere. Gates widened, because a second pilot silently disarms the first pilot's: test/unit/next_rail_test.rb had the testimonial-verbatim and tenure-derived gates pinned to the pilot-A path, so pilot B could have smoothed the Wozniak quote undetected. Both now iterate next/pilots/**/index.html through one helper that owns the "found nothing" assertion. Verified by breaking it: smoothing pilot B's quote fails naming next/pilots/editorial/fractional-cto/index.html. Three faces self-hosted as ONE static/css/fonts-editorial.css keyed by register rather than by family, because the stub's `font:` key carries one stylesheet and one preload and three files would mean editing the shared landing.html for one pilot. All three families are variable, so each is one latin binary that every declared weight instantiates from. Only shipped weights are rendered - and that is established by WALKING computed font-weight in the browser, not by grepping declarations. The grep missed a real defect that review caught: JetBrains Mono ships 400 only, but .rr-brand-mark inherited 700 from .rr-brand and the nav .rr-btn inherited 600 from :where(.rr-nav-links a), so Chrome fake-bolded both and the same mono label rendered heavier in the nav than in the hero. font-weight is now pinned on both. The walk resolves fontFamily per node and histograms fontWeight: 55 mono nodes, 53 at 400 + 2 synthesized BEFORE the fix, 55 at 400 after; Hanken Grotesk 400 (40) + 600 (10) and Source Serif 4 600 (13) + 700 (5), all shipped weights. Worth noting the visual gate DID catch the fix at difference_level 0.00048 desktop / 0.0034 mobile, in a 12px-tall nav region - it would have passed silently under the old 0.02 default (PR #560). Evidence: bin/hugo-build clean; bin/rake test:unit 292 runs, 6140 assertions, 0 failures; scoped desktop + mobile next_editorial screenshots green against these committed macOS baselines; rendered blockquote diffed against data/testimonials.yaml is empty; noindex present and the canon Clutch profile link present on both built pilots; the banned sweep ($400K, $5K/$15K, 0.5%, 3-6 months, six-month search, 30 days' notice, free code audit, 32 client, five working days, 3+ years, $190, 18 years, monthly cost + the voice-guide set) returns zero matches across the built pilots dir. Browser pass at 1440 and at 390: zero console messages, zero 404s, and walking every text node's computed colour against its resolved background gives 117 elements checked per viewport with zero contrast failures (worst 5.06:1); no page horizontal overflow, the comparison table scrolls inside its own box, h1 lands at the artifact's 33px on mobile. Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
pftg
added a commit
that referenced
this pull request
Aug 21, 2026
…nc (#565) Board carries the three live /next/ URLs, the four-persona ballots with self-objections, the Editorial 4-1 artifact head-to-head, the fable recommendation with its graft package, and the decision protocol - Paul picks. Ledger rows updated for #561-#564 and the #560 gate repair; OKF gains the BASE_URL local-production-check rule. Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 21, 2026
pftg
added a commit
that referenced
this pull request
Aug 22, 2026
…ed (#582) STATUS.md's Test/CI row still said "one known red (stale linux baseline)". That key was a real content difference and is fixed - re-recorded on master, where recorder and tester see the same tree, verified green by CI run 32565008850 with zero failing keys. What replaced it is genuinely open and needs Paul: on an ARM Mac dtest now legitimately fails 8 codeblocks screenshots, because the container renders arm64 while the committed linux baselines come from CI on amd64. That was invisible until #578 fixed dtest comparing nothing at all from a worktree, so it is pre-existing rather than a regression. Three options recorded with their trade-off, plus the contradiction to reconcile first: .dev/compose.yml declares platform linux/amd64 on the `t` service while bin/dc exports DOCKER_DEFAULT_PLATFORM=linux/arm64/v8. Also refreshed the 2608 sprint summary, which still claimed "#560-#567" and "CI Linux red went from 22 keys to 1" - it is 0 now, and the range runs to #578. Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
One constant + its first test coverage + a truthful doc layer. The visual gate's default tolerance let ~41,472 pixels of a 1920×1080 frame change before failing — a real recolour sat at difference_level 0.0059 and reported green — and on every PASS the capybara-screenshot-diff gem restores the git-HEAD image over the fresh capture, so baselines fossilize. Diagnosed end-to-end today (three agents: probe → mechanism → measurement); every number below reproduced independently by an adversarial reviewer to 16 decimal places.
DEFAULT_SCREENSHOT_CONFIGtolerance 0.02 → 0.0001 (noise floor measured 0.0 — repeat runs return identical difference_levels; ~30 call sites pin their own tolerance and are unaffected)test/support/screenshot_section_config.rb(pure module) + 5-case test, mutation-verified twice (author and reviewer, different mutations).okf/build/test-gates.md: 0.02 → 2 fails, shipped → 3, shield-deleted → 5 on the services page)SCREENSHOT_TOLERANCEenv var,assert_stable_problematic_screenshot+ its 2.5% threshold recommendations, per-OS multiplier table — all grep-verified absent from code); 20.10 §4 marked FIXEDWhy now
The register-pilot lane (design comparison on /next/) is gated on a trustworthy visual instrument: pilot A's author proved a text change invisible to the desktop gate, which triggered the diagnosis.
Evidence
desktop/services/_testimonials-headerfails at 0.005923 (was green at 0.02)🤖 Generated with Claude Code
https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ