Close the three fault-injection gaps: 8 of 8 - #576
Merged
Merged
Conversation
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…sted The 2026-08-22 audit planted eight defects and the suite caught three. Three of the five misses get gates here. Every one was injected, watched fail, reverted and watched pass - green alone proves a test runs, not that it works. D - a frozen tenure number. The rendered assertion cannot tell `derived` from `frozen` while both read "18+" in 2026, and saying so in the test body is honest but guards nothing. The freeze is plainly visible one layer down, so NextRailTest now asserts both halves: the rendered value, and that every pilot stub carries `derived: tenure` with no stat frozen at today's derived string. Not a config test - the hardcoded literal IS the defect class, the way the 2026-08-14 canon audit found the founding year wrong in eight places because each kept its own copy. G - the PurgeCSS trap the audit called unguardable. It is unguardable by the VISUAL gate, which builds production and so purges exactly as production does, matching its own baseline. It is trivially visible to a set diff: styled in the register's source CSS, present on an element in the rendered HTML, absent from the bundle the page links. next_purge_guard_test.rb is keyed on that mechanism rather than on `th`, so the next element type Hugo stops recording is covered without an edit, and it reads the bundle from the page's own href - stale fingerprinted bundles sit in the same directory. B and the contrast hole - computed styles, not per-band screenshots. The audit's own recommendation buys four more baselines per pilot, each needing its noise floor measured on the platform that judges it, and still returns a pixel delta rather than a ratio - so an intentional recolour and an AA regression stay indistinguishable. next_pilot_contrast_test.rb asserts the comparison table's "theirs" column computes the register's own --*-muted token, then walks every visible text run for 4.5:1 (3:1 large, SC 1.4.3) and every control fill for 3:1 against its surround (SC 1.4.11). ~1.3s for three pages, no baselines, wired into test:critical. It closes the hole the audit flagged separately: nothing in this suite measured contrast as a number, and a 3.33:1 black-on-ruby button shipped this rail on human review alone. One mistake worth keeping. The muted-column assertion was first written as "differs from the sibling cell and from body colour" and PASSED the exact #564 injection - in that register the "ours" column IS body colour and the injected ink is a third value. Wrong for two runs, green both times; only the injection said so. Injections and failures, all reverted before the next: D `derived: tenure` -> `value: "18+"` the tenure stat must be `derived: tenure` so layouts/next/landing.html computes it from site.Params.foundingYear D a second stat frozen at "18+" alongside the derived one Expected ["Years of average developer experience"] to be empty G class="rr-th-theirs" on a <th> + a matching rule Expected ["rr-th-theirs"] to be empty control: the SAME class moved onto the <td> in that row goes green, so it flags the purge and not merely a new class B .rr-td-muted -> var(--ed-ink), below the fold computes rgb(27, 28, 28), not --ed-muted (rgb(107, 101, 96)). It resolves to --ed-ink A .rr-btn-primary label -> var(--ed-ink) on the accent fill 2.31:1 (needs 4.5:1) text a.rr-btn.rr-btn-primary A' .rr-td-muted -> #b5b0ab, below the fold - invisible to every screenshot 2.15:1 (needs 4.5:1) on five cells A" a near-paper button fill with a readable label 1.12:1 (needs 3.0:1) fill a.rr-btn.rr-btn-primary Each walk asserts it measured something (>50 text pairs, >0 fills, a non-empty styled-and-rendered set) - a walk that finds nothing is empty, and empty passes. Gates: test:unit 289 runs / 6187 assertions / 0 failures in 1.42s (287 / 6143 / 1.33s before - the purge guard reuses the build the suite already makes); test:critical 38 runs / 0 failures, 55 screenshots compared, no failures, no baseline rewritten; bin/hugo-build clean; standardrb clean on the new files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the audit (#572) and the gate fixes (#574). The three defects the suite could not see are now caught, each verified by injecting the defect, watching it fail, and reverting — per the rule #574 added.
D — frozen tenure (was blind until 2027-01-01)
The rendered assertion cannot tell derived from frozen while both read "18+". Added the missing half: the stub must use
derived: tenure, not a literal. The hardcoded number IS the defect class — it is how the founding year ended up wrong in eight places on 2026-08-14, each with its own copy.Injection →
the tenure stat must be \derived: tenure` so layouts/next/landing.html computes it from site.Params.foundingYear`. A second injection froze a different stat, proving the second assertion is live rather than shadowed by the first.G — the PurgeCSS trap (the audit called it unguardable)
It is unguardable by the visual gate, which builds production and therefore purges exactly as production does. A unit test catches it: classes styled in the register's source CSS ∩ present on a rendered element − present in the fingerprinted bundle that page links. Non-empty = a silently-purged rule. Keyed on the mechanism, not on
<th>.Injection → flags
rr-th-theirs. The control is what makes it believable: the same class moved onto a<td>— an elementwriteStatsdoes record — goes green. It flags the purge, not merely a new class.Costs three file reads; the unit suite already builds production.
B + the contrast hole — computed styles, not screenshots
Deliberately not per-band screenshots: those inherit the tolerance problem (font/SVG-heavy bands need a measured floor) and still return a pixel delta rather than a ratio.
--*-mutedtoken. Injection →computes rgb(27,28,28), not --ed-muted ... the comparison stopped comparing (#564).Injections caught: the audit's above-fold defect A at 2.31:1; a below-fold defect at y=1090 that the screenshot gate reports as 0 difference; and a 1.12:1 button fill under SC 1.4.11.
This also closes the hole the audit flagged separately: defect A was previously caught only as an anonymous pixel change, indistinguishable from an intentional recolour. Nothing measured contrast until now — and this session shipped a real 3.33:1 button that only human review caught.
Worth reading in the report
The first version of the muted-column assertion ("differs from the sibling cell") passed the injection — the injected ink was a third value, so it found a difference and approved it. Two green runs on a wrong test, exposed only by breaking it. That is the case for the rule.
Gates
unit 289 runs / 6187 assertions / 0 failures in 1.42s (was 287/6143/1.33s) · critical 38 runs, 55 screenshots compared, no failures, fixtures untouched · hugo-build clean · standardrb clean.
Honest limits
The contrast walk runs at 1920×1080 only (a mobile-breakpoint-only colour is uncovered) and covers the three register pilots, not the whole site. Both are loops, not redesigns.
🤖 Generated with Claude Code
https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ