Fault-injection audit: 3 of 8 planted defects caught, 5 missed - #572
Merged
Conversation
Injected eight defects modelled on ones this repo has shipped, one at a time, each followed by the narrowest gate that should catch it. Predictions were written before any run; two were wrong and both are recorded as wrong. Caught: above-fold contrast (A), smoothed testimonial (C), lost noindex (E). Missed: below-fold visual (B), frozen tenure (D), banned phrase (F), the <th> purge trap (G), broken internal link (H). Two findings the predictions did not anticipate: - test:links checks almost nothing. The production build renders internal links absolute, and lychee --offline excludes all http(s) URIs, so 133,874 of 149,516 links are excluded; the homepage checks exactly one link, its own #main-content anchor. A --remap of the host onto the build dir was verified to catch the planted break. - MarketingCopyTest's rendered ratchet has three hits of slack (baseline 14, actual 11), which is what let the injected banned phrase through. Below-fold coverage quantified for task #12: the viewport-sized capture gates 28.3% of the editorial pilot (1080px of 3822px). The injected #564-class defect sits at y=1090 - ten pixels outside the frame. No baselines re-recorded; every injection reverted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This was referenced Aug 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Eight defects modelled on ones this repo has actually shipped, injected one at a time, predictions written down BEFORE measuring. 3 caught, 5 missed. Two predictions were wrong (F and H) — both predicted caught, both missed, and both wrong for reasons bigger than the defect.
Two findings that outrank the exercise
1.
bin/rake test:linksinspects almost nothing. Production renders internal links absolute (https://jetthoughts.com/...), andlychee --offlineexcludes every http(s) URI by design. Site-wide: 133,874 of 149,516 links excluded. On the homepage: 141 total, 140 excluded, 1 OK — and that one is its own#main-contentskip-link anchor. The job has been green because it checks nothing. Fix verified rather than asserted: adding--remapturns that page into35 OK, 1 Errorand names the planted break.2. Below-fold blindness, quantified. The capture is viewport-sized: at 1920x1080 it gates 1080px of a 3822px page — 28.3% covered, 71.7% dark. The planted #564-class defect sits at y=1090, ten pixels outside the frame. Four of the five content bands begin below the fold.
Matrix
<th>(purge trap)Why F was missed
MarketingCopyTestcarriesRENDERED_BASELINE = 14against an actual count of 11 — three hits of slack, measured with the gate itself (baseline 0 → clean 11, injected 12). SeparatelySURFACESnever globscontent/next/**, so the rendered pass is the entire v2 rail's only cover.Why G cannot be caught in principle
Four counts prove the mechanism: rule in source CSS = 1, rule in built CSS = 0, class in rendered HTML = 1, class in
hugo_stats.json= 0. The visual gate buildsENVIRONMENT=production, so it purges exactly as production does and matches its baseline.Caveat on A
It was caught as a pixel change, not a contrast failure — the same gate fires identically on an intentional recolour. Nothing in this suite measures contrast ratios.
🤖 Generated with Claude Code
https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ