From b4c53118978b1ab65a83f63d3150cff92fb92e30 Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Sat, 22 Aug 2026 11:59:14 +0200 Subject: [PATCH] Gate fault-injection audit: 3 of 8 defects caught Injected eight defects modelled on ones this repo has shipped, one at a time, each followed by the narrowest gate that should catch it. Predictions were written before any run; two were wrong and both are recorded as wrong. Caught: above-fold contrast (A), smoothed testimonial (C), lost noindex (E). Missed: below-fold visual (B), frozen tenure (D), banned phrase (F), the purge trap (G), broken internal link (H). Two findings the predictions did not anticipate: - test:links checks almost nothing. The production build renders internal links absolute, and lychee --offline excludes all http(s) URIs, so 133,874 of 149,516 links are excluded; the homepage checks exactly one link, its own #main-content anchor. A --remap of the host onto the build dir was verified to catch the planted break. - MarketingCopyTest's rendered ratchet has three hits of slack (baseline 14, actual 11), which is what let the injected banned phrase through. Below-fold coverage quantified for task #12: the viewport-sized capture gates 28.3% of the editorial pilot (1080px of 3822px). The injected #564-class defect sits at y=1090 - ten pixels outside the frame. No baselines re-recorded; every injection reverted. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ --- ...te-fault-injection-2026-08-22-reference.md | 159 ++++++++++++++++++ 1 file changed, 159 insertions(+) create mode 100644 docs/20-29-testing-qa/20.11-gate-fault-injection-2026-08-22-reference.md diff --git a/docs/20-29-testing-qa/20.11-gate-fault-injection-2026-08-22-reference.md b/docs/20-29-testing-qa/20.11-gate-fault-injection-2026-08-22-reference.md new file mode 100644 index 000000000..975a31afd --- /dev/null +++ b/docs/20-29-testing-qa/20.11-gate-fault-injection-2026-08-22-reference.md @@ -0,0 +1,159 @@ +# Gate coverage measured by fault injection (2026-08-22) + +**Method.** Eight defects modelled on real ones this repo has shipped were +injected one at a time onto a branch cut from `origin/master`, each followed by +the narrowest gate that should catch it, then reverted. Predictions were written +and committed to this file BEFORE any gate was run - the point of the exercise is +to find where the mental model of the gates diverges from their behaviour, and a +prediction written after the fact measures nothing. + +Every injection was reverted with `git checkout -- ` and the tree verified +`git status --porcelain` empty before the next one. No baseline was re-recorded. + +## Predictions (written first, before any run) + +| # | Defect | Narrowest gate | Predicted | Reasoning at prediction time | +|---|---|---|---|---| +| A | `.rr-btn-primary { color: var(--ed-ink) }` - dark CTA text on the accent fill, hero, above fold | `DesktopSiteTest#test_next_editorial` screenshot | **CAUGHT** | Baseline `macos/desktop/next/editorial.png` is 1920x1080, tolerance floor 0.0001 (~207px of 2,073,600). Recolouring the hero CTA label changes a few thousand pixels inside the captured fold. | +| B | `.rr-table tbody .rr-td-muted { color: var(--ed-ink) }` - comparison table's muted column, below the fold | same screenshot test | **MISSED** | Capture is viewport-only (baseline is exactly the 1920x1080 emulated viewport, not a full-page image) and `test_next_editorial` does not call `preload_all_images`/scroll. Anything past 1080px is outside the frame. | +| C | Testimonial smoothed: "helping us to find" -> "helping us find" | `bin/rake test:unit` (`NextRailTest#test_testimonial_quote_is_canon_verbatim`) | **CAUGHT** | The test asserts the rendered blockquote is a verbatim substring of `data/testimonials.yaml`. Dropping a word breaks the substring relation. | +| D | `derived: tenure` replaced by hardcoded `value: "18+"` | `bin/rake test:unit` | **MISSED** | The test's own comment declares the HONEST LIMIT: in 2026 the derived value `2026-2008 = 18` is character-identical to the frozen `18+`, so `assert_includes values, "18+"` passes on both. The gate cannot distinguish them until 2027-01-01. | +| E | `noindex: false` in the `/next/` cascade | `bin/rake test:unit` (`test_every_register_pilot_is_noindexed_and_unlisted`) | **CAUGHT** | Asserted against RENDERED ``, not frontmatter. `private: true` is untouched so the sitemap half still passes; the robots half should fail. | +| F | Canon-banned phrase ("world-class") in `content/next/pilots/editorial/fractional-cto.md` | `bin/rake test:unit` (`MarketingCopyTest`) | **CAUGHT by the rendered pass, MISSED by the source pass** | `SURFACES` does not glob `content/next/**`, so the source pass is blind to it. `RENDERED_GLOBS` includes `next/**/*.html` and the ratchet fails when the count exceeds `RENDERED_BASELINE = 14`. | +| G | `class` on a `` in `layouts/next/landing.html` + a matching CSS rule | any | **MISSED by every automated gate** | Hugo's `writeStats` does not record class attrs on ``, so PurgeCSS strips the rule. The visual gate builds with `ENVIRONMENT=production` (`bin/build-if-stale`), which exports `HUGO_ENVIRONMENT` and therefore runs PurgeCSS - so the test build strips the rule exactly as production does, renders identically to the baseline, and passes green. `css_orphan_guard_test` is file-level, not rule-level. Caught by human review in #563. | +| H | Internal link pointed at a path that does not exist | `bin/rake test:links` (lychee, offline) | **CAUGHT** | The task globs every emitted `*.html` and passes them to `lychee --offline --root-dir`, so an unresolvable internal href fails the build. `lychee 0.24.2` is on PATH locally, so this runs without CI. | + +## Results + +**Score: 3 caught, 5 missed.** Two predictions were wrong (F and H) - both were +predicted CAUGHT and both were missed, and both for reasons that matter more than +the defects themselves. + +A control run preceded the visual injections: `DesktopSiteTest#test_next_editorial` +on the unmodified tree reported `1 screenshot compared, no failures`. Without that, +a later red proves nothing about the gate. + +| # | Defect | Gate run | Predicted | Actual | Evidence | +|---|---|---|---|---|---| +| A | CTA text -> `--ed-ink` on the accent fill | `bin/test test/system/desktop_site_test.rb -n test_next_editorial` | CAUGHT | **CAUGHT** | `Screenshot does not match for 'desktop/next/editorial': ({"area_size":2508.0,"region":[402.0,523.0,611.0,535.0],"difference_level":0.0004576581790123457...})` - 4.6x the 0.0001 floor, region well inside the fold | +| B | `.rr-td-muted` -> `--ed-ink`, below fold | same | MISSED | **MISSED** | `1 runs, 3 assertions, 0 failures` on BOTH desktop and mobile, while the built bundle the page loads contains `tbody .rr-td-muted{color:var(--ed-ink)}` - the defect shipped and the gate was green | +| C | Testimonial "helping us to find" -> "helping us find" | `bin/test test/unit/next_rail_test.rb -n test_testimonial_quote_is_canon_verbatim` | CAUGHT | **CAUGHT** | `the rendered quote is not a verbatim slice of the canon testimonial. Expected "...helping us to find problems..." to include "...helping us find problems..."` | +| D | `derived: tenure` -> `value: "18+"` | `bin/test test/unit/next_rail_test.rb` | MISSED | **MISSED** | `7 runs, 93 assertions, 0 failures`. Control: the same edit with `"19+"` DOES fail (`Expected ["19+", "5", "8+"] to include "18+"`), proving the frontmatter reaches the render and the gate is live - it simply cannot separate a frozen 18 from a derived 18 in 2026 | +| E | `noindex: false` in the `/next/` cascade | `bin/test test/unit/next_rail_test.rb` | CAUGHT | **CAUGHT** | Two failures. `Expected "index, follow, max-snippet:-1, max-video-preview:-1, max-image-preview:large" to include "noindex"` - the rendered robots meta flipped to indexable, which is the actual production consequence | +| F | "world-class" in the editorial pilot stub | `bin/test test/unit/marketing_copy_test.rb` | CAUGHT (rendered pass) | **MISSED** | `3 runs, 7 assertions, 0 failures` while `grep -c "world-class"` on the built page returns 1. Measured with the gate itself (baseline temporarily set to 0): clean tree = `baseline 0, now 11`, injected tree = `baseline 0, now 12`, against `RENDERED_BASELINE = 14` | +| G | `class` on a `` + a matching CSS rule | visual gate + `bin/rake test:unit` | MISSED | **MISSED** | Visual: `1 screenshot compared, no failures`. Units: `287 runs, 6143 assertions, 0 failures`. Mechanism proven by four counts: rule in source CSS = 1, rule in BUILT CSS = 0, class in rendered HTML = 1, class in `hugo_stats.json` = 0 | +| H | Internal link -> `/clients/agent-inbox-typo/` | `bin/rake test:links` | CAUGHT | **MISSED** | `149516 Total, 31937 Unique, 15642 OK, 0 Errors, 133874 Excluded`. Per-link: `[EXCLUDED] https://jetthoughts.com/clients/agent-inbox-typo/ ... This is due to your 'exclude' values` | + +## Where the predictions were wrong + +**F - the ratchet has slack.** The prediction assumed a ratchet sits on the +current count. It does not: `RENDERED_BASELINE = 14` while the tree actually +renders 11 hits. Three new banned phrases can be added to rendered marketing +surfaces before the gate says a word, and the injected one landed in that gap +(11 -> 12). The file's own comment predicted this failure mode - "a ratchet left +slack lets the win regress silently" - and then the number was left at 14 when +the real count fell to 11. The source pass missed it for the predicted reason +(`SURFACES` does not glob `content/next/**`), so the rendered pass was the only +thing standing, and its slack let the defect through. + +**H - `--offline` excludes every absolute URL, and the production build makes +every internal link absolute.** The prediction treated "Broken Internal Links" as +an internal-link gate. It is not one in practice. `bin/hugo-build` with +`ENVIRONMENT=production` renders internal links against the real baseURL, so they +emit as `https://jetthoughts.com/...`; `lychee --offline` excludes all http(s) +URIs by design. The homepage measures the effect exactly: `141 Total, 140 +Excluded, 1 OK`, and the single checked link is the in-page `#main-content` +anchor from the skip link. Site-wide, 133,874 of 149,516 links are excluded and +`--verbose` prints zero `[OK]` lines. The job passes because it checks almost +nothing, not because the links are sound. + +The fix was verified, not assumed. Re-running the same scan with a remap of the +public host onto the local build directory turns the same page from "all +excluded" into a real check that finds the injected defect: + +``` +lychee --offline --root-dir \ + --remap "https://jetthoughts.com/(.*) file:///\$1" /**/*.html + +[ERROR] file:///clients/agent-inbox-typo/ | File not found. + | Remaps: https://jetthoughts.com/clients/agent-inbox-typo/ --> file:///clients/agent-inbox-typo/ +41 Total, 32 Unique, 35 OK, 1 Error, 5 Excluded +``` + +35 links checked instead of ~0, and the planted break is found. + +## What nothing guards + +1. **Everything below 1080px on a landing page.** The capture is viewport-sized, + not full-page. Measured on `/next/pilots/editorial/fractional-cto/` at + 1920x1080: page height 3822px, captured 1080px - **28.3% of the page is + gated, 71.7% is not**. The first `.rr-td-muted` cell sits at y=1090, ten + pixels past the frame. This is the #564 defect class, and it quantifies task + #12. Consequence: any colour, spacing, or layout regression in the comparison + table, the timeline, the client band, the testimonial, or the closing CTA + ships green. Four of the page's five bands begin below the fold (y=1524, + 2219, 2909). + +2. **A frozen tenure number, until 2027-01-01.** `18+` frozen and `18+` derived + are the same characters this year. The gate is honest about this in its own + comment, so it is a known-and-accepted gap rather than a surprise - but it is + a real hole right now, and the pilot stubs are exactly where a hardcoded + number gets pasted. + +3. **The next three banned marketing phrases.** `RENDERED_BASELINE = 14` against + an actual 11. Consequence: three regressions on the pages a prospect reads + are pre-authorised. Source-side, `content/next/**` is not globbed at all, so + the rendered pass is the only cover the `/next/` rail has. + +4. **Any CSS rule keyed to a class that only ever appears on a ``** (and, by + the same mechanism, any element whose class attrs Hugo's `writeStats` does not + record). The rule is stripped from the production bundle, the markup still + ships, and no gate compares intent to output. The visual gate cannot help + here even in principle: it builds with `ENVIRONMENT=production`, so it purges + exactly as production does and renders the same un-styled result as the + baseline. Caught by human review in #563; nothing automated stands behind that. + +5. **Broken internal links, site-wide.** See H above. The gate name promises + coverage the configuration does not deliver - 89.5% of all links excluded, and + the ones actually checked are in-page anchors. + +## Recommended (cheapest fix first) + +1. **Add `--remap` to `test:links`** (Rakefile `task :links`), mapping the site's + baseURL host onto the build directory. One flag, and the existing job starts + doing the thing its name claims. Verified above to catch a planted break. + Expect it to surface a backlog of real breaks on first run - that is the point, + and it should be triaged before the task is made blocking again. +2. **Lower `RENDERED_BASELINE` from 14 to 11.** One number. Closes the pre- + authorised regressions immediately. Worth doing in the same pass that adds a + note to re-tighten it whenever a batch is cleared, since that is the step that + was skipped last time. +3. **Add `content/next/**/*.md` to `MarketingCopyTest::SURFACES`.** One glob line. + Gives the `/next/` rail source-side cover so it does not depend solely on a + ratchet with slack. +4. **Below-fold capture for the landing pilots** (task #12). The expensive one, so + scope it: these pages are four-to-five discrete bands, and a per-band + screenshot keyed to the existing `.rr-band` elements costs four more captures + per pilot and covers the 71.7% that is currently dark. A whole-page capture is + the tempting version and the wrong one - it re-introduces the font-swap and + lazy-image flake the current design deliberately avoids. +5. **The `` purge trap has no cheap automated fix** and should stay a review- + checklist item rather than gain a bespoke gate. The honest cheap mitigation is + a note where it bites - `layouts/next/landing.html` already carries one - plus + preferring `nth-child` over classes on ``. If it recurs a third time, + the proportionate gate is a build-time check that every class in a template's + markup survives into the built bundle, which is a bigger piece of work than + the defect has so far justified. + +## Method notes + +- Every injection was reverted and the tree checked `git status --porcelain` + empty before the next; the final tree is identical to `origin/master` apart + from this file. No baseline was re-recorded. +- Exit codes were captured directly (`cmd > log; echo $?`), never through a pipe - + `cmd | tail` reports tail's status, not the command's. +- Green runs were only trusted after proving the injection reached the artifact + under test: for B and G by grepping the BUILT CSS bundle the page actually + loads, for D by a `19+` sentinel that made the same gate fail, for F by grepping + the built HTML. A pass at a pinned tolerance proves "under that tolerance", not + "identical", and a pass on an unbuilt change proves nothing at all.