Skip to content

Fault-injection audit: 3 of 8 planted defects caught, 5 missed #572

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pftg merged 1 commit into from
Aug 22, 2026
Merged

Fault-injection audit: 3 of 8 planted defects caught, 5 missed #572

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
159 changes: 159 additions & 0 deletions docs/20-29-testing-qa/20.11-gate-fault-injection-2026-08-22-reference.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,159 @@
# Gate coverage measured by fault injection (2026-08-22)

**Method.** Eight defects modelled on real ones this repo has shipped were
injected one at a time onto a branch cut from `origin/master`, each followed by
the narrowest gate that should catch it, then reverted. Predictions were written
and committed to this file BEFORE any gate was run - the point of the exercise is
to find where the mental model of the gates diverges from their behaviour, and a
prediction written after the fact measures nothing.

Every injection was reverted with `git checkout -- <path>` and the tree verified
`git status --porcelain` empty before the next one. No baseline was re-recorded.

## Predictions (written first, before any run)

| # | Defect | Narrowest gate | Predicted | Reasoning at prediction time |
|---|---|---|---|---|
| A | `.rr-btn-primary { color: var(--ed-ink) }` - dark CTA text on the accent fill, hero, above fold | `DesktopSiteTest#test_next_editorial` screenshot | **CAUGHT** | Baseline `macos/desktop/next/editorial.png` is 1920x1080, tolerance floor 0.0001 (~207px of 2,073,600). Recolouring the hero CTA label changes a few thousand pixels inside the captured fold. |
| B | `.rr-table tbody .rr-td-muted { color: var(--ed-ink) }` - comparison table's muted column, below the fold | same screenshot test | **MISSED** | Capture is viewport-only (baseline is exactly the 1920x1080 emulated viewport, not a full-page image) and `test_next_editorial` does not call `preload_all_images`/scroll. Anything past 1080px is outside the frame. |
| C | Testimonial smoothed: "helping us to find" -> "helping us find" | `bin/rake test:unit` (`NextRailTest#test_testimonial_quote_is_canon_verbatim`) | **CAUGHT** | The test asserts the rendered blockquote is a verbatim substring of `data/testimonials.yaml`. Dropping a word breaks the substring relation. |
| D | `derived: tenure` replaced by hardcoded `value: "18+"` | `bin/rake test:unit` | **MISSED** | The test's own comment declares the HONEST LIMIT: in 2026 the derived value `2026-2008 = 18` is character-identical to the frozen `18+`, so `assert_includes values, "18+"` passes on both. The gate cannot distinguish them until 2027-01-01. |
| E | `noindex: false` in the `/next/` cascade | `bin/rake test:unit` (`test_every_register_pilot_is_noindexed_and_unlisted`) | **CAUGHT** | Asserted against RENDERED `<meta name="robots">`, not frontmatter. `private: true` is untouched so the sitemap half still passes; the robots half should fail. |
| F | Canon-banned phrase ("world-class") in `content/next/pilots/editorial/fractional-cto.md` | `bin/rake test:unit` (`MarketingCopyTest`) | **CAUGHT by the rendered pass, MISSED by the source pass** | `SURFACES` does not glob `content/next/**`, so the source pass is blind to it. `RENDERED_GLOBS` includes `next/**/*.html` and the ratchet fails when the count exceeds `RENDERED_BASELINE = 14`. |
| G | `class` on a `<th>` in `layouts/next/landing.html` + a matching CSS rule | any | **MISSED by every automated gate** | Hugo's `writeStats` does not record class attrs on `<th>`, so PurgeCSS strips the rule. The visual gate builds with `ENVIRONMENT=production` (`bin/build-if-stale`), which exports `HUGO_ENVIRONMENT` and therefore runs PurgeCSS - so the test build strips the rule exactly as production does, renders identically to the baseline, and passes green. `css_orphan_guard_test` is file-level, not rule-level. Caught by human review in #563. |
| H | Internal link pointed at a path that does not exist | `bin/rake test:links` (lychee, offline) | **CAUGHT** | The task globs every emitted `*.html` and passes them to `lychee --offline --root-dir`, so an unresolvable internal href fails the build. `lychee 0.24.2` is on PATH locally, so this runs without CI. |

## Results

**Score: 3 caught, 5 missed.** Two predictions were wrong (F and H) - both were
predicted CAUGHT and both were missed, and both for reasons that matter more than
the defects themselves.

A control run preceded the visual injections: `DesktopSiteTest#test_next_editorial`
on the unmodified tree reported `1 screenshot compared, no failures`. Without that,
a later red proves nothing about the gate.

| # | Defect | Gate run | Predicted | Actual | Evidence |
|---|---|---|---|---|---|
| A | CTA text -> `--ed-ink` on the accent fill | `bin/test test/system/desktop_site_test.rb -n test_next_editorial` | CAUGHT | **CAUGHT** | `Screenshot does not match for 'desktop/next/editorial': ({"area_size":2508.0,"region":[402.0,523.0,611.0,535.0],"difference_level":0.0004576581790123457...})` - 4.6x the 0.0001 floor, region well inside the fold |
| B | `.rr-td-muted` -> `--ed-ink`, below fold | same | MISSED | **MISSED** | `1 runs, 3 assertions, 0 failures` on BOTH desktop and mobile, while the built bundle the page loads contains `tbody .rr-td-muted{color:var(--ed-ink)}` - the defect shipped and the gate was green |
| C | Testimonial "helping us to find" -> "helping us find" | `bin/test test/unit/next_rail_test.rb -n test_testimonial_quote_is_canon_verbatim` | CAUGHT | **CAUGHT** | `the rendered quote is not a verbatim slice of the canon testimonial. Expected "...helping us to find problems..." to include "...helping us find problems..."` |
| D | `derived: tenure` -> `value: "18+"` | `bin/test test/unit/next_rail_test.rb` | MISSED | **MISSED** | `7 runs, 93 assertions, 0 failures`. Control: the same edit with `"19+"` DOES fail (`Expected ["19+", "5", "8+"] to include "18+"`), proving the frontmatter reaches the render and the gate is live - it simply cannot separate a frozen 18 from a derived 18 in 2026 |
| E | `noindex: false` in the `/next/` cascade | `bin/test test/unit/next_rail_test.rb` | CAUGHT | **CAUGHT** | Two failures. `Expected "index, follow, max-snippet:-1, max-video-preview:-1, max-image-preview:large" to include "noindex"` - the rendered robots meta flipped to indexable, which is the actual production consequence |
| F | "world-class" in the editorial pilot stub | `bin/test test/unit/marketing_copy_test.rb` | CAUGHT (rendered pass) | **MISSED** | `3 runs, 7 assertions, 0 failures` while `grep -c "world-class"` on the built page returns 1. Measured with the gate itself (baseline temporarily set to 0): clean tree = `baseline 0, now 11`, injected tree = `baseline 0, now 12`, against `RENDERED_BASELINE = 14` |
| G | `class` on a `<th>` + a matching CSS rule | visual gate + `bin/rake test:unit` | MISSED | **MISSED** | Visual: `1 screenshot compared, no failures`. Units: `287 runs, 6143 assertions, 0 failures`. Mechanism proven by four counts: rule in source CSS = 1, rule in BUILT CSS = 0, class in rendered HTML = 1, class in `hugo_stats.json` = 0 |
| H | Internal link -> `/clients/agent-inbox-typo/` | `bin/rake test:links` | CAUGHT | **MISSED** | `149516 Total, 31937 Unique, 15642 OK, 0 Errors, 133874 Excluded`. Per-link: `[EXCLUDED] https://jetthoughts.com/clients/agent-inbox-typo/ ... This is due to your 'exclude' values` |

## Where the predictions were wrong

**F - the ratchet has slack.** The prediction assumed a ratchet sits on the
current count. It does not: `RENDERED_BASELINE = 14` while the tree actually
renders 11 hits. Three new banned phrases can be added to rendered marketing
surfaces before the gate says a word, and the injected one landed in that gap
(11 -> 12). The file's own comment predicted this failure mode - "a ratchet left
slack lets the win regress silently" - and then the number was left at 14 when
the real count fell to 11. The source pass missed it for the predicted reason
(`SURFACES` does not glob `content/next/**`), so the rendered pass was the only
thing standing, and its slack let the defect through.

**H - `--offline` excludes every absolute URL, and the production build makes
every internal link absolute.** The prediction treated "Broken Internal Links" as
an internal-link gate. It is not one in practice. `bin/hugo-build` with
`ENVIRONMENT=production` renders internal links against the real baseURL, so they
emit as `https://jetthoughts.com/...`; `lychee --offline` excludes all http(s)
URIs by design. The homepage measures the effect exactly: `141 Total, 140
Excluded, 1 OK`, and the single checked link is the in-page `#main-content`
anchor from the skip link. Site-wide, 133,874 of 149,516 links are excluded and
`--verbose` prints zero `[OK]` lines. The job passes because it checks almost
nothing, not because the links are sound.

The fix was verified, not assumed. Re-running the same scan with a remap of the
public host onto the local build directory turns the same page from "all
excluded" into a real check that finds the injected defect:

```
lychee --offline --root-dir <dest> \
--remap "https://jetthoughts.com/(.*) file://<dest>/\$1" <dest>/**/*.html

[ERROR] file://<dest>/clients/agent-inbox-typo/ | File not found.
| Remaps: https://jetthoughts.com/clients/agent-inbox-typo/ --> file://<dest>/clients/agent-inbox-typo/
41 Total, 32 Unique, 35 OK, 1 Error, 5 Excluded
```

35 links checked instead of ~0, and the planted break is found.

## What nothing guards

1. **Everything below 1080px on a landing page.** The capture is viewport-sized,
not full-page. Measured on `/next/pilots/editorial/fractional-cto/` at
1920x1080: page height 3822px, captured 1080px - **28.3% of the page is
gated, 71.7% is not**. The first `.rr-td-muted` cell sits at y=1090, ten
pixels past the frame. This is the #564 defect class, and it quantifies task
#12. Consequence: any colour, spacing, or layout regression in the comparison
table, the timeline, the client band, the testimonial, or the closing CTA
ships green. Four of the page's five bands begin below the fold (y=1524,
2219, 2909).

2. **A frozen tenure number, until 2027-01-01.** `18+` frozen and `18+` derived
are the same characters this year. The gate is honest about this in its own
comment, so it is a known-and-accepted gap rather than a surprise - but it is
a real hole right now, and the pilot stubs are exactly where a hardcoded
number gets pasted.

3. **The next three banned marketing phrases.** `RENDERED_BASELINE = 14` against
an actual 11. Consequence: three regressions on the pages a prospect reads
are pre-authorised. Source-side, `content/next/**` is not globbed at all, so
the rendered pass is the only cover the `/next/` rail has.

4. **Any CSS rule keyed to a class that only ever appears on a `<th>`** (and, by
the same mechanism, any element whose class attrs Hugo's `writeStats` does not
record). The rule is stripped from the production bundle, the markup still
ships, and no gate compares intent to output. The visual gate cannot help
here even in principle: it builds with `ENVIRONMENT=production`, so it purges
exactly as production does and renders the same un-styled result as the
baseline. Caught by human review in #563; nothing automated stands behind that.

5. **Broken internal links, site-wide.** See H above. The gate name promises
coverage the configuration does not deliver - 89.5% of all links excluded, and
the ones actually checked are in-page anchors.

## Recommended (cheapest fix first)

1. **Add `--remap` to `test:links`** (Rakefile `task :links`), mapping the site's
baseURL host onto the build directory. One flag, and the existing job starts
doing the thing its name claims. Verified above to catch a planted break.
Expect it to surface a backlog of real breaks on first run - that is the point,
and it should be triaged before the task is made blocking again.
2. **Lower `RENDERED_BASELINE` from 14 to 11.** One number. Closes the pre-
authorised regressions immediately. Worth doing in the same pass that adds a
note to re-tighten it whenever a batch is cleared, since that is the step that
was skipped last time.
3. **Add `content/next/**/*.md` to `MarketingCopyTest::SURFACES`.** One glob line.
Gives the `/next/` rail source-side cover so it does not depend solely on a
ratchet with slack.
4. **Below-fold capture for the landing pilots** (task #12). The expensive one, so
scope it: these pages are four-to-five discrete bands, and a per-band
screenshot keyed to the existing `.rr-band` elements costs four more captures
per pilot and covers the 71.7% that is currently dark. A whole-page capture is
the tempting version and the wrong one - it re-introduces the font-swap and
lazy-image flake the current design deliberately avoids.
5. **The `<th>` purge trap has no cheap automated fix** and should stay a review-
checklist item rather than gain a bespoke gate. The honest cheap mitigation is
a note where it bites - `layouts/next/landing.html` already carries one - plus
preferring `nth-child` over classes on `<th>`. If it recurs a third time,
the proportionate gate is a build-time check that every class in a template's
markup survives into the built bundle, which is a bigger piece of work than
the defect has so far justified.

## Method notes

- Every injection was reverted and the tree checked `git status --porcelain`
empty before the next; the final tree is identical to `origin/master` apart
from this file. No baseline was re-recorded.
- Exit codes were captured directly (`cmd > log; echo $?`), never through a pipe -
`cmd | tail` reports tail's status, not the command's.
- Green runs were only trusted after proving the injection reached the artifact
under test: for B and G by grepping the BUILT CSS bundle the page actually
loads, for D by a `19+` sentinel that made the same gate fail, for F by grepping
the built HTML. A pass at a pinned tolerance proves "under that tolerance", not
"identical", and a pass on an unbuilt change proves nothing at all.