Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 24 additions & 4 deletions .okf/build/hugo-stats-th-classes.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,13 +33,33 @@ Development builds hide it completely: `postcss.config.js` sets
entirely, so `bin/hugo-build` renders the styling correctly while production
ships it unstyled.

# Why no gate catches it
# Why the VISUAL gate cannot catch it, and what does

The visual regression suite compares a baseline against a candidate that were
**both** built the same way, so a rule that is purged in both is invisible to
it. `css_orphan_guard_test.rb` does not help either - it asks whether a FILE
is reachable from a template, never whether a SELECTOR survived the purge.
This class of defect reaches production green.
it - not at a lower tolerance, not with a bigger capture. It is unguardable
there in principle. `css_orphan_guard_test.rb` does not help either: it asks
whether a FILE is reachable from a template, never whether a SELECTOR survived
the purge.

A unit test does catch it, because the defect is a set relation, not a picture.
`test/unit/next_purge_guard_test.rb` flags any class that is (a) styled in the
register's source CSS, (b) present on an element in the rendered production
HTML, and (c) missing from the fingerprinted bundle that page actually links.
All three at once is a silently-purged rule every time. It is keyed on the
MECHANISM, not on `th`, so the next element type Hugo stops recording is
covered without an edit.

Two things make it trustworthy rather than decorative:

- It reads the bundle from the page's own `<link href>`, never a glob of
`css/` - stale fingerprinted bundles from earlier builds sit in the same
directory, and reading one measures a build nobody ships.
- Broken before trusted (2026-08-22): injecting `class="rr-th-theirs"` on a
`<th>` plus a matching rule failed it with `Expected ["rr-th-theirs"] to be
empty`; moving the SAME class onto the `<td>` in the same row went green,
which is the control that proves it flags the purge and not merely a new
class.

# What to do instead

Expand Down
4 changes: 2 additions & 2 deletions .okf/build/index.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Build & Test

* [Hugo build pipeline](hugo-build.md) - bin/hugo-build with the 8 course validators; also the PurgeCSS cold-start race and the minified-unquoted-attribute audit-tool trap
* [Test gates](test-gates.md) - the local suites, when each is a commit blocker, the 0.0001 default tolerance, why the SECTION_CONFIGS shield was deleted and what measuring it cost, why a green run never refreshes a baseline, and why below-fold content is invisible at any tolerance, bin/record-baselines for accepting only the baselines you meant to move, and why a deleted source file still serves from every local _dest/ tree, plus the NULL CHANGE - a diff that passes every gate and alters nothing - and what `okf_validate` actually guards (shape, not truth; error-only conformance) with the two-spec trap
* [Test gates](test-gates.md) - the local suites, when each is a commit blocker, the 0.0001 default tolerance, why the SECTION_CONFIGS shield was deleted and what measuring it cost, why a green run never refreshes a baseline, and why below-fold content is invisible at any tolerance, bin/record-baselines for accepting only the baselines you meant to move, and why a deleted source file still serves from every local _dest/ tree, plus the NULL CHANGE - a diff that passes every gate and alters nothing - and what `okf_validate` actually guards (shape, not truth; error-only conformance) with the two-spec trap, and why computed-style assertions beat more screenshots for below-fold coverage and for contrast (a screenshot returns a pixel delta, never a ratio)
* [CI gates](ci-gates.md) - what GitHub Actions enforces: build, unit, path-scoped link check (visual regression is report-only), and what gates a PR never sees
* [Template PDFs](pdf-templates.md) - regenerating the downloadable course PDFs
* [Classes only on &lt;th&gt; get purged](hugo-stats-th-classes.md) - Hugo 0.165 writeStats records no class attrs on `<th>`, so PurgeCSS deletes those rules from the production bundle while dev builds look fine and the visual gate stays green; also two sibling false-greens on the /next/ rail (site-scripts in a landing baseof, computed `resources.Get` paths defeating the orphan guard)
* [Classes only on &lt;th&gt; get purged](hugo-stats-th-classes.md) - Hugo 0.165 writeStats records no class attrs on `<th>`, so PurgeCSS deletes those rules from the production bundle while dev builds look fine and the visual gate stays green - unguardable by a screenshot, guardable by a set-diff unit test (`next_purge_guard_test.rb`); also two sibling false-greens on the /next/ rail (site-scripts in a landing baseof, computed `resources.Get` paths defeating the orphan guard)
57 changes: 50 additions & 7 deletions .okf/build/test-gates.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,9 +196,45 @@ This is why the suite leans on per-section screenshots - each one scrolls its
section INTO the viewport first (`verify_section_for` →
`scroll_to find(css)`), which is the existing workaround, not an accident.
A page asserted only as one top-of-page shot is verified for its first fold and
nothing else. Closing the gap properly means either more section shots or
full-page capture; both are out of scope for the tolerance change and belong to
the follow-up that re-records baselines.
nothing else.

# Computed styles beat more screenshots for below-fold and for contrast

More captures is the tempting answer to the fold problem and the expensive one:
each new baseline inherits the tolerance problem above (a font/SVG-heavy band
needs its noise floor MEASURED by recording twice - the 0.0001 default is
unsafe there), and every one is a file two platforms have to agree on.

A computed-style assertion in a system test has neither cost. It does not care
about the viewport, and it returns a NUMBER rather than a pixel delta - which
matters because a screenshot cannot tell an intentional recolour from an
accessibility regression. Nothing in this suite measured contrast until
2026-08-22; a 3.33:1 black-on-ruby button shipped and was caught in human
review.

`test/system/next_pilot_contrast_test.rb` is the worked example, ~1.3s for
three pages, no baselines, wired into `test:critical`:

- The comparison table's "theirs" column must compute the register's own
`--*-muted` token. Asserted against the token, not a literal and not the
sibling cell: two registers set the "ours" column to the body colour and one
sets it to ink, so a sibling comparison passes on a real defect in the
registers where ink and body differ. That mistake was made and caught here by
injection, not by reading.
- A contrast walk over every visible text run: composite each translucent layer
down to an opaque background, then require 4.5:1 (3:1 for large text, SC
1.4.3) and 3:1 for a control's fill against its surround (SC 1.4.11). Text
over a background image or gradient is REPORTED as unresolvable rather than
measured against a guessed white.
- Guarded against the false green it would otherwise be: the walk asserts it
measured more than 50 text pairs and at least one fill, because a walk that
finds nothing is empty and empty passes.

Broken before trusted (2026-08-22), all three branches: an above-fold
`.rr-btn-primary { color: var(--ed-ink) }` failed at `2.31:1 (needs 4.5:1)`; a
below-fold `.rr-td-muted { color: #b5b0ab }` - the class the screenshot gate
cannot see at all - failed at `2.15:1` on five cells; a low-contrast fill with
a readable label failed only the fill branch at `1.12:1 (needs 3.0:1)`.

# Rake tasks and suite layout

Expand Down Expand Up @@ -815,10 +851,17 @@ Skipping step 1 has cost this repo repeatedly:
on a site with five real broken links, one of them a conversion path and one
a post's own canonical pointing at a 404.

**When a gate cannot discriminate yet, write that in the test.** The derived
tenure assertion cannot tell `derived` from `frozen` while both read "18+" in
2026 - it starts biting on 2027-01-01. That is stated in the test body, so the
next reader does not mistake a passing run for proof.
**When a gate cannot discriminate yet, write that in the test - then ask what
CAN see the defect today.** The rendered tenure assertion cannot tell `derived`
from `frozen` while both read "18+" in 2026; it starts biting on 2027-01-01,
and that limit is stated in the test body so a passing run is not mistaken for
proof. But the limit is a property of the RENDERED surface, not of the defect:
the freeze is plainly visible in the stub's frontmatter. `NextRailTest`
therefore carries both halves, and they are one gate - the rendered assertion
for the value, a source assertion that the tenure stat is `derived: tenure` and
that no stat is frozen at today's derived string. This is not a config test:
the hardcoded literal IS the defect class, the way the 2026-08-14 canon audit
found the founding year wrong in eight places because each kept its own copy.

Full fault-injection matrix, including what nothing guards:
`docs/20-29-testing-qa/20.11-gate-fault-injection-2026-08-22-reference.md`.
47 changes: 47 additions & 0 deletions .okf/log.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,53 @@ make it green: restructure same-day entries under one heading, and add
`timestamp` to the 23 concepts missing it (anchored to each file's last commit
time, which is verifiable - never invented).

## 2026-08-22 - closing three of the fault-injection gaps, each broken before it was trusted

Three of the five misses above now have gates. Every one of them was injected,
watched fail, reverted and watched pass - the procedure this bundle added the
same day, applied to its own output.

**A gate blind until 2027 needed a second half, not a note.** The rendered
tenure assertion cannot separate a frozen `18+` from a derived one in 2026, and
saying so in the test body is honest but does not guard anything. The freeze is
plainly visible one layer down, in the stub's frontmatter, so `NextRailTest`
now asserts both: the rendered value AND that every pilot's tenure stat is
`derived: tenure` with no stat frozen at today's derived string. General
lesson: when a gate cannot discriminate on one surface, ask which surface it
CAN discriminate on before accepting the hole.

**"Unguardable" was true only of the gate that was reached for first.** The
`<th>` PurgeCSS trap really is invisible to the visual suite in principle -
baseline and candidate are both built purged. It is trivially visible to a set
diff: styled in source CSS ∩ present in rendered HTML − present in the built
bundle. `test/unit/next_purge_guard_test.rb` is 60 lines, adds ~0.1s, and is
keyed on the mechanism rather than on `th`. The control that makes it
believable is the second injection, not the first: the same class moved onto a
`<td>` goes green, so it flags the purge and not merely a new class.

**The audit's own recommendation was the expensive answer.** It proposed
per-band screenshots for the 71.7% of a pilot page below the fold. That buys
four more baselines per pilot, each needing its noise floor measured on the
platform that judges it, and still returns a pixel delta rather than a ratio -
so an intentional recolour and an AA regression stay indistinguishable.
`test/system/next_pilot_contrast_test.rb` does both jobs in ~1.3s with no
baselines: computed styles ignore the viewport, and contrast comes out as a
number. It closes the separately-flagged hole that nothing in this suite
measured contrast at all - a 3.33:1 black-on-ruby button shipped this rail and
only human review caught it.

One mistake worth keeping. The muted-column assertion was first written as
"differs from the sibling cell and from body colour". It PASSED the exact #564
injection, because in that register the "ours" column is body colour and the
injected ink is a third value. Rewritten to assert the register's own
`--*-muted` token, it fails with the token it actually landed on. The test was
wrong for two runs and green both times; only the injection said so.

Gates: `test:unit` 289 runs / 6187 assertions / 0 failures in 1.42s (287 /
6143 / 1.33s before - the purge guard reuses the build the suite already
makes); `test:critical` 38 runs / 0 failures, 55 screenshots compared, no
failures, no baseline rewritten; `bin/hugo-build` clean.

## 2026-08-22 - fault injection: the suite caught 3 of 8, and the misses were the point

Eight realistic defects planted one at a time, predictions written down BEFORE
Expand Down
7 changes: 6 additions & 1 deletion Rakefile
Original file line number Diff line number Diff line change
Expand Up @@ -25,10 +25,15 @@ namespace :test do
# (2026-07-19 analysis: blog + homepage + course + site-wide dominate
# 6-month history) + the lead-gen funnel (services, contact,
# free-consultation). Full system suite = test:system; everything = test:all.
# NextPilotContrastTest rides along (~1.3s, no baselines): it is the only
# thing in this suite that measures contrast as a NUMBER rather than as a
# pixel delta, and the only thing that sees the 71.7% of a register pilot
# that sits below the 1080px capture (20.11 §"What nothing guards").
CRITICAL_TESTS =
"/homepage|blog_index|blog_post$|visit_blog_post|course_|old_blog_url|" \
"test_services($|_menu)|contact_us|free_consultation|not_found|" \
"hamburger|color_system|mermaid|codeblock_language|friday_report/"
"hamburger|color_system|mermaid|codeblock_language|friday_report|" \
"NextPilotContrastTest/"

Rake::TestTask.new(:critical) do |t|
t.libs << "test"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,30 @@ lychee --offline --root-dir <dest> \
markup survives into the built bundle, which is a bigger piece of work than
the defect has so far justified.

## What was closed afterwards (same day)

The results above are the measurement and stay as recorded. Three of the five
misses have gates now, each injected, failed, reverted and passed before being
believed:

| # | Gate added | Where |
|---|---|---|
| D | Source half of the pair - every pilot's tenure stat must be `derived: tenure`, and no stat may be frozen at today's derived string. Bites now, not on 2027-01-01. | `test/unit/next_rail_test.rb` |
| G | Set diff: styled in the register's source CSS ∩ present in the rendered HTML − present in the bundle the page links. Keyed on the mechanism, so the next blind element type is covered. | `test/unit/next_purge_guard_test.rb` |
| B + the contrast hole | Computed styles instead of per-band screenshots: the "theirs" column must compute the register's `--*-muted` token, and every visible text pair must clear 4.5:1 (3:1 large, 3:1 for control fills). Viewport-independent, no baselines, in `test:critical`. | `test/system/next_pilot_contrast_test.rb` |

Two of this document's own conclusions were wrong and are worth naming:

- **Recommendation 5 - "the `<th>` purge trap has no cheap automated fix" -
was false.** It is unguardable by the VISUAL gate, which is what the analysis
actually established; a unit test finds it in a set operation for ~0.1s. The
reasoning generalised "this gate cannot" into "no gate can".
- **Recommendation 4 - per-band screenshots for the below-fold gap - was the
expensive answer**, and it inherits the tolerance problem: four more
baselines per pilot, each needing its noise floor measured on the platform
that judges it, and each still reporting a pixel delta rather than a ratio.
Computed styles cover the whole page and return the number.

## Method notes

- Every injection was reverted and the tree checked `git status --porcelain`
Expand Down
Loading
Loading