From 35e8f4e5d6186e6db6ac93d56f32b50d59cefc5a Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Thu, 20 Aug 2026 23:43:05 +0200 Subject: [PATCH] OKF: the stall can mask a real failure; a local visual red needs a master run Two gate rules sharpened after a six-PR merge pass. Both are increments on rules the bundle already had, added to the existing concepts rather than as new ones. ci-gates already said "re-run, don't raise the cap" and "if the job got past checkout it is a real failure". Necessary, not sufficient. On #511 `Unit Tests` read `fail 10m2s` - the stall signature exactly - and "known flake, merge it" was the tempting call. The fresh run got past checkout and failed again in 1m48s on a genuine defect: PavedPathGuardTest caught friday-report.css wired into bin/qtest but missing from css-bundle-ownership-map.md. Same check, same PR, two unrelated causes. Rule: a slow failure and a fast failure are different failures, and the first red's explanation must never carry over to the second. test-gates gains the local-vs-master rule. Verifying #511, `bin/qtest --changed` went red pointing at services/fractional-cto - a page it never touched - and the available story was that its postcss.config.js edit had shifted CSS site-wide. It had not: that edit only ADDS purgecss safelist entries, and safelisting more can only preserve more CSS, never remove any. The check that settled it was the same system test on clean origin/master on this macOS host: 34 runs, 6 failures, 8 of 77 screenshots mismatched. The suite is red on master here, so the local leg could not answer the question; CI's native-Linux Screenshot Tests passed #511 in 16m15s and was right. Validated: okf_validate --strict conformant, 0 errors (the 70 warnings are the pre-existing house log-heading convention). bin/hugo-build green. Co-Authored-By: Claude Opus 5 (1M context) --- .okf/build/ci-gates.md | 12 ++++++++++++ .okf/build/test-gates.md | 17 ++++++++++++++++- .okf/log.md | 40 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 68 insertions(+), 1 deletion(-) diff --git a/.okf/build/ci-gates.md b/.okf/build/ci-gates.md index d7cdd3585..846d40f73 100644 --- a/.okf/build/ci-gates.md +++ b/.okf/build/ci-gates.md @@ -12,6 +12,8 @@ verified: at: 2026-08-07T00:00:00Z - by: claude/sonnet-5 at: 2026-08-20T00:00:00Z + - by: claude/opus-5 + at: 2026-08-21T00:00:00Z --- # What CI enforces on a PR @@ -70,6 +72,16 @@ Consequences for how to react: (`gh api repos///actions/jobs//logs`) and looking for that group/error pair. If the job got PAST checkout, it is a real failure and a re-run is the wrong move. +* **Re-read the re-run — the stall can be hiding a real failure** (2026-08-21). + On PR #511 `Unit Tests` read `fail 10m2s`, the stall signature exactly, and + "known flake, merge it" was the tempting call. The fresh run got past + checkout and failed again in **1m48s** — a genuine defect (`PavedPathGuardTest` + caught a page CSS bundle wired into `bin/qtest` but absent from + `css-bundle-ownership-map.md`). Same check, same PR, two different causes. So + the rule above is necessary but not sufficient: **a slow failure and a fast + failure are different failures.** Multi-minute with no assertion output is + infrastructure; fast with an assertion is the code. Never let the first red's + explanation carry over to the second. * **Shrink what we ask for.** We cannot fix upstream, only reduce exposure. This repo is a heavy ask: **1.70 GiB pack**, and `content/` alone is **625 MB across 1,576 images**, downloaded by every job regardless of depth. diff --git a/.okf/build/test-gates.md b/.okf/build/test-gates.md index f753e2998..6629281d7 100644 --- a/.okf/build/test-gates.md +++ b/.okf/build/test-gates.md @@ -8,7 +8,8 @@ generated: { by: claude/opus-4-8, at: 2026-08-12T20:20:00Z } verified: - { by: claude/fable-5, at: 2026-08-01T11:30:00Z } - { by: claude/sonnet-5, at: 2026-08-20T00:00:00Z } -timestamp: 2026-08-20T00:00:00Z + - { by: claude/opus-5, at: 2026-08-21T00:00:00Z } +timestamp: 2026-08-21T00:00:00Z --- # The suites @@ -149,6 +150,20 @@ Minitest under `test/`, driven by `Rakefile` (`Rake::TestTask`). baselines - cost a full false "bistable render" investigation). Re-record = run the suite, keep the rewritten PNG, COMMIT it; only then can a rerun go green. +- **A local visual red cannot condemn a branch until you have run the same + thing on master** (2026-08-21). Verifying PR #511 (template/CSS class), + `bin/qtest --changed` went red pointing at `services/fractional-cto` + baselines - a page the PR never touched. The plausible story was that its + `postcss.config.js` edit had shifted CSS site-wide; it had not, because that + edit only ADDS purgecss safelist entries and safelisting more can only + preserve more CSS, never remove any. The decisive check was the same system + test on clean `origin/master` on the macOS host: **34 runs, 6 failures, 8 of + 77 screenshots mismatched** - the suite is red on master here, so a local red + said nothing about the branch. The run also rewrites two of those baselines + mid-run and then fails its own dirty-check, so repeated + `git checkout -- test/fixtures/screenshots/` never converges. Reproduce on + master first; if master is red the same way, fall back to CI's native-Linux + `Screenshot Tests` (which passed #511 in 16m15s while local was red). - `bin/dtest` from a git WORKTREE is VACUOUS-GREEN: the worktree's `.git` is a pointer file to a directory outside the container mount, so git fails inside the container, baselines resolve to nothing, and every screenshot diff --git a/.okf/log.md b/.okf/log.md index f24e7e1eb..aef816873 100644 --- a/.okf/log.md +++ b/.okf/log.md @@ -2,6 +2,46 @@ Newest first. Entries before 2026-08-19 are squashed to one line each (compacted 2026-08-20); their full text is in this file's git history. +## 2026-08-21 - Two gate lessons from merging six PRs: the stall masks, the local red lies + +Both are sharpenings of rules the bundle already had, and both came from the +same PR (#511) during a queue-clearing pass. + +**The checkout stall can hide a real failure behind it.** ci-gates already said +"re-run, don't raise the cap" and "if the job got past checkout it is a real +failure". Necessary, not sufficient. #511's `Unit Tests` read `fail 10m2s` - the +stall signature exactly - and the tempting call was "known flake, merge". The +fresh run got past checkout and failed again in **1m48s** on a genuine defect: +`PavedPathGuardTest` caught `friday-report.css` wired into `bin/qtest` but +missing from `css-bundle-ownership-map.md`, which is the other half of +new-page.md step 6. Same check, same PR, two unrelated causes. The rule added: +**a slow failure and a fast failure are different failures** - multi-minute with +no assertion output is infrastructure, fast with an assertion is the code, and +the first red's explanation must never carry over to the second. + +**A local visual red cannot condemn a branch until master has been run the same +way.** `bin/qtest --changed` on #511 went red pointing at +`services/fractional-cto` - a page the PR never touched - and the available +story was that its `postcss.config.js` edit had shifted CSS site-wide. It had +not: the edit only ADDS purgecss safelist entries, and safelisting more can only +preserve more CSS, never remove any. The check that settled it was the same +system test on clean `origin/master` on this macOS host: **34 runs, 6 failures, +8 of 77 screenshots mismatched.** The suite is red on master here, so the local +leg was unusable for the question. CI's native-Linux Screenshot Tests passed +#511 in 16m15s; CI was right and local was noise. Also worth knowing: the run +rewrites two of those baselines mid-run and then fails its own dirty-check, so +`git checkout -- test/fixtures/screenshots/` never converges. + +**Half-wired page bundles are a real recurring class.** The guard exists because +`docs/workflows/new-page.md` step 6 has two halves - `bin/qtest` PAGE_TESTS and +the ownership map - and #511 did one. Anyone adding `themes/beaver/assets/css/pages/*.css` +does both or the guard reds. + +**Repo hygiene:** #505 landed two Phase-1a verification PNGs (392KB) at the repo +root. Every other screenshot in this repo lives under `docs/projects//`. +Removed in #514. Binary at the repo root is the kind of thing the next session +copies because it looks sanctioned. + ## 2026-08-21 - The course's "good positions" were an artifact; GA4 UI setup closed out **Course discovery diagnosed, and the inherited premise is retracted.** Two