diff --git a/.okf/build/rendering-stack.md b/.okf/build/rendering-stack.md index 198615b5a..5f68be817 100644 --- a/.okf/build/rendering-stack.md +++ b/.okf/build/rendering-stack.md @@ -1,7 +1,7 @@ --- type: build-concept title: A screenshot baseline is a recording of a rendering stack -description: Which environment records the linux/ baselines, why local dtest can differ from CI, the all-arm64 migration, and the five wrong explanations for one drift - two of them formally withdrawn +description: Which environment records the linux/ baselines, why bin/dtest is now authoritative for detecting visual defects, and the five wrong explanations for a drift whose real cause was stale baselines tags: [testing, screenshots, docker, ci, rendering] timestamp: 2026-08-22T00:00:00Z --- @@ -22,19 +22,31 @@ produced the baseline and which produced the candidate**. | **base OS** | **Debian 13 trixie** (freetype 2.13.3) | **Ubuntu 24.04** | Architecture, browser and fonts are deliberately pinned to match. The base OS -is not - so freetype/harfbuzz may differ, which would be enough to move dense -monospace text. 8 `mobile/blog/special/codeblocks/*` screenshots differ -~0.055-0.063 with no defect present, and **the base OS is the last unpinned -variable, not a demonstrated cause** - see the open question below. +is not - and **measurement on 2026-08-22 says that does not matter today.** +Against baselines recorded by CI on Ubuntu ARM, local `bin/dtest` on Debian ARM +reports `55 screenshots compared, no failures`. Pixel-identical across the +distro boundary. + +**Measured, not structural.** Debian and Ubuntu upgrade freetype/harfbuzz +independently, so this agreement can end without anyone changing this repo. +It is a fact about today's two images, not a guarantee - which is why +"re-record where the tester runs" stays a rule below. **macOS is a third stack.** `bin/test` records `macos/`; `bin/dtest` and CI record `linux/`. Neither OS's set is a subset of the other, and a candidate rendered on one must never be committed as a baseline for another. -## Five wrong explanations, in the order they were believed +## Five wrong explanations, and the boring answer (RESOLVED 2026-08-22) + +**The cause was stale baselines.** Nothing else. The `linux/` set had been +recorded on amd64/Chrome-152 against older content; every "drift" was that old +recording compared against a current render. Re-recording on master made all 55 +match everywhere, local and CI. -Each sounded mechanical and each was asserted without measuring the thing it -named. Listed because the *shape* recurs, not the specific causes: +Five explanations were believed before that. Each sounded mechanical, each was +asserted without measuring the thing it named, and the true cause was never +proposed as a hypothesis because it was not interesting enough to be suspicious +of. Listed because the *shape* recurs, not the specific causes: 1. **"arm64 vs amd64 drift."** The container was x86_64 at the time. `.dev/compose.yml` pins `platform: linux/amd64` on the `t` service and that DOES override @@ -49,19 +61,33 @@ named. Listed because the *shape* recurs, not the specific causes: both see the same tree. 4. **"A date-gated post appeared."** Both candidate posts were dated before the baseline was recorded. -5. **"Debian container vs Ubuntu runner."** Believed 2026-08-22 and asserted in - #589's commit message. The base OS is the last unpinned variable, which makes - it the standing candidate - but it has never been measured either. Withdrawn; - see the codeblocks section below for the observation that fooled it. - -The measurement that resolved #1-#4 took one command and should have been -first: read the container's OS and library versions. #5 has no such measurement -yet, which is exactly why it is listed here rather than in Rules. +5. **"Debian container vs Ubuntu runner."** Believed 2026-08-22, asserted in + #589's commit message, and **DISPROVEN the same day**: after the re-record, + Debian ARM and Ubuntu ARM agree on all 55. What made it persuasive was that + local and CI failed the *same 8 keys* - which reads as corroboration and is + not, because both were comparing against the same stale baselines, so every + hypothesis predicted red on both. **Two observers agreeing tells you nothing + when they share the input you are trying to test.** + +Why this family and not others: dense monospace text is where any rendering +difference lands first, so the codeblocks screenshots were the most sensitive +surface in the suite. They were not arbitrary - they were the canary, which is +also why they looked like evidence for whichever theory was current. + +The lesson is the ranking. Every one of these five was a story about a +*mechanism*; none was a check on the *instrument*. "Are the baselines current?" +is cheap, boring, and would have ended it in a day. ## Rules +- **Suspect the baselines BEFORE the mechanism.** "Are these current?" is the + cheapest question available and it is the one that resolved a drift five + mechanism-theories failed on. Ask it first, every time. - **Identical `difference_level` across two runs = stale baseline, not flake.** Flake varies; a stack mismatch does not. +- **Agreement between two observers proves nothing if they share the input + under test.** Local and CI failing the same keys looked like corroboration + and was an artifact: both were reading the same stale baselines. - **Re-record where the tester runs.** An `update-baselines` dispatch on a feature branch records the branch tree while PR runs test the merge commit; dispatch on **master** so recorder and tester agree. That is what fixed @@ -86,9 +112,9 @@ on some hypothetical x86 box does not qualify; neither does an unexplained pixel diff, which is a measurement job, not grounds to go back. Treat "should we return to amd64?" as answered unless you can name the thing ARM cannot do. -This matters because the 8 unexplained codeblocks keys below are exactly the -kind of loose end that invites a retreat to the old stack. They predate the -migration and survived it unchanged - so they are not evidence against ARM. +The loose end that would have invited a retreat - the 8 codeblocks keys - is +closed: they were stale baselines, not an ARM problem, and the re-record made +them green. ARM cost nothing in correctness and removed the emulation tax. ### How it became available (was "waiting for Stable") @@ -130,47 +156,40 @@ Two things the plan did not anticipate, both of which would have broken CI too: kept-and-now-correct: it is precisely what made wrong explanation #1 above look plausible. -### The 8 codeblocks keys: still UNRESOLVED, and beware the false confirmation - -After changing both architecture and Chrome major version, 47 of 55 screenshots -still matched baselines recorded on amd64/Chrome-152. The 8 that did not are the -same `mobile/blog/special/codeblocks/*` family, at the same magnitudes - so the -divergence is neither arch nor Chrome version. - -**It does NOT follow that the base OS is the cause.** Local ARM (Debian) and CI -ARM (Ubuntu) fail the *same 8 keys*, which reads like confirmation and is not: -both are being compared against baselines recorded on the OLD stack, so every -hypothesis predicts red on both. The observation cannot discriminate. An earlier -commit message in this very migration asserted the distro cause anyway - the -fifth instance of the exact shape catalogued above. - -**The discriminating experiment, still to run:** re-record `linux/` via an -`update-baselines` dispatch on master, then run local `bin/dtest`. Green means -the distro never mattered; the same 8 red means it does. Until that runs, the -cause is unknown, not "Debian vs Ubuntu". - -## The open decision: one rendering stack, or two? - -Running CI inside this same container makes local and CI identical by -construction and lets `bin/dtest` be authoritative for visuals; both sides are -now arm64 native so there is no emulation either way, and the cost is image -build/pull per job - -**measure it before committing.** Feasibility checked 2026-08-22: the repo is -PUBLIC so GHCR is free, the image is 2.54 GB uncompressed, and GitHub Actions -can run a whole job inside it via the job-level `container:` key. Publish on -changes to `.dev/Dockerfile` or `.dev/cft-version` only - the same hash that -already keys the Chrome cache. Note the honest comparison is not "pull vs -nothing": CI already spends the same class of time on -`setup-ruby-and-dependencies`, the Chrome-for-Testing download and the font -packages, all of which the container replaces. - -The tempting cheaper alternative - move the CONTAINER to Ubuntu so it matches -the runner - is blocked: the image builds on `ruby:$RUBY_VERSION-slim`, and the -official Ruby images are Debian-only. Matching that way means hand-rolling Ruby -on an Ubuntu base, which is more moving parts than publishing one image. - -The alternative is to accept the split: CI owns pixel truth, `bin/dtest` is a -behavioural gate, and the divergent keys are screened. Free today, but an -expected-red list is exactly what rotted into "everything is expected red" -before #566. Pinning font and library versions across two distros is a third -option and not a serious one - they drift independently, forever. +### The 8 codeblocks keys: CLOSED 2026-08-22 + +The experiment: re-record `linux/` via an `update-baselines` dispatch on master +(`dc11791ab`, all 55 rewritten), then run local `bin/dtest` against it. + +**Result: `55 screenshots compared, no failures`.** The distro never mattered. + +Then, because a green gate and a blind gate look identical, the instrument was +checked: forcing `background-color: #ff0000` on `.post-article .highlight pre` +produced `16 failures` - all 8 desktop plus all 8 mobile codeblocks keys, and +nothing else. Reverting returned it to 0. Clean 0 -> injected 16 -> reverted 0 +is the evidence that the green means something. + +## One stack or two: DECIDED 2026-08-22 - two, and do nothing + +The plan was to publish this image to GHCR and run CI inside it, so local and +CI would be identical *by construction*. **Do not build that.** Its entire +justification was closing a Debian-vs-Ubuntu gap that measurement says is not +open: the two images already agree on all 55 screenshots. A 2.54 GB build/pull +per job, retired before it was built. + +The other rejected option was accepting a split where CI owned pixel truth and +`bin/dtest` was merely behavioural, with divergent keys screened. Also dead, and +good riddance: an expected-red list is exactly what rotted into "everything is +expected red" before #566. + +What holds instead: + +- **`bin/dtest` is authoritative for DETECTING visual defects.** It agrees with + CI, so a local red is a real defect. There is no expected-red list. +- **CI remains authoritative for RECORDING.** Agreement is measured, not + structural - Debian and Ubuntu move independently - so re-recording stays a + master dispatch. Detection and recording are different powers; this split is + deliberate, not leftover caution. +- **Revisit only on evidence**, i.e. a `bin/dtest` red that a master re-record + then makes green. That is the signature of the distros having drifted apart, + and it is the only thing that should reopen the GHCR question. diff --git a/.okf/log.md b/.okf/log.md index 118fac248..f15e3d745 100644 --- a/.okf/log.md +++ b/.okf/log.md @@ -51,6 +51,29 @@ make it green: restructure same-day entries under one heading, and add `timestamp` to the 23 concepts missing it (anchored to each file's last commit time, which is verifiable - never invented). +## 2026-08-22 - the codeblocks drift was STALE BASELINES; five theories closed at once + +**`rendering-stack`**: re-recorded `linux/` on master (`dc11791ab`, all 55) and +ran local `bin/dtest` against it: **`55 screenshots compared, no failures`**. +Debian ARM and Ubuntu ARM are pixel-identical. The drift that produced five +successive explanations over three weeks was old baselines compared against +current renders - nothing more. + +Verified the gate can still fail before believing the green: forcing +`background-color: #ff0000` on `.post-article .highlight pre` produced 16 +failures (all 8 desktop + all 8 mobile codeblocks, nothing else), and reverting +returned it to 0. + +Consequences recorded in the concept: +- **The GHCR container-publishing plan is CANCELLED.** It existed solely to + close the Debian-vs-Ubuntu gap by construction. There is no gap; a 2.54 GB + build/pull per CI job is retired before being built. +- **`bin/dtest` is authoritative for DETECTING visual defects** - local red now + means a real defect, and there is no expected-red list. CI keeps RECORDING, + because the agreement is measured, not structural. +- Two new rules: suspect the baselines before the mechanism, and two observers + agreeing proves nothing when they share the input under test. + ## 2026-08-22 - ARM migration shipped, and the "Debian vs Ubuntu" answer is withdrawn **`rendering-stack`**: the whole stack moved to arm64 - CfT pinned to diff --git a/STATUS.md b/STATUS.md index e66bf279f..7fd5e89dc 100644 --- a/STATUS.md +++ b/STATUS.md @@ -24,7 +24,7 @@ | 2605 course | v2 live; measuring. Diagnosis: **arrival, not content**; course SEO/AEO **closed** (Paul 2026-08-21) | LinkedIn arrival-test cards **LI-0…LI-D** in [`content-plan`](linkedin-posts/content-plan.md); no new funnel posts on the unproven bridge | [`TASK-TRACKER`](docs/projects/2605-tech-for-non-technical-founders/TASK-TRACKER.md) | | 2607 campaign tasks | Cold-public-sourcing premise tested and **failed** (3 sweeps, 4 venues, 0 verified-fresh rows; Reddit still un-openable) | Sept-restart Paul decision: retire the cold lane or buy Reddit API access (backlog §c) | [`2607 backlog`](docs/projects/2607-vibe-code-rescue/backlog.md) | | Test/CI hygiene | **Gates rebuilt 2026-08-22.** Fault-injecting 8 realistic defects caught **3**; now catches **8** ([#576](https://github.com/jetthoughts/jetthoughts.github.io/pull/576), audit in [`20.11`](docs/20-29-testing-qa/20.11-gate-fault-injection-2026-08-22-reference.md)). Link job was excluding 90% of links — 114,050 checked now vs 15,642, and it found 5 real site defects ([#574](https://github.com/jetthoughts/jetthoughts.github.io/pull/574)). `bin/dtest` was comparing **nothing** from a worktree ([#578](https://github.com/jetthoughts/jetthoughts.github.io/pull/578)). CI Linux screenshot job green (run 32565008850) | **Paul decides the dtest arch policy** (below); otherwise nothing queued | [`test-gates`](.okf/build/test-gates.md) | -| Rendering stack → **arm64 is THE stack** | **Settled (Paul 2026-08-22): arm64 everywhere, reversed only if something is found that genuinely cannot support it.** Shipping in [#589](https://github.com/jetthoughts/jetthoughts.github.io/pull/589): CfT pins `153.0.8010.5` (first with `linux-arm64`), container + CI both `arm64` (`ubuntu-24.04-arm`), compose PINS arm64 rather than following the host. Non-visual CI green; **Screenshot Tests red is expected** — `linux/` baselines are amd64/Chrome-152 recordings of pre-[#583](https://github.com/jetthoughts/jetthoughts.github.io/pull/583) content. `mem_limit: 4g` stays (Paul) — settled, not pending | Merge #589 → dispatch `update-baselines` **on master** (recorder and tester must see one tree) → local `bin/dtest`. The 8 codeblocks keys are an open *measurement*, NOT a reason to revisit arm64 — they predate the migration and survived it unchanged | [`rendering-stack`](.okf/build/rendering-stack.md) | +| Rendering stack → **arm64, and DONE** | **Closed 2026-08-22.** arm64 everywhere ([#589](https://github.com/jetthoughts/jetthoughts.github.io/pull/589) merged `efe85c228`, CfT `153.0.8010.5`, CI `ubuntu-24.04-arm`, compose PINS arm64). Baselines re-recorded on master (`dc11791ab`, all 55). **Local `bin/dtest` is now GREEN against CI's baselines — `55 compared, no failures`** — so the year-old codeblocks drift was stale baselines, not Debian-vs-Ubuntu, and five explanations were wrong. Gate verified discriminating by fault injection (0 → 16 → 0). `mem_limit: 4g` stays (Paul) | **Nothing queued.** `bin/dtest` is now authoritative for DETECTING visual defects (no expected-red list); CI still owns RECORDING. The GHCR container-publishing plan is CANCELLED — it existed only to close a gap that measurement says isn't open | [`rendering-stack`](.okf/build/rendering-stack.md) | **Not in flight**: 2604 typography — closed 2026-08-08 (P2 leftovers re-homed). 2509 CSS migration — **project complete 2026-07-19** (tracker in maintenance mode; the dormant remainder is the Phase-E groomed backlog + trigger-conditioned items); the 2608 clean-slate rail is the live CSS strategy ([ADR-0006](docs/adr/0006-clean-slate-dual-run.md)). diff --git a/bin/dtest b/bin/dtest index 7fd47e9f8..8107e0dbd 100755 --- a/bin/dtest +++ b/bin/dtest @@ -25,12 +25,16 @@ bin/build-if-stale _dest/public-dtest # Discarding those candidates costs nothing, because a candidate produced here # is never committable - and it is one `git add -A` from corrupting the set. # -# Since 2026-08-22 the ARCHITECTURE argument for that no longer applies: both -# this container and CI run arm64 on the same pinned Chrome. Do not read that -# as permission to commit local candidates. The base OS still differs (Debian -# trixie here, Ubuntu 24.04 on the runner) and that is the last unpinned -# rendering variable - see .okf/build/rendering-stack.md, which also records -# why "same failures on both sides" is NOT evidence the two agree. +# Measured 2026-08-22: this container and CI now agree on all 55 screenshots +# (same arch, same pinned Chrome, and the Debian-vs-Ubuntu difference turned +# out not to matter). So a red here is a REAL defect - there is no +# expected-red list any more. +# +# Detecting and RECORDING are still different powers, which is why the restore +# below stays. The agreement is measured, not structural: Debian and Ubuntu +# upgrade freetype/harfbuzz independently, so it can end without anything in +# this repo changing. Re-recording remains a master `update-baselines` +# dispatch. See .okf/build/rendering-stack.md. # The only legitimate linux re-record is the CI `update-baselines` dispatch # (.okf/build/test-gates.md). Diff artifacts (*.diff.png, *.heatmap.diff.png) # are untracked and survive the restore, so a red run stays inspectable. diff --git a/docs/20-29-testing-qa/screenshot-testing/20.10-visual-suite-speed-research-reference.md b/docs/20-29-testing-qa/screenshot-testing/20.10-visual-suite-speed-research-reference.md index bfd2cef9c..bef1c7ca6 100644 --- a/docs/20-29-testing-qa/screenshot-testing/20.10-visual-suite-speed-research-reference.md +++ b/docs/20-29-testing-qa/screenshot-testing/20.10-visual-suite-speed-research-reference.md @@ -68,12 +68,14 @@ minutes. Puma per process; `JOBS=1` in the dtest container — port pin + ~2 emulated CPUs.) - **O3 — Docker vs host: CONFIRMED** (~18% faster on smoke, 41.6s vs 50.5s). - The 10 red Linux baselines are NOT stale. **The "emulation drift" diagnosis - recorded here (green on CI-native amd64, red only under local Apple-Silicon - emulation) is WITHDRAWN as of 2026-08-22** — emulation ended when the stack - went all-arm64, and the reds did not, so emulation was never the cause. The - operational rule is unchanged and still correct: do NOT re-record locally; - trust CI. The cause is open — see [`rendering-stack`](../../../.okf/build/rendering-stack.md). + **RESOLVED 2026-08-22, and this entry had it exactly backwards.** The red + Linux baselines WERE stale — that was the whole cause. The "emulation drift" + diagnosis (green on CI-native amd64, red only under local Apple-Silicon + emulation) is withdrawn: emulation ended when the stack went all-arm64 and + the reds did not. After a master re-record, local `bin/dtest` is green + (`55 compared, no failures`). Note the shape — "NOT stale" was asserted here + and repeated for three weeks, and it was the one thing worth checking. See + [`rendering-stack`](../../../.okf/build/rendering-stack.md). - **O4 — direct-visit instead of menu-walk.** Several desktop tests reach their subject page via `visit "/"` + hover/click when a direct URL exists (`test_about_us`, `test_contact_us`, `test_free_consultation`, ...). Each pays