diff --git a/.dev/compose.yml b/.dev/compose.yml index 07f4a654b..0659a9ac5 100644 --- a/.dev/compose.yml +++ b/.dev/compose.yml @@ -12,7 +12,8 @@ services: - ..:/app:delegated - hugo_cache:/tmp/hugo_cache ports: - - '1313:1313' + # Fixed 1313 let only one worktree run this service at a time. + - '${HUGO_PORT:-1313}:1313' command: server --bind 0.0.0.0 --poll 500ms --disableFastRender=false environment: HUGO_CACHEDIR: /tmp/hugo_cache diff --git a/.okf/build/test-gates.md b/.okf/build/test-gates.md index cd2d7659f..78edfdb95 100644 --- a/.okf/build/test-gates.md +++ b/.okf/build/test-gates.md @@ -865,3 +865,81 @@ found the founding year wrong in eight places because each kept its own copy. Full fault-injection matrix, including what nothing guards: `docs/20-29-testing-qa/20.11-gate-fault-injection-2026-08-22-reference.md`. + +# A green visual run means nothing without its `[snap_diff]` line + +The gem resolves every baseline with `git show HEAD:`. Anything that +breaks git turns the whole visual gate into a no-op: captures are still +written over the baselines, nothing is compared, and the run is green. + +**Measured 2026-08-22.** Run from a git WORKTREE, `.git` is a FILE pointing at +an absolute host path under the main repo (`
/.git/worktrees/`), +which compose does not mount. Git inside the container reports "not a git +repository", every baseline lookup returns empty, and a contact page carrying +`body { background: red !important }` - **difference_level 0.68, 68% of the +frame** - passed with `0 failures`. The tell was one missing line: macOS runs +end with `[snap_diff] N screenshots compared`, the container run printed no +such line at all. + +**Two fixes, each verified by breaking it:** + +- `Capybara::Screenshot::Diff.fail_if_new = true` (test/support/setup_snap_diff.rb). + Outside CI the gem defaulted this to false, so an unresolvable baseline + passed silently - that default is what made the no-op invisible rather than + loud. Probe with a screenshot name absent from git: `1 runs, 1 assertions, + 1 failures`. A genuinely new page now fails its first run until recorded, + which is the same run-to-fail -> inspect -> commit flow this file already + documents. +- `bin/dtest` now MAKES git work from a worktree instead of refusing to run: + it mounts the common git dir (`git rev-parse --path-format=absolute + --git-common-dir`) at `/gitcommon` and sets `GIT_DIR=/gitcommon/worktrees/ + ` + `GIT_WORK_TREE=/app`. The worktree's own `commondir` file is + relative (`../..`), so it resolves inside the container with nothing + rewritten. Gated on `[ -f .git ]`, so a normal checkout is untouched. + Verified by re-injecting the red body: `[snap_diff] 55 screenshots compared, + 10 failures`, with `desktop/contact_us` and `mobile/contact_us` among them. + +**Reading rule: a visual run that does not print `[snap_diff] N screenshots +compared` compared nothing.** Check for that line before believing green - +counting `0 failures` is not the same as counting comparisons. + +# arm64 dtest and amd64 CI genuinely disagree on text-heavy pages + +Once the worktree no-op above was fixed and comparisons actually ran, `bin/dtest` +on an ARM Mac failed **8 `mobile/blog/special/codeblocks/*` screenshots at +~0.055-0.063** against the CI-recorded baselines. Syntax-highlighted code is +the most rasteriser-sensitive surface on the site, and `bin/dc` pins +`DOCKER_DEFAULT_PLATFORM=linux/arm64/v8` while CI records on amd64. + +**A retraction, because the method matters more than the conclusion.** Earlier +the same day this drift was declared refuted, on a measurement that compared +the committed baselines against *themselves*: the candidates had already been +restored by dtest's cleanup before they were measured. A vacuous comparison +returns 0 and looks like proof of agreement. Measure candidates BEFORE any +restore runs, or measure nothing. + +Consequence: on an ARM Mac, dtest is expected-red on that family. The options, +none free - accept and screen those keys, pin the container to `linux/amd64` +so local matches CI (correct, slower under emulation), or let CI own the Linux +leg entirely. Not decided here; recorded so the next person does not read the +red as a regression they caused. + +# Running N worktrees in parallel + +Five agents in five worktrees can run `bin/dev`, `bin/test` and `bin/dtest` +concurrently. What was actually shared, and what was not: + +| Surface | Shared? | Resolution | +|---|---|---| +| compose project name | **was fixed `jtcom`** | derived per worktree in `bin/dc` - containers, networks and volumes are namespaced by it, and `bin/docked` passes `--remove-orphans`, so two runs under one project would delete each other's container mid-test | +| `bin/dev` port | **was fixed 1313** | derived per worktree (stable across restarts, so handed-out review links keep working); the main checkout keeps 1313; explicit `PORT` still wins | +| compose `hugo` published port | **was fixed 1313** | `${HUGO_PORT:-1313}` | +| `bin/test` server port | no | Capybara picks a free port unless `TEST_SERVER_PORT` is set | +| `bin/dtest` `TEST_SERVER_PORT=1314` | no | inside the container's own network namespace, never published | +| `_dest/public-*` builds | no | each worktree has its own tree | +| screenshot fixtures | no | per worktree; and each worktree has its own git index, so concurrent `git checkout --` does not contend | +| the git STASH stack | **yes** | unchanged hazard - never a bare `git stash` (see CLAUDE.md) | + +Per-worktree compose projects mean per-worktree named volumes, so the first +`bin/dtest` in a new worktree repopulates `hugo_cache_dtest`. Gems are baked +into the image, not a volume, so nothing re-bundles. diff --git a/.okf/log.md b/.okf/log.md index 868f42e23..b2875ba99 100644 --- a/.okf/log.md +++ b/.okf/log.md @@ -51,6 +51,32 @@ make it green: restructure same-day entries under one heading, and add `timestamp` to the 23 concepts missing it (anchored to each file's last commit time, which is verifiable - never invented). +## 2026-08-22 - the Linux visual gate was green because it was not testing + +`bin/dtest` run from a git worktree compared NOTHING. A worktree's `.git` is a +file pointing at an absolute host path under the main repo, compose mounts only +the worktree, so git inside the container cannot resolve it - and the gem reads +every baseline via `git show HEAD:`. Captures were written over the +baselines, nothing was compared, every run was green. + +Proven rather than inferred: injected `body { background: red !important }`, +confirmed the rule reached the built bundle and that the page references that +fingerprinted CSS, then measured the captured PNG - candidate [255,0,0], +baseline [255,255,255], **difference_level 0.68** - and the run reported +`0 failures`. Three earlier injections had failed to go red and I had blamed +the injections; the fourth proved the gate. + +I had reported earlier the same day that "bin/dtest is green, nothing to fix on +the Linux side". That was worthless: it was green because it was not testing. +Paul's main checkout has a real .git, so his dtest DID compare - which is why +he saw 31 modified baselines from a red run and I could never reproduce one. + +Root cause is not worktree-specific: outside CI the gem defaults `fail_if_new` +to false, so a baseline it cannot RETRIEVE is treated as a new screenshot and +passes. Now `true`, verified by a probe with a name absent from git. The tell to +remember is a missing line - a real run ends with `[snap_diff] N screenshots +compared`; the no-op run printed none, while still reporting `0 failures`. + ## 2026-08-22 - closing three of the fault-injection gaps, each broken before it was trusted Three of the five misses above now have gates. Every one of them was injected, diff --git a/bin/dc b/bin/dc index 738681103..a0a4a3405 100755 --- a/bin/dc +++ b/bin/dc @@ -12,7 +12,20 @@ if [ "$(uname -s)" = "Darwin" ]; then esac fi -echo "Running docker compose with project name: ${DOCKED_PROJECT_NAME:-jtcom}" +# One compose project PER WORKTREE. The project name namespaces containers, +# networks and volumes, so a fixed "jtcom" meant two worktrees running +# bin/dtest at once shared them - and `--remove-orphans` (bin/docked) would +# delete the other run's container mid-test. Derived from the checkout root so +# it is stable across runs in one worktree and distinct across worktrees; +# sanitised because compose only accepts [a-z0-9_-]. +if [ -z "${DOCKED_PROJECT_NAME:-}" ]; then + _root="$(git rev-parse --show-toplevel 2>/dev/null || pwd)" + _slug="$(basename "$_root" | tr '[:upper:]' '[:lower:]' | tr -c 'a-z0-9_-' '-')" + DOCKED_PROJECT_NAME="jtcom-${_slug%-}" + export DOCKED_PROJECT_NAME +fi + +echo "Running docker compose with project name: ${DOCKED_PROJECT_NAME}" pwd export DOCKER_BUILDKIT=1 @@ -20,5 +33,5 @@ export COMPOSE_DOCKER_CLI_BUILD=1 docker "compose" \ -f ".dev/compose.yml" \ - -p "${DOCKED_PROJECT_NAME:-jtcom}" \ + -p "${DOCKED_PROJECT_NAME}" \ "$@" diff --git a/bin/dev b/bin/dev index 96a46b1cd..c73d61e66 100755 --- a/bin/dev +++ b/bin/dev @@ -12,7 +12,19 @@ SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" PATH="$SCRIPT_DIR/../node_modules/.bin:$PATH" export PATH -PORT="${PORT:-1313}" +# Default the port PER WORKTREE instead of everyone landing on 1313. Derived +# from the checkout root, so it is stable across restarts in one worktree (the +# review links you hand out keep working) and distinct across parallel +# worktrees. Explicit PORT still wins. +if [ -z "${PORT:-}" ]; then + _root="$(git rev-parse --show-toplevel 2>/dev/null || pwd)" + if [ "$(basename "$_root")" = "jetthoughts.github.io" ]; then + PORT=1313 # the main checkout keeps 1313 + else + _h="$(printf '%s' "$_root" | cksum | cut -d' ' -f1)" + PORT=$((20000 + _h % 15000)) # sidecar rides PORT+1000 + fi +fi echo "🚀 Hugo dev: http://localhost:$PORT (board sidecar on $((PORT + 1000)))" PORT=$((PORT + 1000)) "$SCRIPT_DIR/li-review" & LI_PID=$! diff --git a/bin/dtest b/bin/dtest index 4eb092022..0eaa7897f 100755 --- a/bin/dtest +++ b/bin/dtest @@ -10,14 +10,59 @@ bin/build-if-stale _dest/public-dtest # path would be exec'd as the command itself (mirrors bin/dtest-all). # HUGO_DEFAULT_PATH must be passed explicitly: bin/test honors it, and # PRECOMPILED_ASSETS stops the container (which has no hugo) from building. -echo "Running tests in Docker..." -bin/docked --env PRECOMPILED_ASSETS=true --env TEST_SERVER_PORT=1314 \ - --env HUGO_DEFAULT_PATH=_dest/public-dtest t bin/test "$@" - -# Green run: the container rewrote the linux/ baselines through the bind mount -# with sub-tolerance Rosetta pixel drift (~0.004%/run); restore so the host -# dirty-fixture guard doesn't block the next bin/test/bin/qtest. Red runs exit -# above (set -e) and keep candidates for inspection. +# The container rewrites every linux/ baseline it captures, through the bind +# mount, whether the run is green or red. Cleanup therefore happens at BOTH +# ends and neither end is a special case: +# +# BEFORE, because a red or killed run used to leave every captured baseline +# modified - 31 files after one red `test:critical`, not just the failures - +# and the in-container dirty-fixtures guard then aborted every subsequent +# run until someone reset by hand. +# +# AFTER, via trap, so a red or interrupted run cleans up after itself. The +# old restore sat on the success path only, below `set -e`. +# +# Discarding those candidates costs nothing: bin/dc pins +# DOCKER_DEFAULT_PLATFORM=linux/arm64/v8 on an ARM Mac while the committed +# linux/ baselines are recorded by CI on amd64, so a candidate produced here +# is never committable - and it is one `git add -A` from corrupting the set. +# The only legitimate linux re-record is the CI `update-baselines` dispatch +# (.okf/build/test-gates.md). Diff artifacts (*.diff.png, *.heatmap.diff.png) +# are untracked and survive the restore, so a red run stays inspectable. if [ -z "${FORCE_SCREENSHOT_UPDATE:-}" ]; then - git checkout -- test/fixtures/screenshots/linux + git checkout -- test/fixtures/screenshots/linux 2>/dev/null || true + trap 'git checkout -- test/fixtures/screenshots/linux 2>/dev/null || true' EXIT +fi + +# Make git work inside the container when this is a git WORKTREE. +# +# A worktree's .git is a FILE holding `gitdir:
/.git/worktrees/` - +# an absolute HOST path that compose never mounts. Git in the container then +# reports "not a git repository", and because snap_diff resolves every +# baseline with `git show HEAD:`, the visual gate silently compares +# NOTHING: captures are written over the baselines and every run is green. +# Measured 2026-08-22 - a contact page with `body { background: red }`, +# difference_level 0.68, passed with 0 failures, and the only tell was a +# missing `[snap_diff] N screenshots compared` line. +# +# Mount the COMMON git dir (the main repo's .git, which holds objects/refs and +# every worktree's entry) and point GIT_DIR at this worktree inside it. The +# worktree's own `commondir` file is relative (`../..`), so it resolves there +# without rewriting anything. A normal checkout has a .git DIRECTORY, already +# inside the mount, and needs none of this. +git_args=() +if [ -f .git ]; then + git_common=$(git rev-parse --path-format=absolute --git-common-dir) + git_wt=$(basename "$(git rev-parse --absolute-git-dir)") + git_args=( + -v "${git_common}:/gitcommon" + --env "GIT_DIR=/gitcommon/worktrees/${git_wt}" + --env "GIT_WORK_TREE=/app" + ) + echo "bin/dtest: git worktree detected - mounting ${git_common} so baselines resolve" fi + +echo "Running tests in Docker..." +bin/docked ${git_args[@]+"${git_args[@]}"} \ + --env PRECOMPILED_ASSETS=true --env TEST_SERVER_PORT=1314 \ + --env HUGO_DEFAULT_PATH=_dest/public-dtest t bin/test "$@" diff --git a/test/support/setup_snap_diff.rb b/test/support/setup_snap_diff.rb index 10dc3de5c..ccbbd6a8c 100644 --- a/test/support/setup_snap_diff.rb +++ b/test/support/setup_snap_diff.rb @@ -25,6 +25,23 @@ Capybara::Screenshot::Diff.perceptual_threshold = 2.0 Capybara::Screenshot::Diff.delayed = true +# A screenshot whose baseline cannot be RETRIEVED must fail, not pass. The gem +# resolves baselines with `git show HEAD:`, so anything that breaks git +# silently turns the whole visual gate into a no-op: captures are written, +# nothing is compared, every run is green. +# +# Measured 2026-08-22: run from a git WORKTREE, `.git` is a file pointing at an +# absolute host path under the main repo, which the container does not mount - +# so git inside it reports "not a git repository", every baseline lookup +# returns nothing, and a contact page with a `background: red !important` body +# (difference_level 0.68) passed with 0 failures. Defaulting fail_if_new to +# false outside CI is what made that invisible. +# +# Consequence: a genuinely NEW page fails its first run until recorded. That is +# the same run-to-fail -> inspect -> commit flow .okf/build/test-gates.md +# already documents for re-records; record mode below still bypasses it. +Capybara::Screenshot::Diff.fail_if_new = true + if ENV["FORCE_SCREENSHOT_UPDATE"] == "true" Capybara::Screenshot::Diff.fail_on_difference = false # Record mode must be able to create FIRST baselines for new pages: with