bin/dtest: the Linux visual gate was green because it compared nothing - #578
Merged
Conversation
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
bin/dtest run from a git worktree compared NOTHING. A worktree's .git is a
file pointing at an absolute host path under the main repo; compose mounts
only the worktree, so git inside the container cannot resolve it - and the
gem reads every baseline via `git show HEAD:<path>`. Captures were written
over the baselines, nothing was compared, every run was green.
Proven, not inferred: injected `body { background: red !important }`,
confirmed the rule reached the built bundle and that the page references that
fingerprinted CSS, then measured the captured PNG - candidate [255,0,0],
baseline [255,255,255], difference_level 0.68 - and the run reported
0 failures. Three earlier injections had failed to go red and I blamed the
injections; the fourth proved the gate.
Root cause is NOT worktree-specific. Outside CI the gem defaults fail_if_new
to false, so a baseline it cannot RETRIEVE is treated as a new screenshot and
passes silently. That default is what made the no-op invisible instead of
loud, and it would hide any future breakage of baseline resolution.
Two fixes, each verified by breaking it:
- fail_if_new = true. Probe with a screenshot name absent from git:
"1 runs, 1 assertions, 1 failures". A genuinely new page now fails its
first run until recorded - the same run-to-fail -> inspect -> commit flow
test-gates.md already documents.
- bin/dtest aborts from a worktree (exit 2) naming the unreachable gitdir,
rather than emitting 29 baseline errors that describe a symptom.
Also folded in: dtest now restores linux/ baselines at BOTH ends. The old
restore sat on the success path below `set -e`, so a red or killed run left
every captured baseline modified - 31 files after one red critical run, not
just the failures - and the in-container dirty-fixtures guard then aborted
every subsequent run until someone reset by hand. Discarding those candidates
costs nothing: bin/dc pins linux/arm64/v8 on an ARM Mac while the committed
linux baselines come from CI on amd64, so a candidate produced locally is
never committable. Untracked .diff.png artifacts survive the restore.
The reading rule now in test-gates.md: a visual run that does not print
"[snap_diff] N screenshots compared" compared nothing. Counting "0 failures"
is not the same as counting comparisons.
Verified: test:critical 38 runs, 126 assertions, 0 failures,
[snap_diff] 55 screenshots compared, fixtures untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ
Refusing was the cowardly fix. A worktree's .git is a file pointing at <main>/.git/worktrees/<name>, and its commondir entry is RELATIVE (../..) - so mounting the common git dir at one path inside the container makes everything resolve with nothing rewritten: GIT_DIR=/gitcommon/worktrees/<name>, GIT_WORK_TREE=/app. Gated on `[ -f .git ]`, so a normal checkout takes none of it. Verified by re-injecting the red body that previously passed with 0 failures: now "[snap_diff] 55 screenshots compared, 10 failures", with desktop/contact_us and mobile/contact_us among them. The gate compares from a worktree. The same run exposed something the no-op had been hiding: 8 mobile/blog/special/codeblocks screenshots fail at ~0.06 because the container renders arm64 while CI records amd64. Earlier today I called that drift refuted - on a measurement that compared the baselines against themselves, because dtest's cleanup had already restored the candidates before I measured. A vacuous comparison returns 0 and reads as agreement. Recorded in test-gates with the options, so the next person does not read that red as their own regression. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ
Two things were genuinely shared and would collide; everything else already
isolated correctly and was left alone.
The compose project name was fixed at "jtcom". It namespaces containers,
networks and volumes, so two worktrees running bin/dtest concurrently shared
all three - and bin/docked passes --remove-orphans, which would delete the
other run's container mid-test. Now derived from the checkout root: stable
across runs in one worktree, distinct across worktrees, sanitised to compose's
[a-z0-9_-].
bin/dev defaulted to 1313 for everyone, with the board sidecar on PORT+1000.
Now derived per worktree from the checkout root, so it is stable across
restarts (review links handed out earlier keep working) and distinct between
worktrees. The main checkout keeps 1313 so nothing changes for a human at the
keyboard; an explicit PORT still wins. Verified across five paths: 1313 /
22782 / 22102 / 32027 / 25006, no port and no sidecar colliding, and the same
worktree returning the same number twice.
The compose hugo service also published a fixed 1313; it is ${HUGO_PORT:-1313}
now.
Deliberately unchanged, because they are already safe: bin/test lets Capybara
pick a free port unless TEST_SERVER_PORT is set; bin/dtest's TEST_SERVER_PORT
1314 lives inside the container's network namespace and is never published;
_dest/public-* and the screenshot fixtures are per-worktree, and each worktree
has its own git index so concurrent `git checkout --` does not contend. The
git STASH stack stays shared - that hazard is unchanged and already documented.
Cost: per-worktree compose projects mean per-worktree named volumes, so the
first bin/dtest in a new worktree repopulates hugo_cache_dtest. Gems are baked
into the image rather than a volume, so nothing re-bundles.
Verified: bin/dev prints 22782 here (matching the derivation), compose config
validates under the new project name and HUGO_PORT.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ
pftg
added a commit
that referenced
this pull request
Aug 22, 2026
…ed (#582) STATUS.md's Test/CI row still said "one known red (stale linux baseline)". That key was a real content difference and is fixed - re-recorded on master, where recorder and tester see the same tree, verified green by CI run 32565008850 with zero failing keys. What replaced it is genuinely open and needs Paul: on an ARM Mac dtest now legitimately fails 8 codeblocks screenshots, because the container renders arm64 while the committed linux baselines come from CI on amd64. That was invisible until #578 fixed dtest comparing nothing at all from a worktree, so it is pre-existing rather than a regression. Three options recorded with their trade-off, plus the contradiction to reconcile first: .dev/compose.yml declares platform linux/amd64 on the `t` service while bin/dc exports DOCKER_DEFAULT_PLATFORM=linux/arm64/v8. Also refreshed the 2608 sprint summary, which still claimed "#560-#567" and "CI Linux red went from 22 keys to 1" - it is 0 now, and the range runs to #578. Claude-Session: https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
pftg
added a commit
that referenced
this pull request
Aug 22, 2026
Paul asked whether the post was delivered. It was not, and I would not have noticed - I had reported it shipped. `how-to-audit-content-you-didnt-write` carried `date: 2026-08-23`. Today is the 22nd. Production does not build future-dated content, so the post merged to master, passed every gate, and was absent from the live site. Date corrected to 2026-08-22. **Both pieces of evidence I cited were vacuous, and that is the part worth keeping.** `bin/hugo-build` passed - it writes to `_dest/public-dev`, which DOES build the future, so my post was there and I read that as proof. Production writes elsewhere. The build never claimed what I took it to claim. `bin/rake test:links` reported zero errors across 1,768 pages - because the post was not among them. A crawl that skips the thing you wrote reports a clean green for exactly that reason. It scanned everything except the artifact under test. This is the same shape as the dtest incident (#578, green because it compared nothing) and the reason the fault-injection work exists: a gate that cannot see the artifact reports success indistinguishable from real success. The falsifiable check, now in blog-pipeline.md STEP 7 and the blog-write skill: build production, find the slug in it, and QUOTE THE PAGE-COUNT DELTA. This run went 1,768 -> 1,775 pages when the post entered the set. That delta is the evidence; "the build passed" is not. bin/rake test:links find _dest/public-linkcheck -type d -name "<slug>" "Merged" and "delivered" are different claims. Only the second needs this check, and I made the first while reporting the second. Verified after the fix: post present in _dest/public-linkcheck, 1,775 pages, 31,908 unique links, zero errors. Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #576. Started as "fix the dirty-baseline friction", became a real defect.
The defect
bin/dtestrun from a git worktree compared nothing. A worktree's.gitis a file pointing at an absolute host path under the main repo; compose mounts only the worktree, so git in the container reports "not a git repository" — and the gem resolves every baseline viagit show HEAD:<path>. Captures were written over the baselines, nothing compared, every run green.Proven, not inferred: injected
body { background: red !important }, confirmed the rule reached the built bundle and that the page references that fingerprinted CSS, then measured the captured PNG — candidate[255,0,0], baseline[255,255,255], difference_level 0.68 — and the run reported0 failures.Three earlier injections had failed to go red and I blamed the injections. The fourth proved the gate. Worth stating: I reported "bin/dtest is green, nothing to fix on the Linux side" earlier the same day — that was worthless, it was green because it wasn't testing.
Root cause is NOT worktree-specific
Outside CI the gem defaults
fail_if_newto false, so a baseline it cannot retrieve is treated as a new screenshot and passes silently. That default is what made the no-op invisible rather than loud, and it would hide any future breakage of baseline resolution.Fixes, each verified by breaking it
fail_if_new = true1 runs, 1 assertions, 1 failuresbin/dtestaborts from a worktreeEXIT=2, naming the unreachable gitdirlinux/at both ends viatrapThe old restore sat on the success path below
set -e, so a red or killed run left every captured baseline modified (31 files after one red critical run, not just the failures) and the dirty-fixtures guard then killed every later run. Discarding those candidates costs nothing:bin/dcpinslinux/arm64/v8on an ARM Mac while committedlinux/baselines come from CI on amd64, so a locally-produced candidate is never committable. Untracked.diff.pngartifacts survive the restore, so red runs stay inspectable.The reading rule this produced
A visual run that does not print
[snap_diff] N screenshots comparedcompared nothing. That missing line was the entire tell. Counting0 failuresis not counting comparisons. Now in.okf/build/test-gates.md.Verified
test:critical38 runs, 126 assertions, 0 failures,[snap_diff] 55 screenshots compared, fixtures untouched.🤖 Generated with Claude Code
https://claude.ai/code/session_011SP5gaqXEgUie8pdFrmbeJ