test: pinned-version controls must survive the bump they guard (#114) - #237
Merged
Merged
Conversation
Six of the seven controls anchored on the exact version in the tree:
"bluenviron/mediamtx:1.20.1"
"FROM node:24.20.0-bookworm-slim@"
So a bump made the anchor vanish and the control reported VACUOUS —
silencing itself on precisely the pull request that changes the thing it
guards, which is the one moment it most needs to work.
This is not hypothetical. It has happened twice already: the node 22 -> 24
bump silenced two controls earlier in this work, and Dependabot's MediaMTX
1.21.0 bump (#218) silenced another. Both times the fix was to hand-edit
the anchor, which leaves the shape intact for next time.
`Anchor` matches whatever version is pinned now and rewrites it to a
sentinel, so a bump changes nothing about whether the control fires.
Verified by simulating four simultaneous bumps — MediaMTX, node, Caddy,
python — where one control previously went vacuous and now none do.
Two smaller things the conversion forced, both worth keeping:
* the runner now refuses a mutation that leaves the file byte-identical.
A substitution that changes nothing tests nothing, however many times
its anchor matched — and a pattern anchor makes that failure mode
reachable in a way a literal one never was.
* one control asserted on the guard's message INCLUDING a version
number ("tells operators to install lap>=0.5"). That re-introduces the
same fragility one layer along, so it now asserts on the sentence
without the version.
VACUOUS still fails the run. The design was right; the anchors were the
part that could not survive contact with Dependabot.
Signed-off-by: wms2537 <weimengsoh0827@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while diagnosing why Dependabot's MediaMTX PR reported
VACUOUSfor the control covering that exact dependency.The shape of the problem
Six of the seven controls in
scripts/test_pinned_versions.pyanchored on the exact version currently in the tree:A bump makes the anchor vanish, so the control reports
VACUOUS— silencing itself on precisely the pull request that changes the thing it guards. That is the one moment it most needs to work.This is not hypothetical. It has already happened twice:
Both times the repair was to hand-edit the anchor to the new version — which fixes the instance and leaves the shape intact for next time.
The fix
Anchormatches whatever version is pinned now and rewrites it to a sentinel:A bump changes nothing about whether the control fires.
Verified against the failure it is for
Simulating four simultaneous bumps — MediaMTX 1.20.1→1.21.0, node 24→26, Caddy 2.11.4→2.12.0, python 3.14.7→3.15.0:
Two smaller things the conversion forced
The runner now refuses a mutation that leaves the file byte-identical. A substitution that changes nothing tests nothing, however many times its anchor matched — and a pattern anchor makes that failure mode reachable in a way a literal one never was, so the guard against it belongs here.
One control asserted on the guard's message including a version number —
"tells operators to install lap>=0.5". That re-introduces the same fragility one layer along, so it now asserts on the sentence without the version.What is unchanged
VACUOUSstill fails the run, and should. The design was right — a control that cannot fire says so rather than passing quietly. It was the anchors that could not survive contact with Dependabot.9/9 controls pass;
check_pinned_versions.pypasses on the tree as committed.🤖 Generated with Claude Code