Skip to content

test: pinned-version controls must survive the bump they guard (#114) - #237

Merged
wms2537 merged 1 commit into
mainfrom
fix/pinned-version-controls-survive-bumps
Sep 8, 2026
Merged

test: pinned-version controls must survive the bump they guard (#114)#237
wms2537 merged 1 commit into
mainfrom
fix/pinned-version-controls-survive-bumps

Conversation

@wms2537

@wms2537 wms2537 commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Found while diagnosing why Dependabot's MediaMTX PR reported VACUOUS for the control covering that exact dependency.

The shape of the problem

Six of the seven controls in scripts/test_pinned_versions.py anchored on the exact version currently in the tree:

"bluenviron/mediamtx:1.20.1",
"FROM node:24.20.0-bookworm-slim@",
'VERSION="${CADDY_VERSION:-2.11.4}"',

A bump makes the anchor vanish, so the control reports VACUOUSsilencing itself on precisely the pull request that changes the thing it guards. That is the one moment it most needs to work.

This is not hypothetical. It has already happened twice:

Both times the repair was to hand-edit the anchor to the new version — which fixes the instance and leaves the shape intact for next time.

The fix

Anchor matches whatever version is pinned now and rewrites it to a sentinel:

Anchor(r"bluenviron/mediamtx:[\d.]+", "bluenviron/mediamtx:0.0.1"),

A bump changes nothing about whether the control fires.

Verified against the failure it is for

Simulating four simultaneous bumps — MediaMTX 1.20.1→1.21.0, node 24→26, Caddy 2.11.4→2.12.0, python 3.14.7→3.15.0:

before after
controls still live 8 9
VACUOUS 1 0

Two smaller things the conversion forced

The runner now refuses a mutation that leaves the file byte-identical. A substitution that changes nothing tests nothing, however many times its anchor matched — and a pattern anchor makes that failure mode reachable in a way a literal one never was, so the guard against it belongs here.

One control asserted on the guard's message including a version number"tells operators to install lap>=0.5". That re-introduces the same fragility one layer along, so it now asserts on the sentence without the version.

What is unchanged

VACUOUS still fails the run, and should. The design was right — a control that cannot fire says so rather than passing quietly. It was the anchors that could not survive contact with Dependabot.

9/9 controls pass; check_pinned_versions.py passes on the tree as committed.

🤖 Generated with Claude Code

Six of the seven controls anchored on the exact version in the tree:

    "bluenviron/mediamtx:1.20.1"
    "FROM node:24.20.0-bookworm-slim@"

So a bump made the anchor vanish and the control reported VACUOUS —
silencing itself on precisely the pull request that changes the thing it
guards, which is the one moment it most needs to work.

This is not hypothetical. It has happened twice already: the node 22 -> 24
bump silenced two controls earlier in this work, and Dependabot's MediaMTX
1.21.0 bump (#218) silenced another. Both times the fix was to hand-edit
the anchor, which leaves the shape intact for next time.

`Anchor` matches whatever version is pinned now and rewrites it to a
sentinel, so a bump changes nothing about whether the control fires.
Verified by simulating four simultaneous bumps — MediaMTX, node, Caddy,
python — where one control previously went vacuous and now none do.

Two smaller things the conversion forced, both worth keeping:

  * the runner now refuses a mutation that leaves the file byte-identical.
    A substitution that changes nothing tests nothing, however many times
    its anchor matched — and a pattern anchor makes that failure mode
    reachable in a way a literal one never was.
  * one control asserted on the guard's message INCLUDING a version
    number ("tells operators to install lap>=0.5"). That re-introduces the
    same fragility one layer along, so it now asserts on the sentence
    without the version.

VACUOUS still fails the run. The design was right; the anchors were the
part that could not survive contact with Dependabot.

Signed-off-by: wms2537 <weimengsoh0827@gmail.com>
@wms2537
wms2537 merged commit 49220e9 into main Sep 8, 2026
16 checks passed
@wms2537
wms2537 deleted the fix/pinned-version-controls-survive-bumps branch September 8, 2026 03:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant