Skip to content

ci(race): add gcc to the Linux CI image and fix the one race we own - #171

Merged
luthermonson merged 1 commit into
mainfrom
ci/race-prereqs
Aug 18, 2026
Merged

ci(race): add gcc to the Linux CI image and fix the one race we own#171
luthermonson merged 1 commit into
mainfrom
ci/race-prereqs

Conversation

@luthermonson

Copy link
Copy Markdown
Contributor

Prerequisites for running -race in CI. Split from the workflow change so this can land green — the published runner-ci-linux image has no C compiler, so a ci.yml step running -race would fail on its first run. Merging this triggers build-images.yml (it watches images/**), which publishes an image with gcc; the workflow step follows in a second PR.

-race was not running anywhere

mage cigo test -tags containers_image_openpgp -count=1 ./... (magefile.go:31). No -race, and no other workflow runs unit tests at all.

Worth knowing separately: the macOS and Windows CI jobs run zero tests — they only mage build:* and check --version. So there was never a Windows/macOS race job to break, and this leaves them alone. (runner-ci-windows is servercore with no C toolchain; the macOS image is asset-staging only.)

CGO_ENABLED=0 is set only on cross-compile build targets, never on the test path — so cgo was not being suppressed, the compiler was simply absent.

The detector was actually run — WSL, gcc 15.2, Go 1.26.2

target runs result
pkg/upgrade (#167's new watchdog) 20 clean
pkg/scheduler 20 clean
pkg/dind several 1 race — in containerd, not ours
pkg/forgerunner 30×-count=5 1 real race we own
./... (post-fix) 3 clean, ~55s

All 53 pkg/upgrade tests were confirmed to execute on Linux rather than GOOS-skip, so those 20 clean runs cover the code #167 just added. 20 runs is not proof of absence for a rare interleaving.

The race we own — fixed here

pkg/forgerunner/runner_test.go, TestPoll_ContextCancellation: Run returns the instant the deadline fires, leaving a FetchTask handler mid-flight on the httptest goroutine, and the assertion read that handler's fetchCount with no happens-before edge. Now atomic.Int64.

Intermittent — 26/30 pre-fix, 0/30 post-fix, and it did not fire on the first full-suite run. Precisely the flake that would otherwise have landed later as an unexplained red.

The race we don't own — quarantined, with the reason

authHandler.doBearerAuth reads a cached expirationTime under ah.Lock while the goroutine owning the in-flight fetch writes token/refreshToken/err/expirationTime from a defer holding no lock (core/remotes/docker/authorizer.go:289 read vs :303 write, v2.2.2). TestPushHandlerEndToEnd trips it 10/10.

containerd main was checked — still unfixed upstream, so a dependency bump will not clear it. Quarantined under -race only, via build-tagged files, and verified in both directions: SKIP under -race, still PASS in a normal mage ci. Worth filing upstream.

Changes

  • images/runner-ci-linux/Dockerfilegcc + libc6-dev, --no-install-recommends. Minimal deliberately; build-essential would pull g++/make/dpkg-dev and ~250 MB for nothing.
  • magefile.go — a TestRace target, deliberately not in mage ci so mage ci stays cgo-free and runnable on Windows dev boxes.
  • pkg/dind/racedetector_{race,norace}_test.go + the skip in registry_e2e_test.go.
  • pkg/forgerunner/runner_test.go — the atomic fix.

Not touched

Two pre-existing failures, both environmental and both failing without -race too: pkg/runner/TestFindTarball_MatchesPlatform (dev checkout has a stale Windows runner zip and no Linux tarball; CI's download.All fixes it) and pkg/workflow/TestSniffGitInfo_CurrentRepo (fails inside a git worktree, passes in a normal clone).

No image was built or pushed, and no workflow run was triggered.

Prerequisites for enabling -race in CI, split from the workflow change so
this can land green: the published runner-ci-linux image has no C
compiler, so a ci.yml step running -race would fail on its first run.

- images/runner-ci-linux/Dockerfile: gcc + libc6-dev. Minimal on purpose;
  build-essential would add g++/make/dpkg-dev and ~250MB for nothing.
- magefile.go: TestRace target, deliberately NOT in `mage ci` so
  `mage ci` stays cgo-free and runnable on Windows dev boxes.
- pkg/forgerunner/runner_test.go: a real race we own. Run returns the
  instant the deadline fires, leaving a FetchTask handler mid-flight on
  the httptest goroutine, and the assertion read fetchCount with no
  happens-before edge. Now atomic.Int64. Intermittent: 26/30 pre-fix,
  0/30 post-fix, and it did not fire on the first full-suite run.
- pkg/dind: quarantine TestPushHandlerEndToEnd under -race only. The race
  is in containerd, not us - authHandler.doBearerAuth reads cached
  expirationTime under ah.Lock while the fetching goroutine writes
  token/refreshToken/err/expirationTime from a defer holding no lock
  (core/remotes/docker/authorizer.go:289 vs :303, v2.2.2). Reproduces
  10/10. Still unfixed on containerd main, so a bump will not clear it.

Race detector run in WSL (gcc 15.2, Go 1.26.2): pkg/upgrade 20/20 clean
- including #167's new watchdog code, with all 53 tests confirmed
executing rather than GOOS-skipped - and pkg/scheduler 20/20 clean. 20
runs is not proof of absence for a rare interleaving.
@luthermonson
luthermonson merged commit 004ffac into main Aug 18, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant