Skip to content

fix(ci): halve smoke-lite concurrency, which #715 made too high for podman - #718

Merged
pofallon merged 1 commit into
mainfrom
fix/smoke-lite-concurrency-podman
Aug 28, 2026
Merged

fix(ci): halve smoke-lite concurrency, which #715 made too high for podman#718
pofallon merged 1 commit into
mainfrom
fix/smoke-lite-concurrency-podman

Conversation

@pofallon

Copy link
Copy Markdown
Contributor

test_compose_override_command_lifecycle_runs has failed 2 of the 3 Podman runs since #715, always the same way — podman cannot start an exec session:

container create failed (no logs from conmon): conmon bytes ""

which surfaces as a bare deacon up failed and reads like a lifecycle defect. It isn't one.

It's contention, and #715 caused it

Compose calls used to run on a hardcoded docker even under DEACON_CONTAINER_RUNTIME=podman. So on the Podman lane, four concurrent smoke-lite tests were four concurrent docker compose projects. Now they're four concurrent rootless podman ones, each with its own conmon and network helper, on a shared runner.

Evidence

Result
Podman CI since #715 2 of 3 runs failed, same test, same conmon error
Same test locally, serial, real rootless podman 4.9.3 5/5 pass
Full integration suite locally under podman 433/433 pass

Passes serially, fails under concurrency — that's contention, not a defect in the test or the code.

This is the blunt lever, deliberately

[test-groups] has no per-profile form, so 4 → 2 slows smoke tests on every lane, not just Podman. I'm taking that trade because a lane failing two runs in three is worse than a slower one — but flagging it rather than burying it.

The narrower alternative, if the wall-clock cost shows up: move only the compose smoke binaries into their own group and leave the rest of smoke-lite at 4. Happy to switch to that instead.

Not a re-run

I re-ran the first occurrence and it passed, which is why I didn't act then — one data point with a plausible story isn't evidence. Three data points, a failed reproduction attempt, and a mechanism that explains why it passes locally and fails in CI is a different situation.

🤖 Generated with Claude Code

https://claude.ai/code/session_016SFzA2sTh2EpX8MZU3TWNS

…odman

`test_compose_override_command_lifecycle_runs` has failed 2 of the 3 Podman runs
since #715, always the same way: podman cannot start an exec session and reports

  container create failed (no logs from conmon): conmon bytes ""

which surfaces as a bare `deacon up failed` and reads like a lifecycle defect.

It is contention. The same test passes 5/5 locally when run serially against real
rootless podman 4.9.3, and the full suite passes 433/433 there. What changed is
underneath the test: compose calls used to run on a hardcoded `docker` even under
DEACON_CONTAINER_RUNTIME=podman, so on the Podman lane four concurrent smoke-lite
tests were four concurrent DOCKER compose projects. Now they are four concurrent
rootless podman ones, each with its own conmon and network helper, on a shared
runner.

This is the blunt lever: `[test-groups]` has no per-profile form, so 4 -> 2 slows
smoke tests on every lane, not only Podman. Taken deliberately -- a lane that
fails two runs in three is worse than a slower one, and the alternative of moving
only the compose smoke binaries to their own group is a narrower fix that can
replace this if the cost shows up in wall-clock.

Not a re-run: three data points, one reproduction attempt, and a mechanism that
explains why it passes locally and fails in CI.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016SFzA2sTh2EpX8MZU3TWNS
@github-actions github-actions Bot added the fix Bug fix label Aug 28, 2026
@pofallon
pofallon merged commit a1ada8d into main Aug 28, 2026
16 checks passed
@pofallon
pofallon deleted the fix/smoke-lite-concurrency-podman branch August 28, 2026 11:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant