fix(ci): halve smoke-lite concurrency, which #715 made too high for podman - #718
Merged
Conversation
…odman `test_compose_override_command_lifecycle_runs` has failed 2 of the 3 Podman runs since #715, always the same way: podman cannot start an exec session and reports container create failed (no logs from conmon): conmon bytes "" which surfaces as a bare `deacon up failed` and reads like a lifecycle defect. It is contention. The same test passes 5/5 locally when run serially against real rootless podman 4.9.3, and the full suite passes 433/433 there. What changed is underneath the test: compose calls used to run on a hardcoded `docker` even under DEACON_CONTAINER_RUNTIME=podman, so on the Podman lane four concurrent smoke-lite tests were four concurrent DOCKER compose projects. Now they are four concurrent rootless podman ones, each with its own conmon and network helper, on a shared runner. This is the blunt lever: `[test-groups]` has no per-profile form, so 4 -> 2 slows smoke tests on every lane, not only Podman. Taken deliberately -- a lane that fails two runs in three is worse than a slower one, and the alternative of moving only the compose smoke binaries to their own group is a narrower fix that can replace this if the cost shows up in wall-clock. Not a re-run: three data points, one reproduction attempt, and a mechanism that explains why it passes locally and fails in CI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016SFzA2sTh2EpX8MZU3TWNS
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test_compose_override_command_lifecycle_runshas failed 2 of the 3 Podman runs since #715, always the same way — podman cannot start an exec session:which surfaces as a bare
deacon up failedand reads like a lifecycle defect. It isn't one.It's contention, and #715 caused it
Compose calls used to run on a hardcoded
dockereven underDEACON_CONTAINER_RUNTIME=podman. So on the Podman lane, four concurrentsmoke-litetests were four concurrent docker compose projects. Now they're four concurrent rootless podman ones, each with its own conmon and network helper, on a shared runner.Evidence
Passes serially, fails under concurrency — that's contention, not a defect in the test or the code.
This is the blunt lever, deliberately
[test-groups]has no per-profile form, so 4 → 2 slows smoke tests on every lane, not just Podman. I'm taking that trade because a lane failing two runs in three is worse than a slower one — but flagging it rather than burying it.The narrower alternative, if the wall-clock cost shows up: move only the compose smoke binaries into their own group and leave the rest of
smoke-liteat 4. Happy to switch to that instead.Not a re-run
I re-ran the first occurrence and it passed, which is why I didn't act then — one data point with a plausible story isn't evidence. Three data points, a failed reproduction attempt, and a mechanism that explains why it passes locally and fails in CI is a different situation.
🤖 Generated with Claude Code
https://claude.ai/code/session_016SFzA2sTh2EpX8MZU3TWNS