Skip to content

fix(compose): run Compose on the resolved runtime, not a literal docker - #715

Merged
pofallon merged 3 commits into
mainfrom
fix/compose-follows-resolved-runtime
Aug 28, 2026
Merged

fix(compose): run Compose on the resolved runtime, not a literal docker#715
pofallon merged 3 commits into
mainfrom
fix/compose-follows-resolved-runtime

Conversation

@pofallon

Copy link
Copy Markdown
Contributor

Closes #710. Closes #706.

What was wrong

deacon ran docker compose no matter which runtime it had resolved. Under --runtime podman the service image was built into docker's store, and every podman-side step after it — tag, run, exec — reported image not known.

Two layers, both wrong for different reasons:

  • deacon build's compose path built its ComposeManager with ComposeManager::new(), whose docker_path defaults to the literal "docker". That ignored --docker-path as well, not just --runtime.
  • up, down, exec, up's port-event handler and read-configuration each passed the raw --docker-path flag. binary_for maps an untouched default to the flavor's own name only inside resolve_runtime, so with no explicit path the flag still read "docker".

Same class as #708, one layer up: the command validated against the resolved runtime and then invoked a different one.

Measured

Shim on --docker-path (logs argv, execs the real binary), real compose config:

compose invocations reaching the named binary
build, before 0 — only -v, image inspect, tag
build, after 3, the service build among them
up --runtime podman, before 0 — only ps, inspect
up --runtime podman, after 6, plus every ps / inspect / exec

Two false diagnoses, corrected in the ledger

Both prior localhost/ attempts on #706 were derived from a naming diagnosis and both were measured wrong. They could not have worked: the image was in another daemon's store, so no spelling would find it.

The follow-up issue's own hypothesis was false too — it supposed podman's external compose provider talks to the docker daemon. Measured by pointing compose_providers at a script that dumps its environment, podman does the opposite: it points the provider at its own socket, exporting DOCKER_HOST=unix:///run/podman/podman.sock (plus DOCKER_BUILDKIT=0 and an emptied DOCKER_CONFIG). deacon never reached that delegation.

A compensation removed with it

execute_compose_lifecycle deliberately built its exec client from --docker-path rather than the resolved runtime, with a comment recording that docker compose created the project in the docker daemon so podman exec could not find it. That was a correct workaround for the wrong split — and would have re-opened the gap from the other side now that compose follows the runtime. It follows runtime_bin like everything else.

Guard

build_runtime_routing::compose_build_runs_on_the_runtime_named_by_docker_path — hermetic, no daemon, #[cfg(unix)]. Sabotage-verified: reverting the build fix fails it with exactly the pre-fix log (-v, image inspect, and nothing else).

Verification

  • make test-nextest-fast — 3678 passed
  • integration_compose_features_build on Docker — 7/7
  • compose + read-configuration suites — 84 passed
  • down / exec / runtime-selection — 41 passed
  • fmt + clippy --all-targets --all-features clean

integration_compose_features_build returns to the Podman CI step as a whole binary; that lane goes from 29 selected tests to 36.

🤖 Generated with Claude Code

https://claude.ai/code/session_016SFzA2sTh2EpX8MZU3TWNS

deacon ran `docker compose` no matter which runtime it had resolved, so under
`--runtime podman` the service image was built into DOCKER's store and every
podman-side step after it — tag, run, exec — reported `image not known`.

Two layers were wrong. `deacon build`'s compose path built its ComposeManager
with `ComposeManager::new()`, whose `docker_path` defaults to the literal
"docker" and which therefore ignored `--docker-path` too. `up`, `down`, `exec`,
`up`'s port-event handler and `read-configuration` each passed the raw
`--docker-path` flag, which `binary_for` maps to the flavor's own name only
inside `resolve_runtime` — so with no explicit path it still read "docker".

Measured with a shim on `--docker-path`, on a real compose config: before, the
named binary received `-v`, `image inspect` and `tag`, and zero `compose`
invocations. After: three, the service build among them. For `up --runtime
podman` the same shim went from `ps`+`inspect` only to six `compose` calls plus
every `ps`, `inspect` and `exec`.

`execute_compose_lifecycle` had a compensation for this that has to go with it:
it deliberately built its exec client from `--docker-path` rather than the
resolved runtime, with a comment recording that `docker compose` created the
project in the docker daemon so `podman exec` could not find it. Correct for the
wrong split, and it would re-open the gap from the other side now that compose
follows the runtime.

Two earlier `localhost/` fixes for the first symptom were derived from a naming
diagnosis and both were measured wrong; the image was in another daemon's store
the whole time, so no spelling could have found it. The ledger row records the
correction, including the fact that the follow-up issue's own hypothesis — that
podman's external compose provider talks to the docker daemon — is also false:
podman points the provider at its own socket, exporting
DOCKER_HOST=unix:///run/podman/podman.sock. deacon never reached that
delegation.

Guarded by a hermetic test (no daemon) that records invocations through a shim
and asserts a `compose` call reaches the named binary; verified to fail on the
pre-fix code. `integration_compose_features_build` returns to the Podman CI step
as a whole binary, taking that lane from 29 selected tests to 36.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016SFzA2sTh2EpX8MZU3TWNS
…se gap

Measured against real rootless podman 4.9.3 rather than inferred from the fix,
which turned up two things the lane needed that the fix alone does not give it.

The socket. `podman compose` is a shim: it execs an external provider — the
docker-compose already on the runner — and points it at podman by exporting
DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock. This step's comment said
"no socket" and meant it, so every compose call would have failed with `Cannot
connect to the Docker daemon` — a fact about the job, not about deacon. Started
directly rather than via `systemctl --user`, so the lane does not depend on a
user systemd session existing on the runner image, and the step fails loudly with
the service log if it never answers.

The BuildKit gap. Podman sets DOCKER_BUILDKIT=0 for the provider, pinning
docker-compose to the classic builder, while deacon passes Feature content as an
additional build context — BuildKit-only. Three of the binary's seven fail on
`the classic builder doesn't support additional contexts`, all `build:`-shape
services; the `image:` shape builds its Feature image with `podman build`
directly and passes. That is a fourth capability gap of the same kind as the
three buildx exporter ones, so it is excluded by name next to them rather than
by taking the whole binary back out.

Measured result: 4 of 7 pass, including
`compose_features_image_shape_installs_feature` — the test #706 was originally
about. The lane selects 30 rather than 26.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016SFzA2sTh2EpX8MZU3TWNS
@pofallon

Copy link
Copy Markdown
Contributor Author

Validated against real rootless podman

I got podman working properly in the devcontainer (#716) and re-measured this on a genuine podman substrate instead of reasoning about it. Two corrections to the PR as it stood, both pushed in f1221dc.

The lane needed podman's API socket started, and its setup step explicitly disclaimed one (# deacon's PodmanRuntime drives the podman CLI directly (no socket)). Since podman compose execs an external provider and points it at that socket via DOCKER_HOST, every compose call would have failed with Cannot connect to the Docker daemon — a fact about the job, not about deacon. Started directly rather than through systemctl --user, so the lane doesn't depend on a user systemd session existing on the runner image.

Three of the seven hit a fourth capability gap and stay excluded, by name, alongside the three buildx exporter gaps. Podman sets DOCKER_BUILDKIT=0 for the compose provider, pinning docker-compose to the classic builder; deacon passes Feature content as an additional build context, which only BuildKit supports:

the classic builder doesn't support additional contexts, set DOCKER_BUILDKIT=1 to use BuildKit

Only build:-shape services hit it — the image: shape builds its Feature image with podman build directly.

Measured result

4 of 7 pass, including compose_features_image_shape_installs_feature — the test #706 was originally filed about. The three failures are the BuildKit gap above, confirmed individually rather than assumed from one sample.

The lane selects 30 tests, not the 26 it did before. I'd earlier written "36" in the ledger; that was a shell-quoting artifact in my own verification command, and it's corrected.

What this changes about the fix itself

Nothing — it confirms it. On real podman, podman compose build puts its image in podman's own store, where podman tag finds it and podman run executes it. That was the exact operation #710 reported as impossible under every spelling, and it was impossible only because deacon was running docker compose.

The Podman lane went red on four compose smoke tests, and all four are the same
TEST defect #705 already fixed in `integration_build{,_output}` — the assertion
shells out to a hardcoded `docker` to inspect what deacon produced, reading a
different daemon than the deacon under test wrote to.

They passed before only because compose itself also ran on docker. Making the
compose client follow the resolved runtime moved the containers to podman and
left the assertions looking in the old place, so `docker exec <id>` could not
find a container that exists — reported as `postCreateCommand should have created
/tmp/deacon-lifecycle-marker`, which reads like a lifecycle bug and is not one.

Uses the existing `support::runtime_bin()` rather than adding a ninth per-file
copy of the helper. No behavior change on docker, where it returns "docker".

CI's full failure set was exactly these four out of 433; verified locally under
real rootless podman 4.9.3 (10/10) and on docker (10/10).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016SFzA2sTh2EpX8MZU3TWNS
@pofallon

Copy link
Copy Markdown
Contributor Author

Podman lane was red; fixed in 246567a

Four failures out of 433, and all four are the same test defect #705 already fixed in integration_build{,_output} — the assertion shells out to a hardcoded docker to inspect what deacon produced, reading a different daemon than the deacon under test wrote to.

deacon::smoke_compose_edges            test_compose_down_stops_but_keeps_containers
deacon::smoke_compose_override_command test_compose_override_command_default_keeps_service_alive
deacon::smoke_compose_override_command test_compose_override_command_lifecycle_runs
deacon::smoke_compose_override_command test_compose_override_command_explicit_false_runs_natural_command

They passed before only because compose itself also ran on docker. Moving the compose client onto the resolved runtime moved the containers to podman and left the assertions looking in the old place, so docker exec <id> couldn't find a container that exists. It surfaced as:

postCreateCommand should have created /tmp/deacon-lifecycle-marker

which reads like a lifecycle bug and isn't one — worth flagging, because that message would send the next person after the wrong thing.

Uses the existing support::runtime_bin() rather than adding a ninth per-file copy of the helper.

Verified

  • under real rootless podman 4.9.3: 10/10 (was 6/10)
  • on docker: 10/10runtime_bin() returns "docker" by default, so no behavior change there
  • CI's full failure set was exactly these four; Test (MVP integration) (docker) was already green

This is the honest-lane effect from #708 showing up one layer further out: the fix didn't break these tests, it stopped them from being verified against the wrong daemon.

@pofallon
pofallon merged commit d723c12 into main Aug 28, 2026
24 of 25 checks passed
@pofallon
pofallon deleted the fix/compose-follows-resolved-runtime branch August 28, 2026 02:42
pofallon added a commit that referenced this pull request Aug 28, 2026
…odman (#718)

`test_compose_override_command_lifecycle_runs` has failed 2 of the 3 Podman runs
since #715, always the same way: podman cannot start an exec session and reports

  container create failed (no logs from conmon): conmon bytes ""

which surfaces as a bare `deacon up failed` and reads like a lifecycle defect.

It is contention. The same test passes 5/5 locally when run serially against real
rootless podman 4.9.3, and the full suite passes 433/433 there. What changed is
underneath the test: compose calls used to run on a hardcoded `docker` even under
DEACON_CONTAINER_RUNTIME=podman, so on the Podman lane four concurrent smoke-lite
tests were four concurrent DOCKER compose projects. Now they are four concurrent
rootless podman ones, each with its own conmon and network helper, on a shared
runner.

This is the blunt lever: `[test-groups]` has no per-profile form, so 4 -> 2 slows
smoke tests on every lane, not only Podman. Taken deliberately -- a lane that
fails two runs in three is worse than a slower one, and the alternative of moving
only the compose smoke binaries to their own group is a narrower fix that can
replace this if the cost shows up in wall-clock.

Not a re-run: three data points, one reproduction attempt, and a mechanism that
explains why it passes locally and fails in CI.


Claude-Session: https://claude.ai/code/session_016SFzA2sTh2EpX8MZU3TWNS

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci CI/CD changes fix Bug fix

Projects

None yet

1 participant