From bb7ce39db2cd7dd41ca070f603b99b323694f513 Mon Sep 17 00:00:00 2001 From: Luther Monson Date: Mon, 17 Aug 2026 15:30:18 -0700 Subject: [PATCH 1/2] docs(readme): resolve the Windows `container:` self-contradiction MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The README showed a Windows `container:` job as a worked example under "Choosing the Image" and, 300 lines later, listed the same key under Known Limitations as blocked by the GitHub runner binary. Both cannot be true, and neither was accurate. The "runner blocks it" claim is wrong: nothing in ephemerd rejects `container:` on Windows (image resolution in pkg/scheduler resolveImage is OS-agnostic), and the runner binary does not refuse either — an observed Windows `container:` job failed looking for docker.exe, which the servercore default image does not ship. Whether it works once a Docker CLI is present is genuinely unknown: dind's bind-mount translation assumes a Linux runner container on overlayfs with POSIX source paths, and docs/arch/dind-bind-translation.md lists Windows-native `container:` as a deferred follow-up. ephemerd's own CI never uses `container:` on any platform. So: drop the worked example, split the image section into per-OS subsections, and rewrite the Known Limitations entry to state exactly what is and is not known rather than asserting a verdict in either direction. Also scope the "same image runs on every host" line to Linux images, which is what it always meant. --- README.md | 35 ++++++++++++++++++++++++----------- 1 file changed, 24 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index 57efe7a..63de0c4 100644 --- a/README.md +++ b/README.md @@ -234,7 +234,7 @@ runs-on: [self-hosted, linux, x64] ## Choosing the Image -### Linux and Windows jobs (OCI containers) +### Linux jobs (OCI containers) Use the standard `container:` key in your workflow. ephemerd's containerd pulls the image and runs the job inside it: @@ -247,16 +247,20 @@ jobs: steps: - uses: actions/checkout@v4 - run: make build - - build-windows: - runs-on: [self-hosted, windows, x64] - container: - image: ghcr.io/myorg/windows-build:latest - steps: - - uses: actions/checkout@v4 - - run: nmake ``` +### Windows jobs (OCI containers) + +Windows jobs also run in OCI containers (Hyper-V isolated). The image is +resolved the same way for every OS — workflow `container.image`, then the +per-repo `[runner.images]` override, then the per-OS default, then a +host-matched `mcr.microsoft.com/windows/servercore:ltsc20XX` fallback +(`pkg/scheduler/scheduler.go`, `pkg/runtime/image_windows.go`). + +The configured default is the well-trodden path on Windows. Setting +`container:` on a Windows job is **not** — see +[Known Limitations](#known-limitations) before you reach for it. + ### macOS jobs (VMs) macOS jobs run in ephemeral VMs, not containers. GitHub-hosted macOS runners ignore the `container:` key, but on ephemerd it has a specific meaning: the OCI image whose layers are extracted onto the running VM via virtio-fs. Use it to deliver pre-built SDKs, toolchains, or release artifacts alongside the job: @@ -578,11 +582,18 @@ docker build -t ghcr.io/your-org/ephemerd-build:latest . docker push ghcr.io/your-org/ephemerd-build:latest ``` -The same image runs on every host — Linux directly, Windows via Hyper-V Linux VM, macOS via Virtualization.framework Linux VM. +The same *Linux* image runs on every host — Linux directly, Windows via Hyper-V Linux VM, macOS via Virtualization.framework Linux VM. Windows-native jobs need a separate Windows-base image; see [docs/guides/runner-images.md](docs/guides/runner-images.md). ## Known Limitations -**Windows `services:` / `container:` YAML keys** — GitHub's runner binary blocks these on Windows. Use `docker run` in job steps instead: +**Windows `services:` / `container:` YAML keys — unverified, treat as unsupported.** These are not refused, and ephemerd has no code path that rejects them on Windows; they are simply not known to work end to end. What is actually known: + +- ephemerd *does* honour `container.image` on Windows for picking the runner container's image — image resolution is OS-agnostic (`pkg/scheduler/scheduler.go` `resolveImage`, `pkg/github/client.go` `FetchJobImage`). +- The GitHub runner binary then does its own thing with `container:` / `services:`: it shells out to a Docker CLI and asks for a sibling container (`docker pull`, `docker create` with a long `-v` list, `docker exec` per step). It does not refuse on Windows — an observed Windows `container:` job got as far as looking for `docker.exe`. +- The auto-detected Windows default image (`mcr.microsoft.com/windows/servercore:ltsc20XX`) ships **no Docker CLI**, so that lookup fails there. `images/runner-ci-windows/Dockerfile` installs one; a custom image would have to do the same. +- Even with a Docker CLI present, the piece that makes sibling containers work — dind bind-mount translation — is written for a Linux runner container on overlayfs and takes POSIX source paths (`pkg/dind/bindtranslate.go`, `pkg/dind/containers.go` `buildBindMounts`). Windows-native `container:` is an explicit deferred follow-up in [docs/arch/dind-bind-translation.md](docs/arch/dind-bind-translation.md) ("needs its own translation layer or a clean 'not supported' rejection at request time"), and nothing in ephemerd's own CI exercises `container:` on any platform. + +So: no verdict either way. If you need sidecars on Windows today, drive them from job steps, which does not depend on any of the above: ```yaml - run: docker run -d --name mysql -p 3306:3306 mysql:8 @@ -590,6 +601,8 @@ The same image runs on every host — Linux directly, Windows via Hyper-V Linux - run: docker stop mysql ``` +(This still needs a Docker CLI in the job image and `dind.enabled = true`.) + **macOS builds require macOS** — the darwin binary uses Virtualization.framework (CGO + Apple SDK). Cross-compilation from Linux isn't possible. Build on a Mac or use GitHub's macOS hosted runners for the darwin release. **Docker-in-Docker (fake daemon)** — ephemerd mounts a fake Docker Engine API socket at `/var/run/docker.sock` inside each job container. `docker pull`, `docker run`, `docker build`, and `docker push` all work — the fake daemon translates Docker API calls into containerd operations on the host. No real Docker daemon runs, no privileged containers, no `CAP_SYS_ADMIN`. Sidecars created via `docker run` are sibling containers on the same network. Enable with `dind.enabled = true` in config. See [docs/architecture/fake-docker-daemon.md](docs/architecture/fake-docker-daemon.md) for the full design. From 43f382795fe79f16893dc270f1cf165514ec0243 Mon Sep 17 00:00:00 2001 From: Luther Monson Date: Mon, 17 Aug 2026 16:58:11 -0700 Subject: [PATCH 2/2] docs(readme): make the Windows container: entry merge-order-safe and sharpen it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two changes to the Known Limitations entry. Merge-order hazard. The entry asserted "ephemerd has no code path that rejects them on Windows". True on main today, false the moment PR #168 lands its 501 gate for Windows sibling-container creation. Rewritten to describe the outcome (the job fails; where it fails depends on image and version) rather than the absence of a code path, so it reads correctly whichever of the two merges second. The deep-failure bullet now says the symptom is "either an explicit not-implemented from the daemon or a raw snapshotter/runtime error out of containerd", which covers both worlds. Sharpened verdict. The previous closing said "no verdict either way", which was right when written but undersold the evidence now confirmed against main: handleContainerCreate pins linux/, the overlayfs snapshotter and io.containerd.runc.v2 unconditionally (pkg/dind/containers.go), and Windows containerd only offers the windows/windows-lcow snapshotters (pkg/dind/cleanup.go). A Docker CLI genuinely only moves the failure deeper, so the entry now says "not supported" and explains why. That same pinning also condemns the `docker run` workaround this section had recommended since forever — container creation through the fake daemon is Linux-only, so the workaround fails on a Windows-native job for the same reason the thing it worked around does. Removed it rather than keep pointing people at it. `docker build`/`docker push` are unaffected (BuildKit path, exercised on Windows by build-images.yml), and Linux jobs are unaffected everywhere, so both are called out. Also scoped the adjacent dind entry, which claimed `docker run` works with no OS qualification and described the socket transport as if /var/run/docker.sock were universal, and scoped the "One Image, Every Host" line at :91 to Linux images (same defect already fixed at :585). --- README.md | 32 ++++++++++++++------------------ 1 file changed, 14 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 63de0c4..d52553d 100644 --- a/README.md +++ b/README.md @@ -90,7 +90,7 @@ graph TB ### One Image, Every Host -OCI container images work everywhere. The same Dockerfile builds an image that runs on Linux directly, inside a Hyper-V Linux VM on Windows, and inside a Virtualization.framework Linux VM on macOS. +Linux OCI images work everywhere. The same Dockerfile builds an image that runs on Linux directly, inside a Hyper-V Linux VM on Windows, and inside a Virtualization.framework Linux VM on macOS. (Windows-native jobs run Windows containers and need their own Windows-base image.) ```mermaid graph LR @@ -257,9 +257,10 @@ per-repo `[runner.images]` override, then the per-OS default, then a host-matched `mcr.microsoft.com/windows/servercore:ltsc20XX` fallback (`pkg/scheduler/scheduler.go`, `pkg/runtime/image_windows.go`). -The configured default is the well-trodden path on Windows. Setting -`container:` on a Windows job is **not** — see -[Known Limitations](#known-limitations) before you reach for it. +Set the image through config, not through `container:`. Setting `container:` +on a Windows job also drags in the runner's own sibling-container handling, +which does not work on Windows — see +[Known Limitations](#known-limitations). ### macOS jobs (VMs) @@ -586,26 +587,21 @@ The same *Linux* image runs on every host — Linux directly, Windows via Hyper- ## Known Limitations -**Windows `services:` / `container:` YAML keys — unverified, treat as unsupported.** These are not refused, and ephemerd has no code path that rejects them on Windows; they are simply not known to work end to end. What is actually known: +**Windows `services:` / `container:` YAML keys — not supported.** A Windows-native job that sets either key fails. *Where* it fails depends on your image and on your ephemerd version; *that* it fails is structural. The reason is not the one this section used to give: GitHub's runner binary does not block container operations on Windows. It tries, and the attempt fails further down. What actually happens: -- ephemerd *does* honour `container.image` on Windows for picking the runner container's image — image resolution is OS-agnostic (`pkg/scheduler/scheduler.go` `resolveImage`, `pkg/github/client.go` `FetchJobImage`). -- The GitHub runner binary then does its own thing with `container:` / `services:`: it shells out to a Docker CLI and asks for a sibling container (`docker pull`, `docker create` with a long `-v` list, `docker exec` per step). It does not refuse on Windows — an observed Windows `container:` job got as far as looking for `docker.exe`. -- The auto-detected Windows default image (`mcr.microsoft.com/windows/servercore:ltsc20XX`) ships **no Docker CLI**, so that lookup fails there. `images/runner-ci-windows/Dockerfile` installs one; a custom image would have to do the same. -- Even with a Docker CLI present, the piece that makes sibling containers work — dind bind-mount translation — is written for a Linux runner container on overlayfs and takes POSIX source paths (`pkg/dind/bindtranslate.go`, `pkg/dind/containers.go` `buildBindMounts`). Windows-native `container:` is an explicit deferred follow-up in [docs/arch/dind-bind-translation.md](docs/arch/dind-bind-translation.md) ("needs its own translation layer or a clean 'not supported' rejection at request time"), and nothing in ephemerd's own CI exercises `container:` on any platform. +- ephemerd *does* honour `container.image` on Windows for picking the runner container's image — image resolution is OS-agnostic (`pkg/scheduler/scheduler.go` `resolveImage`, `pkg/github/client.go` `FetchJobImage`). That part works, and is the only part that does. +- The runner binary then handles `container:` / `services:` itself: it shells out to a Docker CLI and asks for a *sibling* container (`docker pull`, `docker create` with a long `-v` list, `docker exec` per step). It does not refuse on Windows — an observed Windows `container:` job got as far as looking for `docker.exe`. +- The auto-detected Windows default image (`mcr.microsoft.com/windows/servercore:ltsc20XX`, `pkg/runtime/image_windows.go`) ships **no Docker CLI**, so on that image the failure is a missing `docker.exe`. `images/runner-ci-windows/Dockerfile` installs one; a custom image would have to do the same. +- Supplying a Docker CLI only moves the failure deeper. The fake daemon's container-create path is built for Linux and only Linux: it asks containerd for a `linux/` platform spec, the `overlayfs` snapshotter and the `io.containerd.runc.v2` runtime (`pkg/dind/containers.go` `handleContainerCreate`). A Windows containerd offers none of those — its snapshotters are `windows` / `windows-lcow` (`pkg/dind/cleanup.go`). The bind-mount translation behind it is likewise written for a Linux runner container on overlayfs with POSIX source paths (`pkg/dind/bindtranslate.go`). Depending on the version you run, this surfaces either as an explicit "not implemented" from the daemon or as a raw snapshotter/runtime error out of containerd. +- Windows-native `container:` is an explicit deferred follow-up in [docs/arch/dind-bind-translation.md](docs/arch/dind-bind-translation.md) — "needs its own translation layer or a clean 'not supported' rejection at request time". Nothing in ephemerd's CI exercises `container:` on any platform, so the failure modes above are read off the code rather than off a red test. -So: no verdict either way. If you need sidecars on Windows today, drive them from job steps, which does not depend on any of the above: +The same limit applies to the `docker run` workaround this section used to recommend: on a Windows-native job it runs into exactly the same wall. Container *creation* through the fake daemon is Linux-only. `docker build` and `docker push` are not — those route to the embedded BuildKit solver and are exercised on Windows by this repo's own `build-images.yml`. -```yaml -- run: docker run -d --name mysql -p 3306:3306 mysql:8 -- run: run-tests.sh -- run: docker stop mysql -``` - -(This still needs a Docker CLI in the job image and `dind.enabled = true`.) +Linux jobs are unaffected on every host, including Windows hosts: those run inside the Hyper-V Linux VM, where a separate ephemerd process runs as Linux against a Linux containerd. `container:` and `docker run` work there. **macOS builds require macOS** — the darwin binary uses Virtualization.framework (CGO + Apple SDK). Cross-compilation from Linux isn't possible. Build on a Mac or use GitHub's macOS hosted runners for the darwin release. -**Docker-in-Docker (fake daemon)** — ephemerd mounts a fake Docker Engine API socket at `/var/run/docker.sock` inside each job container. `docker pull`, `docker run`, `docker build`, and `docker push` all work — the fake daemon translates Docker API calls into containerd operations on the host. No real Docker daemon runs, no privileged containers, no `CAP_SYS_ADMIN`. Sidecars created via `docker run` are sibling containers on the same network. Enable with `dind.enabled = true` in config. See [docs/architecture/fake-docker-daemon.md](docs/architecture/fake-docker-daemon.md) for the full design. +**Docker-in-Docker (fake daemon)** — ephemerd mounts a fake Docker Engine API socket at `/var/run/docker.sock` inside each Linux job container (VM-isolated and Windows-native jobs get the same API over `DOCKER_HOST=tcp://…` instead, since a bind-mounted socket cannot cross a kernel boundary). For Linux jobs `docker pull`, `docker run`, `docker build`, and `docker push` all work; on Windows-native jobs `docker run` does not, per the entry above. The fake daemon translates Docker API calls into containerd operations on the host. No real Docker daemon runs, no privileged containers, no `CAP_SYS_ADMIN`. Sidecars created via `docker run` are sibling containers on the same network. Enable with `dind.enabled = true` in config. See [docs/architecture/fake-docker-daemon.md](docs/architecture/fake-docker-daemon.md) for the full design. **ARM64 Windows** — ephemerd supports it at the infrastructure level, but PHP and most build toolchains don't ship ARM64 Windows binaries yet.