Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 27 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ graph TB

### One Image, Every Host

OCI container images work everywhere. The same Dockerfile builds an image that runs on Linux directly, inside a Hyper-V Linux VM on Windows, and inside a Virtualization.framework Linux VM on macOS.
Linux OCI images work everywhere. The same Dockerfile builds an image that runs on Linux directly, inside a Hyper-V Linux VM on Windows, and inside a Virtualization.framework Linux VM on macOS. (Windows-native jobs run Windows containers and need their own Windows-base image.)

```mermaid
graph LR
Expand Down Expand Up @@ -234,7 +234,7 @@ runs-on: [self-hosted, linux, x64]

## Choosing the Image

### Linux and Windows jobs (OCI containers)
### Linux jobs (OCI containers)

Use the standard `container:` key in your workflow. ephemerd's containerd pulls the image and runs the job inside it:

Expand All @@ -247,16 +247,21 @@ jobs:
steps:
- uses: actions/checkout@v4
- run: make build

build-windows:
runs-on: [self-hosted, windows, x64]
container:
image: ghcr.io/myorg/windows-build:latest
steps:
- uses: actions/checkout@v4
- run: nmake
```

### Windows jobs (OCI containers)

Windows jobs also run in OCI containers (Hyper-V isolated). The image is
resolved the same way for every OS — workflow `container.image`, then the
per-repo `[runner.images]` override, then the per-OS default, then a
host-matched `mcr.microsoft.com/windows/servercore:ltsc20XX` fallback
(`pkg/scheduler/scheduler.go`, `pkg/runtime/image_windows.go`).

Set the image through config, not through `container:`. Setting `container:`
on a Windows job also drags in the runner's own sibling-container handling,
which does not work on Windows — see
[Known Limitations](#known-limitations).

### macOS jobs (VMs)

macOS jobs run in ephemeral VMs, not containers. GitHub-hosted macOS runners ignore the `container:` key, but on ephemerd it has a specific meaning: the OCI image whose layers are extracted onto the running VM via virtio-fs. Use it to deliver pre-built SDKs, toolchains, or release artifacts alongside the job:
Expand Down Expand Up @@ -578,21 +583,25 @@ docker build -t ghcr.io/your-org/ephemerd-build:latest .
docker push ghcr.io/your-org/ephemerd-build:latest
```

The same image runs on every host — Linux directly, Windows via Hyper-V Linux VM, macOS via Virtualization.framework Linux VM.
The same *Linux* image runs on every host — Linux directly, Windows via Hyper-V Linux VM, macOS via Virtualization.framework Linux VM. Windows-native jobs need a separate Windows-base image; see [docs/guides/runner-images.md](docs/guides/runner-images.md).

## Known Limitations

**Windows `services:` / `container:` YAML keys** GitHub's runner binary blocks these on Windows. Use `docker run` in job steps instead:
**Windows `services:` / `container:` YAML keys — not supported.** A Windows-native job that sets either key fails. *Where* it fails depends on your image and on your ephemerd version; *that* it fails is structural. The reason is not the one this section used to give: GitHub's runner binary does not block container operations on Windows. It tries, and the attempt fails further down. What actually happens:

```yaml
- run: docker run -d --name mysql -p 3306:3306 mysql:8
- run: run-tests.sh
- run: docker stop mysql
```
- ephemerd *does* honour `container.image` on Windows for picking the runner container's image — image resolution is OS-agnostic (`pkg/scheduler/scheduler.go` `resolveImage`, `pkg/github/client.go` `FetchJobImage`). That part works, and is the only part that does.
- The runner binary then handles `container:` / `services:` itself: it shells out to a Docker CLI and asks for a *sibling* container (`docker pull`, `docker create` with a long `-v` list, `docker exec` per step). It does not refuse on Windows — an observed Windows `container:` job got as far as looking for `docker.exe`.
- The auto-detected Windows default image (`mcr.microsoft.com/windows/servercore:ltsc20XX`, `pkg/runtime/image_windows.go`) ships **no Docker CLI**, so on that image the failure is a missing `docker.exe`. `images/runner-ci-windows/Dockerfile` installs one; a custom image would have to do the same.
- Supplying a Docker CLI only moves the failure deeper. The fake daemon's container-create path is built for Linux and only Linux: it asks containerd for a `linux/<arch>` platform spec, the `overlayfs` snapshotter and the `io.containerd.runc.v2` runtime (`pkg/dind/containers.go` `handleContainerCreate`). A Windows containerd offers none of those — its snapshotters are `windows` / `windows-lcow` (`pkg/dind/cleanup.go`). The bind-mount translation behind it is likewise written for a Linux runner container on overlayfs with POSIX source paths (`pkg/dind/bindtranslate.go`). Depending on the version you run, this surfaces either as an explicit "not implemented" from the daemon or as a raw snapshotter/runtime error out of containerd.
- Windows-native `container:` is an explicit deferred follow-up in [docs/arch/dind-bind-translation.md](docs/arch/dind-bind-translation.md) — "needs its own translation layer or a clean 'not supported' rejection at request time". Nothing in ephemerd's CI exercises `container:` on any platform, so the failure modes above are read off the code rather than off a red test.

The same limit applies to the `docker run` workaround this section used to recommend: on a Windows-native job it runs into exactly the same wall. Container *creation* through the fake daemon is Linux-only. `docker build` and `docker push` are not — those route to the embedded BuildKit solver and are exercised on Windows by this repo's own `build-images.yml`.

Linux jobs are unaffected on every host, including Windows hosts: those run inside the Hyper-V Linux VM, where a separate ephemerd process runs as Linux against a Linux containerd. `container:` and `docker run` work there.

**macOS builds require macOS** — the darwin binary uses Virtualization.framework (CGO + Apple SDK). Cross-compilation from Linux isn't possible. Build on a Mac or use GitHub's macOS hosted runners for the darwin release.

**Docker-in-Docker (fake daemon)** — ephemerd mounts a fake Docker Engine API socket at `/var/run/docker.sock` inside each job container. `docker pull`, `docker run`, `docker build`, and `docker push` all workthe fake daemon translates Docker API calls into containerd operations on the host. No real Docker daemon runs, no privileged containers, no `CAP_SYS_ADMIN`. Sidecars created via `docker run` are sibling containers on the same network. Enable with `dind.enabled = true` in config. See [docs/architecture/fake-docker-daemon.md](docs/architecture/fake-docker-daemon.md) for the full design.
**Docker-in-Docker (fake daemon)** — ephemerd mounts a fake Docker Engine API socket at `/var/run/docker.sock` inside each Linux job container (VM-isolated and Windows-native jobs get the same API over `DOCKER_HOST=tcp://…` instead, since a bind-mounted socket cannot cross a kernel boundary). For Linux jobs `docker pull`, `docker run`, `docker build`, and `docker push` all work; on Windows-native jobs `docker run` does not, per the entry above. The fake daemon translates Docker API calls into containerd operations on the host. No real Docker daemon runs, no privileged containers, no `CAP_SYS_ADMIN`. Sidecars created via `docker run` are sibling containers on the same network. Enable with `dind.enabled = true` in config. See [docs/architecture/fake-docker-daemon.md](docs/architecture/fake-docker-daemon.md) for the full design.

**ARM64 Windows** — ephemerd supports it at the infrastructure level, but PHP and most build toolchains don't ship ARM64 Windows binaries yet.

Expand Down