Sync runner-doctor catalog for ARC/DinD and host-gateway lessons - #8330
Merged
Conversation
Copilot
AI
changed the title
[WIP] Update Runner Doctor with 2 new lessons
Sync runner-doctor catalog for ARC/DinD and host-gateway lessons
Sep 8, 2026
lpcox
marked this pull request as ready for review
September 8, 2026 21:18
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
B34 and portable catalog entries, parity tests, and the regenerated workflow lock are missing.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Updates runner-doctor knowledge for ARC/DinD service connectivity and host-gateway detection.
Changes:
- Adds A25 and A25/B34 quick lookups.
- Updates workflow and portable-agent guidance.
File summaries
| File | Description |
|---|---|
.github/workflows/shared/self-hosted-failure-modes.md |
Adds A25 and lookup entries. |
.github/workflows/self-hosted-runner-doctor.md |
Adds workflow quick lookups. |
.github/agents/self-hosted-runner-doctor.md |
Updates portable quick lookups. |
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 5
- Review effort level: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| - `invalid CapDrop: capability not supported by your kernel or not available in the current environment` → A22 (host capability bounding set trimmed below AWF's hardcoded `cap_drop` list, e.g. Talos; fixed in github/gh-aw-firewall#7795) | ||
| - `error mounting "/dev/null" to .../home/.npmrc: create mountpoint ...: read-only file system` (or `.docker/config.json`, `.composer/auth.json`) on `arc-dind` with `--docker-host-path-prefix` set → A23 (a surviving prefixed `${workDir}-chroot-home:/host$HOME` mount is paired with `/dev/null:/host$HOME/<credential>:ro` overlays because prefixed sources were compared with unprefixed `workDir`/`effectiveHome`; fixed in github/gh-aw-firewall#7998) | ||
| - `error mounting "/dev/null" to .../.npmrc: create mountpoint ...: read-only file system` on `arc-dind` persisting even after upgrading past github/gh-aw-firewall#7998 (A23's fix), where the credential mountpoint is missing under a declared-`rw` home bind backed by a genuinely read-only directory → A24 (`pruneUnmountableCredentialOverlays` only checked declared bind mode, never real filesystem writability for `rw`-declared covers; fixed in github/gh-aw-firewall#8086) | ||
| - `docker network connect --alias <name> awf-net <service_container>` is needed for raw-protocol GitHub Actions `services:` containers under `runner.topology: arc-dind` → A25 (service container must join `awf-net` for direct protocol access while the agent stays isolated; documented in github/gh-aw-firewall#8085) |
| - A setup-action-selected toolchain version is shadowed by the system-default version inside the AWF agent under `sandbox.agent.runtime: docker-sudo-iptables` → B31 (sudoers `secure_path` previously stripped the caller's `$GITHUB_PATH`-augmented `PATH`; fixed in gh-aw#58625, while AWF's `recoverHostPaths()` remains defense in depth) | ||
| - Recurring intermittent `403`/DNS `SERVFAIL` blocking an allowlisted domain (for example `api.github.com`) across otherwise-healthy runs, with no real forbidden-domain escape → B32 (Squid's default 60-second `negative_dns_ttl` replayed a transient upstream `SERVFAIL`; fixed in github/gh-aw-firewall#8171 by setting `negative_dns_ttl 1 seconds`, `dns_retransmit_interval 1 seconds`, and `dns_timeout 10 seconds`) | ||
| - `[DEBUG] Could not check Squid logs: EACCES ... access.log` mid-run, or `[DEBUG] Could not preserve squid logs: chmod ... Operation not permitted` during artifact preservation, even though logs are intact → B33 (the previous shutdown-time repair only changed mode bits and ran after diagnostics; fixed in github/gh-aw-firewall#8251 with a reusable pre-diagnostic `chown`+`chmod` repair) | ||
| - `host.docker.internal` or `(host.docker.internal/redacted)` appears in `network.allowDomains` but the host service still cannot be reached from inside AWF → B34 (host-gateway keyword detection was missing; fixed in github/gh-aw-firewall#8172) |
| - `invalid CapDrop: capability not supported by your kernel or not available in the current environment` → A22 (host capability bounding set trimmed below AWF's hardcoded `cap_drop` list, e.g. Talos; fixed in github/gh-aw-firewall#7795) | ||
| - `error mounting "/dev/null" to .../home/.npmrc: create mountpoint ...: read-only file system` (or `.docker/config.json`, `.composer/auth.json`) on `arc-dind` with `--docker-host-path-prefix` set → A23 (a surviving prefixed `${workDir}-chroot-home:/host$HOME` mount is paired with `/dev/null:/host$HOME/<credential>:ro` overlays because prefixed sources were compared with unprefixed `workDir`/`effectiveHome`; fixed in github/gh-aw-firewall#7998) | ||
| - `error mounting "/dev/null" to .../.npmrc: create mountpoint ...: read-only file system` on `arc-dind` persisting even after upgrading past github/gh-aw-firewall#7998 (A23's fix), where the credential mountpoint is missing under a declared-`rw` home bind backed by a genuinely read-only directory → A24 (`pruneUnmountableCredentialOverlays` only checked declared bind mode, never real filesystem writability for `rw`-declared covers; fixed in github/gh-aw-firewall#8086) | ||
| - `docker network connect --alias <name> awf-net <service_container>` is needed for raw-protocol GitHub Actions `services:` containers under `runner.topology: arc-dind` → A25 (service container must join `awf-net` for direct protocol access while the agent stays isolated; documented in github/gh-aw-firewall#8085) |
| | `error mounting "/dev/null" to .../home/.npmrc: create mountpoint ...: read-only file system` (or `.docker/config.json`, `.composer/auth.json`) on `arc-dind` with `--docker-host-path-prefix` set | A23 (`filterAgentVolumesForSysroot()` compared prefixed mount sources against unprefixed `workDir`/`effectiveHome`, so the bogus chroot-home mount wasn't dropped; fixed in github/gh-aw-firewall#7998) | | ||
| | `error mounting "/dev/null" to .../.npmrc: create mountpoint ...: read-only file system` on `arc-dind` persisting even after upgrading past github/gh-aw-firewall#7998 (A23's fix), where the credential mountpoint is missing under a declared-`rw` home bind backed by a genuinely read-only directory | A24 (`pruneUnmountableCredentialOverlays` only checked declared bind mode, never real filesystem writability for `rw`-declared covers; fixed in github/gh-aw-firewall#8086) | | ||
| | `docker network connect --alias <name> awf-net <service_container>` is needed for raw-protocol GitHub Actions `services:` containers under `runner.topology: arc-dind` | A25 (service container must join `awf-net` for direct protocol access while the agent stays on the isolated `awf-net`; fixed in docs/workaround in github/gh-aw-firewall#8085) | | ||
| | `host.docker.internal` or `(host.docker.internal/redacted)` appears in `network.allowDomains` but the host service still cannot be reached from inside AWF | B34 (`host.docker.internal` keyword handling was missing; fixed in github/gh-aw-firewall#8172) | |
| | A22 | `arc-dind` topology fails to start with Docker rejecting the compose `cap_drop` list: `invalid CapDrop: capability not supported by your kernel or not available in the current environment: "CAP_SYS_MODULE"` (or similar) on hosts, such as Talos Linux, that trim capabilities from the container capability bounding set | `src/services/squid-service.ts` and `src/services/agent-service.ts` hardcoded `cap_drop` lists (`NET_RAW`, `SYS_ADMIN`, `SYS_PTRACE`, `SYS_MODULE`, `MKNOD`, `AUDIT_WRITE`, `SETFCAP` for Squid; a similar list for the agent) for both the Squid and agent containers with no filtering against what the host/daemon's capability bounding set actually supports, so hosts with a trimmed bounding set (for example Talos `gha-runner-scale-set` with `docker:29-dind`) can never satisfy Docker's compose validation | **Fixed in AWF (PR github/gh-aw-firewall#7795, merged 2026-08-28):** `cap_drop` is now filtered against the effective host capability bounding set (read daemon-side from `/proc/self/status` `CapBnd` via a privileged probe container) before writing `docker-compose.yml`; capabilities already absent from the bounding set are silently omitted from `cap_drop` (a safe no-op, since a capability that can't be granted can't be exploited either). Upgrade AWF to include github/gh-aw-firewall#7795. | `getHostCapabilityBoundingSet()` (`src/capability-filter.ts:78-89`) runs `docker run --rm --privileged --network=none alpine:latest cat /proc/self/status` against the daemon and decodes `CapBnd`; reproduce with `docker compose up` on a host missing `CAP_SYS_MODULE` from the bounding set — pre-fix this fails with `invalid CapDrop`, post-fix compose starts normally | github/gh-aw#56127, github/gh-aw-firewall#7788, github/gh-aw-firewall#7795 | | ||
| | A23 | On `runner.topology: arc-dind` with `--docker-host-path-prefix` set, `docker compose up` fails with `error mounting "/dev/null" to .../home/.npmrc: create mountpoint ...: read-only file system` (also seen for `.docker/config.json`, `.composer/auth.json`) | `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) is meant to drop the bogus AWF-owned chroot-home mount that the DinD daemon can't resolve, but it compared already-prefixed mount sources (from `buildAgentVolumes()`, which applies `--docker-host-path-prefix` as its final step) against the *raw, unprefixed* `config.workDir`/`effectiveHome`. Once a host-path prefix is set — always true on real ARC/DinD — the comparison silently stopped matching, so the bogus chroot-home mount survived filtering and Docker tried to create a `.npmrc` credential-hiding overlay mountpoint inside a path the daemon couldn't write to (EROFS). Distinct from A20 (which was about legitimate caller-supplied home mounts being dropped); A23 is the inverse case — the *bogus* mount not being dropped. | **Fixed in AWF (PR github/gh-aw-firewall#7998, merged 2026-09-02):** extracted `prefixHostPath()` in `src/services/host-path-prefix.ts` from `translateBindMountHostPath()` so bare paths can be prefixed consistently; `filterAgentVolumesForSysroot()` now prefixes `config.workDir`/`effectiveHome` before comparing against mount sources, restoring correct detection of daemon-invisible mounts. The existing safe fallback (`dropUnbackedHostHomeOverlays`, warn + skip masking) now engages correctly instead of silently failing; explicit writable `--mount` for the home root still preserves credential masking. Upgrade AWF to include github/gh-aw-firewall#7998. | Reproduce with `runnerTopology: 'arc-dind'` + `dockerHostPathPrefix` set (e.g. `/host`) and inspect generated compose for a surviving prefixed `${workDir}-chroot-home:/host$HOME` mount together with `/dev/null:/host$HOME/<credential>:ro` overlays; on unpatched AWF, `docker compose up` fails with the EROFS mounting error above | github/gh-aw#57468, github/gh-aw-firewall#7994, github/gh-aw-firewall#7998 | | ||
| | A24 | On `runner.topology: arc-dind`, `docker compose up -d --pull never` fails with `error mounting "/dev/null" to rootfs at ".../gh-aw/home/.npmrc": create mountpoint for .../.npmrc mount: ... openat .npmrc: read-only file system` — distinct from A23 in that it persists even after github/gh-aw-firewall#7998 (A23's fix), specifically when the credential mountpoint is missing under a declared-`rw` home bind whose real backing directory is genuinely read-only on the ARC/DinD-staged filesystem | `pruneUnmountableCredentialOverlays` (`src/services/agent-volumes/credential-hiding.ts`) decided whether a `/dev/null` credential mask could be mounted purely from the *declared* compose bind mode (`ro`/`rw`), never checking the real filesystem when the mode was `rw`. AWF's own home-directory mount is declared `rw`, but under `--docker-host-path-prefix` it can resolve to a directory that is genuinely read-only on the runner's staged filesystem; Docker doesn't remount that case read-only inside the container (unlike a declared `ro` bind), so it touches the real host path directly when creating a missing mountpoint and hits `EROFS`, crashing the agent container before it starts. | **Fixed in AWF (PR github/gh-aw-firewall#8086, merged 2026-09-04):** an overlay whose mountpoint already exists is always kept (mounting over an existing path succeeds regardless of declared mode or real writability); a missing mountpoint under a declared-`ro` bind is still always dropped; a missing mountpoint under a declared-`rw` bind is now probed against the real filesystem (walking up to the nearest existing ancestor, mirroring how Docker creates missing intermediate directories) and the overlay is dropped if that real directory isn't writable. Upgrade AWF to include github/gh-aw-firewall#8086. | Reproduce with an absent credential path (for example, `.npmrc`) beneath a `chmod`-based real read-only home directory under `runner.topology: arc-dind` + `--docker-host-path-prefix`; on unpatched AWF (even with github/gh-aw-firewall#7998 applied) `docker compose up` fails with the EROFS mounting error above; on patched AWF the missing overlay is skipped and the agent starts. Existing credential paths remain mountable. | github/gh-aw#57468, github/gh-aw-firewall#8076, github/gh-aw-firewall#8086 | | ||
| | A25 | On `runner.topology: arc-dind`, workloads inside the AWF sandbox need a GitHub Actions `services:` container's native protocol (DB drivers, migration tools, etc.) but cannot reach it — the `services:` container runs on the runner's own bridge network while the AWF agent runs on the isolated `awf-net`, and the two bridges are unrouted; existing host-iptables service-port routing doesn't help because ARC/DinD network isolation never programs host iptables rules | No AWF mechanism previously joined a `services:` container to `awf-net`; raw-protocol clients (e.g. `psql`) have no route from the sandbox to the service | **Documented in AWF (PR github/gh-aw-firewall#8085, merged 2026-09-04)**: new `docs/arc-dind.md` section "Joining `services:` containers to `awf-net` for direct protocol access" documents a verified workaround — a pre-step waits for `awf-net` to exist, then `docker network connect --alias <name> awf-net <service_container>` attaches the service container (never the agent) with a resolvable alias. Security invariant: only the service joins `awf-net`; joining the agent to the runner bridge would bypass the Squid egress firewall. Longer-term direction (`services.<name>.attach: true` compiler sugar) is not yet implemented. | Confirm the `services:` container, not the agent, is the one calling `docker network connect --alias <name> awf-net <container>`; verify the agent can resolve/reach `<name>` after the join; confirm the agent itself never appears attached to the runner's default bridge | github/gh-aw-firewall#8075, github/gh-aw-firewall#8085 | |
Collaborator
|
@copilot address the review feedback |
Contributor
Author
Addressed in commit |
Contributor
|
✅ Copilot review passed with no inline comments. @copilot Add the |
Contributor
✅ Coverage Check PassedOverall Coverage
📁 Per-file Coverage Changes (1 files)
Coverage comparison generated by |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
host.docker.internalhost-gateway detection under self-hosted runs.What changed
services:container routing gapawf-netviadocker network connect --alias <name> awf-net <service_container>host.docker.internalhost-gateway regression that blocked Ollama/host-side services even when allowlisting and host ports were configuredhost.docker.internaland its redacted forms as host-gateway triggersNotes