From dbcdc371eec4d0f759e694b6fc808db3777be1ba Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 8 Sep 2026 17:44:16 +0000 Subject: [PATCH 1/3] Initial plan From 286d1486aa3b043381f8a931c83905a107146071 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 8 Sep 2026 17:48:09 +0000 Subject: [PATCH 2/3] docs: sync runner doctor catalog entries for A25 and B34 --- .github/agents/self-hosted-runner-doctor.md | 3 +++ .github/workflows/self-hosted-runner-doctor.md | 2 ++ .github/workflows/shared/self-hosted-failure-modes.md | 3 +++ 3 files changed, 8 insertions(+) diff --git a/.github/agents/self-hosted-runner-doctor.md b/.github/agents/self-hosted-runner-doctor.md index 84227e41d..8a1f2b884 100644 --- a/.github/agents/self-hosted-runner-doctor.md +++ b/.github/agents/self-hosted-runner-doctor.md @@ -116,6 +116,8 @@ Prefer the narrowest match. Examples: - `mkdirat ... : read-only file system` at agent container startup while a `filesystem.allowWrite` policy is active (not the `chroot.binariesSourcePath`-specific A12 case) → A21; `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` followed by `Token protection will be disabled` → A21 (silent security-feature degradation under `filesystem.allowWrite`, fixed in github/gh-aw-firewall#7679) - `invalid CapDrop: capability not supported by your kernel or not available in the current environment` → A22 (host capability bounding set trimmed below AWF's hardcoded `cap_drop` list, e.g. Talos; fixed in github/gh-aw-firewall#7795) - `error mounting "/dev/null" to .../home/.npmrc: create mountpoint ...: read-only file system` (or `.docker/config.json`, `.composer/auth.json`) on `arc-dind` with `--docker-host-path-prefix` set → A23 (a surviving prefixed `${workDir}-chroot-home:/host$HOME` mount is paired with `/dev/null:/host$HOME/:ro` overlays because prefixed sources were compared with unprefixed `workDir`/`effectiveHome`; fixed in github/gh-aw-firewall#7998) +- `error mounting "/dev/null" to .../.npmrc: create mountpoint ...: read-only file system` on `arc-dind` persisting even after upgrading past github/gh-aw-firewall#7998 (A23's fix), where the credential mountpoint is missing under a declared-`rw` home bind backed by a genuinely read-only directory → A24 (`pruneUnmountableCredentialOverlays` only checked declared bind mode, never real filesystem writability for `rw`-declared covers; fixed in github/gh-aw-firewall#8086) +- `docker network connect --alias awf-net ` is needed for raw-protocol GitHub Actions `services:` containers under `runner.topology: arc-dind` → A25 (service container must join `awf-net` for direct protocol access while the agent stays isolated; documented in github/gh-aw-firewall#8085) - `a network with name awf-net exists but was not created for project` → B27 (orphaned fixed-name `awf-net` from a prior run on a persistent self-hosted runner; fixed in github/gh-aw-firewall#7817) - TLS/certificate verification failure from api-proxy against a custom `--openai-api-target`/`--anthropic-api-target` internal endpoint using a private/corporate CA → B28 (api-proxy sidecar had no custom CA trust extension point; fixed in github/gh-aw-firewall#7816 with `apiProxy.caCert`/`--api-proxy-ca-cert`) - `context-rebuild circuit breaker tripped` together with a failed `cd` into the expected workspace path → B29 (container-workdir not bind-mounted into the chroot; fixed in github/gh-aw-firewall#8021) @@ -123,6 +125,7 @@ Prefer the narrowest match. Examples: - A setup-action-selected toolchain version is shadowed by the system-default version inside the AWF agent under `sandbox.agent.runtime: docker-sudo-iptables` → B31 (sudoers `secure_path` previously stripped the caller's `$GITHUB_PATH`-augmented `PATH`; fixed in gh-aw#58625, while AWF's `recoverHostPaths()` remains defense in depth) - Recurring intermittent `403`/DNS `SERVFAIL` blocking an allowlisted domain (for example `api.github.com`) across otherwise-healthy runs, with no real forbidden-domain escape → B32 (Squid's default 60-second `negative_dns_ttl` replayed a transient upstream `SERVFAIL`; fixed in github/gh-aw-firewall#8171 by setting `negative_dns_ttl 1 seconds`, `dns_retransmit_interval 1 seconds`, and `dns_timeout 10 seconds`) - `[DEBUG] Could not check Squid logs: EACCES ... access.log` mid-run, or `[DEBUG] Could not preserve squid logs: chmod ... Operation not permitted` during artifact preservation, even though logs are intact → B33 (the previous shutdown-time repair only changed mode bits and ran after diagnostics; fixed in github/gh-aw-firewall#8251 with a reusable pre-diagnostic `chown`+`chmod` repair) +- `host.docker.internal` or `(host.docker.internal/redacted)` appears in `network.allowDomains` but the host service still cannot be reached from inside AWF → B34 (host-gateway keyword detection was missing; fixed in github/gh-aw-firewall#8172) - Copilot calls on Business/Enterprise/GHEC use the wrong Authorization scheme specifically for a fine-grained PAT (`github_pat_...`) → C10 (fixed in github/gh-aw-firewall#8038) ### 4. Check for known gaps and notable fixes diff --git a/.github/workflows/self-hosted-runner-doctor.md b/.github/workflows/self-hosted-runner-doctor.md index 2186cee84..21f5aab49 100644 --- a/.github/workflows/self-hosted-runner-doctor.md +++ b/.github/workflows/self-hosted-runner-doctor.md @@ -145,6 +145,7 @@ Prefer the narrowest match. Examples: - `invalid CapDrop: capability not supported by your kernel or not available in the current environment` → A22 (host capability bounding set trimmed below AWF's hardcoded `cap_drop` list, e.g. Talos; fixed in github/gh-aw-firewall#7795) - `error mounting "/dev/null" to .../home/.npmrc: create mountpoint ...: read-only file system` (or `.docker/config.json`, `.composer/auth.json`) on `arc-dind` with `--docker-host-path-prefix` set → A23 (a surviving prefixed `${workDir}-chroot-home:/host$HOME` mount is paired with `/dev/null:/host$HOME/:ro` overlays because prefixed sources were compared with unprefixed `workDir`/`effectiveHome`; fixed in github/gh-aw-firewall#7998) - `error mounting "/dev/null" to .../.npmrc: create mountpoint ...: read-only file system` on `arc-dind` persisting even after upgrading past github/gh-aw-firewall#7998 (A23's fix), where the credential mountpoint is missing under a declared-`rw` home bind backed by a genuinely read-only directory → A24 (`pruneUnmountableCredentialOverlays` only checked declared bind mode, never real filesystem writability for `rw`-declared covers; fixed in github/gh-aw-firewall#8086) +- `docker network connect --alias awf-net ` is needed for raw-protocol GitHub Actions `services:` containers under `runner.topology: arc-dind` → A25 (service container must join `awf-net` for direct protocol access while the agent stays isolated; documented in github/gh-aw-firewall#8085) - `a network with name awf-net exists but was not created for project` → B27 (orphaned fixed-name `awf-net` from a prior run on a persistent self-hosted runner; fixed in github/gh-aw-firewall#7817) - TLS/certificate verification failure from api-proxy against a custom `--openai-api-target`/`--anthropic-api-target` internal endpoint using a private/corporate CA → B28 (api-proxy sidecar had no custom CA trust extension point; fixed in github/gh-aw-firewall#7816 with `apiProxy.caCert`/`--api-proxy-ca-cert`) - `context-rebuild circuit breaker tripped` together with a failed `cd` into the expected workspace path → B29 (container-workdir not bind-mounted into the chroot; fixed in github/gh-aw-firewall#8021) @@ -152,6 +153,7 @@ Prefer the narrowest match. Examples: - A setup-action-selected toolchain version is shadowed by the system-default version inside the AWF agent under `sandbox.agent.runtime: docker-sudo-iptables` → B31 (sudoers `secure_path` previously stripped the caller's `$GITHUB_PATH`-augmented `PATH`; fixed in gh-aw#58625, while AWF's `recoverHostPaths()` remains defense in depth) - Recurring intermittent `403`/DNS `SERVFAIL` blocking an allowlisted domain (for example `api.github.com`) across otherwise-healthy runs, with no real forbidden-domain escape → B32 (Squid's default 60-second `negative_dns_ttl` replayed a transient upstream `SERVFAIL`; fixed in github/gh-aw-firewall#8171 by setting `negative_dns_ttl 1 seconds`, `dns_retransmit_interval 1 seconds`, and `dns_timeout 10 seconds`) - `[DEBUG] Could not check Squid logs: EACCES ... access.log` mid-run, or `[DEBUG] Could not preserve squid logs: chmod ... Operation not permitted` during artifact preservation, even though logs are intact → B33 (the previous shutdown-time repair only changed mode bits and ran after diagnostics; fixed in github/gh-aw-firewall#8251 with a reusable pre-diagnostic `chown`+`chmod` repair) +- `host.docker.internal` or `(host.docker.internal/redacted)` appears in `network.allowDomains` but the host service still cannot be reached from inside AWF → B34 (host-gateway keyword detection was missing; fixed in github/gh-aw-firewall#8172) - Copilot calls on Business/Enterprise/GHEC use the wrong Authorization scheme specifically for a fine-grained PAT (`github_pat_...`) → C10 (fixed in github/gh-aw-firewall#8038) - `SIGSEGV` / `SIGABRT` crash with Claude Code (Bun runtime) under `--container-runtime gvisor`; retries all fail → D7 (JSC JIT incompatible with gVisor W^X restrictions; AWF ≥ github/gh-aw-firewall#6276 auto-injects `BUN_JSC_useJIT=0`; for older AWF pass `--env BUN_JSC_useJIT=0`) diff --git a/.github/workflows/shared/self-hosted-failure-modes.md b/.github/workflows/shared/self-hosted-failure-modes.md index 105b55b00..24ceaea21 100644 --- a/.github/workflows/shared/self-hosted-failure-modes.md +++ b/.github/workflows/shared/self-hosted-failure-modes.md @@ -41,6 +41,7 @@ Establish these facts before matching a failure mode: | A22 | `arc-dind` topology fails to start with Docker rejecting the compose `cap_drop` list: `invalid CapDrop: capability not supported by your kernel or not available in the current environment: "CAP_SYS_MODULE"` (or similar) on hosts, such as Talos Linux, that trim capabilities from the container capability bounding set | `src/services/squid-service.ts` and `src/services/agent-service.ts` hardcoded `cap_drop` lists (`NET_RAW`, `SYS_ADMIN`, `SYS_PTRACE`, `SYS_MODULE`, `MKNOD`, `AUDIT_WRITE`, `SETFCAP` for Squid; a similar list for the agent) for both the Squid and agent containers with no filtering against what the host/daemon's capability bounding set actually supports, so hosts with a trimmed bounding set (for example Talos `gha-runner-scale-set` with `docker:29-dind`) can never satisfy Docker's compose validation | **Fixed in AWF (PR github/gh-aw-firewall#7795, merged 2026-08-28):** `cap_drop` is now filtered against the effective host capability bounding set (read daemon-side from `/proc/self/status` `CapBnd` via a privileged probe container) before writing `docker-compose.yml`; capabilities already absent from the bounding set are silently omitted from `cap_drop` (a safe no-op, since a capability that can't be granted can't be exploited either). Upgrade AWF to include github/gh-aw-firewall#7795. | `getHostCapabilityBoundingSet()` (`src/capability-filter.ts:78-89`) runs `docker run --rm --privileged --network=none alpine:latest cat /proc/self/status` against the daemon and decodes `CapBnd`; reproduce with `docker compose up` on a host missing `CAP_SYS_MODULE` from the bounding set — pre-fix this fails with `invalid CapDrop`, post-fix compose starts normally | github/gh-aw#56127, github/gh-aw-firewall#7788, github/gh-aw-firewall#7795 | | A23 | On `runner.topology: arc-dind` with `--docker-host-path-prefix` set, `docker compose up` fails with `error mounting "/dev/null" to .../home/.npmrc: create mountpoint ...: read-only file system` (also seen for `.docker/config.json`, `.composer/auth.json`) | `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) is meant to drop the bogus AWF-owned chroot-home mount that the DinD daemon can't resolve, but it compared already-prefixed mount sources (from `buildAgentVolumes()`, which applies `--docker-host-path-prefix` as its final step) against the *raw, unprefixed* `config.workDir`/`effectiveHome`. Once a host-path prefix is set — always true on real ARC/DinD — the comparison silently stopped matching, so the bogus chroot-home mount survived filtering and Docker tried to create a `.npmrc` credential-hiding overlay mountpoint inside a path the daemon couldn't write to (EROFS). Distinct from A20 (which was about legitimate caller-supplied home mounts being dropped); A23 is the inverse case — the *bogus* mount not being dropped. | **Fixed in AWF (PR github/gh-aw-firewall#7998, merged 2026-09-02):** extracted `prefixHostPath()` in `src/services/host-path-prefix.ts` from `translateBindMountHostPath()` so bare paths can be prefixed consistently; `filterAgentVolumesForSysroot()` now prefixes `config.workDir`/`effectiveHome` before comparing against mount sources, restoring correct detection of daemon-invisible mounts. The existing safe fallback (`dropUnbackedHostHomeOverlays`, warn + skip masking) now engages correctly instead of silently failing; explicit writable `--mount` for the home root still preserves credential masking. Upgrade AWF to include github/gh-aw-firewall#7998. | Reproduce with `runnerTopology: 'arc-dind'` + `dockerHostPathPrefix` set (e.g. `/host`) and inspect generated compose for a surviving prefixed `${workDir}-chroot-home:/host$HOME` mount together with `/dev/null:/host$HOME/:ro` overlays; on unpatched AWF, `docker compose up` fails with the EROFS mounting error above | github/gh-aw#57468, github/gh-aw-firewall#7994, github/gh-aw-firewall#7998 | | A24 | On `runner.topology: arc-dind`, `docker compose up -d --pull never` fails with `error mounting "/dev/null" to rootfs at ".../gh-aw/home/.npmrc": create mountpoint for .../.npmrc mount: ... openat .npmrc: read-only file system` — distinct from A23 in that it persists even after github/gh-aw-firewall#7998 (A23's fix), specifically when the credential mountpoint is missing under a declared-`rw` home bind whose real backing directory is genuinely read-only on the ARC/DinD-staged filesystem | `pruneUnmountableCredentialOverlays` (`src/services/agent-volumes/credential-hiding.ts`) decided whether a `/dev/null` credential mask could be mounted purely from the *declared* compose bind mode (`ro`/`rw`), never checking the real filesystem when the mode was `rw`. AWF's own home-directory mount is declared `rw`, but under `--docker-host-path-prefix` it can resolve to a directory that is genuinely read-only on the runner's staged filesystem; Docker doesn't remount that case read-only inside the container (unlike a declared `ro` bind), so it touches the real host path directly when creating a missing mountpoint and hits `EROFS`, crashing the agent container before it starts. | **Fixed in AWF (PR github/gh-aw-firewall#8086, merged 2026-09-04):** an overlay whose mountpoint already exists is always kept (mounting over an existing path succeeds regardless of declared mode or real writability); a missing mountpoint under a declared-`ro` bind is still always dropped; a missing mountpoint under a declared-`rw` bind is now probed against the real filesystem (walking up to the nearest existing ancestor, mirroring how Docker creates missing intermediate directories) and the overlay is dropped if that real directory isn't writable. Upgrade AWF to include github/gh-aw-firewall#8086. | Reproduce with an absent credential path (for example, `.npmrc`) beneath a `chmod`-based real read-only home directory under `runner.topology: arc-dind` + `--docker-host-path-prefix`; on unpatched AWF (even with github/gh-aw-firewall#7998 applied) `docker compose up` fails with the EROFS mounting error above; on patched AWF the missing overlay is skipped and the agent starts. Existing credential paths remain mountable. | github/gh-aw#57468, github/gh-aw-firewall#8076, github/gh-aw-firewall#8086 | +| A25 | On `runner.topology: arc-dind`, workloads inside the AWF sandbox need a GitHub Actions `services:` container's native protocol (DB drivers, migration tools, etc.) but cannot reach it — the `services:` container runs on the runner's own bridge network while the AWF agent runs on the isolated `awf-net`, and the two bridges are unrouted; existing host-iptables service-port routing doesn't help because ARC/DinD network isolation never programs host iptables rules | No AWF mechanism previously joined a `services:` container to `awf-net`; raw-protocol clients (e.g. `psql`) have no route from the sandbox to the service | **Documented in AWF (PR github/gh-aw-firewall#8085, merged 2026-09-04)**: new `docs/arc-dind.md` section "Joining `services:` containers to `awf-net` for direct protocol access" documents a verified workaround — a pre-step waits for `awf-net` to exist, then `docker network connect --alias awf-net ` attaches the service container (never the agent) with a resolvable alias. Security invariant: only the service joins `awf-net`; joining the agent to the runner bridge would bypass the Squid egress firewall. Longer-term direction (`services..attach: true` compiler sugar) is not yet implemented. | Confirm the `services:` container, not the agent, is the one calling `docker network connect --alias awf-net `; verify the agent can resolve/reach `` after the join; confirm the agent itself never appears attached to the runner's default bridge | github/gh-aw-firewall#8075, github/gh-aw-firewall#8085 | ## Category B — Self-hosted runners @@ -179,6 +180,8 @@ Establish these facts before matching a failure mode: | `invalid CapDrop: capability not supported by your kernel or not available in the current environment` | A22 (host capability bounding set trimmed below AWF's hardcoded `cap_drop` list, e.g. Talos; fixed in github/gh-aw-firewall#7795) | | `error mounting "/dev/null" to .../home/.npmrc: create mountpoint ...: read-only file system` (or `.docker/config.json`, `.composer/auth.json`) on `arc-dind` with `--docker-host-path-prefix` set | A23 (`filterAgentVolumesForSysroot()` compared prefixed mount sources against unprefixed `workDir`/`effectiveHome`, so the bogus chroot-home mount wasn't dropped; fixed in github/gh-aw-firewall#7998) | | `error mounting "/dev/null" to .../.npmrc: create mountpoint ...: read-only file system` on `arc-dind` persisting even after upgrading past github/gh-aw-firewall#7998 (A23's fix), where the credential mountpoint is missing under a declared-`rw` home bind backed by a genuinely read-only directory | A24 (`pruneUnmountableCredentialOverlays` only checked declared bind mode, never real filesystem writability for `rw`-declared covers; fixed in github/gh-aw-firewall#8086) | +| `docker network connect --alias awf-net ` is needed for raw-protocol GitHub Actions `services:` containers under `runner.topology: arc-dind` | A25 (service container must join `awf-net` for direct protocol access while the agent stays on the isolated `awf-net`; fixed in docs/workaround in github/gh-aw-firewall#8085) | +| `host.docker.internal` or `(host.docker.internal/redacted)` appears in `network.allowDomains` but the host service still cannot be reached from inside AWF | B34 (`host.docker.internal` keyword handling was missing; fixed in github/gh-aw-firewall#8172) | | `a network with name awf-net exists but was not created for project` | B27 (orphaned fixed-name `awf-net` from a prior run on a persistent self-hosted runner; fixed in github/gh-aw-firewall#7817) | | TLS/certificate verification failure from api-proxy against a custom `--openai-api-target`/`--anthropic-api-target` internal endpoint using a private/corporate CA | B28 (api-proxy sidecar had no custom CA trust extension point; fixed in github/gh-aw-firewall#7816 with `apiProxy.caCert`/`--api-proxy-ca-cert`) | | `spawn /usr/local/bin/copilot ENOENT` specifically on a tool-cache **hit** (`GITHUB_PATH` already set by the installer) | B23 (gh-aw's `activate_cached_copilot_bin()` skips the `/usr/local/bin/copilot` wrapper on cache hits while the compiler harness spawns that hardcoded path; AWF-side fixed via `ensure_usr_local_bin_shims()`/`prepare_usr_local_bin_overlay()` in github/gh-aw-firewall#7245; durable upstream fix still tracked in github/gh-aw-firewall#7130, open) | From 5ab98688070ae2caf21e4391712a830f901e6e58 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 8 Sep 2026 21:39:17 +0000 Subject: [PATCH 3/3] docs: complete runner doctor catalog parity --- .github/agents/self-hosted-runner-doctor.md | 2 ++ .../self-hosted-runner-doctor.lock.yml | 27 +++++++++---------- .../shared/self-hosted-failure-modes.md | 1 + ...self-hosted-runner-doctor-workflow.test.ts | 8 ++++++ 4 files changed, 24 insertions(+), 14 deletions(-) diff --git a/.github/agents/self-hosted-runner-doctor.md b/.github/agents/self-hosted-runner-doctor.md index 8a1f2b884..b0d4f7ce2 100644 --- a/.github/agents/self-hosted-runner-doctor.md +++ b/.github/agents/self-hosted-runner-doctor.md @@ -283,6 +283,7 @@ Establish these facts before matching a failure mode: | A22 | `arc-dind` topology fails to start with Docker rejecting the compose `cap_drop` list: `invalid CapDrop: capability not supported by your kernel or not available in the current environment: "CAP_SYS_MODULE"` (or similar) on hosts, such as Talos Linux, that trim capabilities from the container capability bounding set | `src/services/squid-service.ts` and `src/services/agent-service.ts` hardcoded `cap_drop` lists (`NET_RAW`, `SYS_ADMIN`, `SYS_PTRACE`, `SYS_MODULE`, `MKNOD`, `AUDIT_WRITE`, `SETFCAP` for Squid; a similar list for the agent) for both the Squid and agent containers with no filtering against what the host/daemon's capability bounding set actually supports, so hosts with a trimmed bounding set (for example Talos `gha-runner-scale-set` with `docker:29-dind`) can never satisfy Docker's compose validation | **Fixed in AWF (PR github/gh-aw-firewall#7795, merged 2026-08-28):** `cap_drop` is now filtered against the effective host capability bounding set (read daemon-side from `/proc/self/status` `CapBnd` via a privileged probe container) before writing `docker-compose.yml`; capabilities already absent from the bounding set are silently omitted from `cap_drop` (a safe no-op, since a capability that can't be granted can't be exploited either). Upgrade AWF to include github/gh-aw-firewall#7795. | `getHostCapabilityBoundingSet()` (`src/capability-filter.ts:78-89`) runs `docker run --rm --privileged --network=none alpine:latest cat /proc/self/status` against the daemon and decodes `CapBnd`; reproduce with `docker compose up` on a host missing `CAP_SYS_MODULE` from the bounding set — pre-fix this fails with `invalid CapDrop`, post-fix compose starts normally | github/gh-aw#56127, github/gh-aw-firewall#7788, github/gh-aw-firewall#7795 | | A23 | On `runner.topology: arc-dind` with `--docker-host-path-prefix` set, `docker compose up` fails with `error mounting "/dev/null" to .../home/.npmrc: create mountpoint ...: read-only file system` (also seen for `.docker/config.json`, `.composer/auth.json`) | `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) is meant to drop the bogus AWF-owned chroot-home mount that the DinD daemon can't resolve, but it compared already-prefixed mount sources (from `buildAgentVolumes()`, which applies `--docker-host-path-prefix` as its final step) against the *raw, unprefixed* `config.workDir`/`effectiveHome`. Once a host-path prefix is set — always true on real ARC/DinD — the comparison silently stopped matching, so the bogus chroot-home mount survived filtering and Docker tried to create a `.npmrc` credential-hiding overlay mountpoint inside a path the daemon couldn't write to (EROFS). Distinct from A20 (which was about legitimate caller-supplied home mounts being dropped); A23 is the inverse case — the *bogus* mount not being dropped. | **Fixed in AWF (PR github/gh-aw-firewall#7998, merged 2026-09-02):** extracted `prefixHostPath()` in `src/services/host-path-prefix.ts` from `translateBindMountHostPath()` so bare paths can be prefixed consistently; `filterAgentVolumesForSysroot()` now prefixes `config.workDir`/`effectiveHome` before comparing against mount sources, restoring correct detection of daemon-invisible mounts. The existing safe fallback (`dropUnbackedHostHomeOverlays`, warn + skip masking) now engages correctly instead of silently failing; explicit writable `--mount` for the home root still preserves credential masking. Upgrade AWF to include github/gh-aw-firewall#7998. | Reproduce with `runnerTopology: 'arc-dind'` + `dockerHostPathPrefix` set (e.g. `/host`) and inspect generated compose for a `.npmrc`/`.docker/config.json`/`.composer/auth.json` `/dev/null` overlay mountpoint targeting an unprefixed `effectiveHome`/`workDir` path; on unpatched AWF, `docker compose up` fails with the EROFS mounting error above | github/gh-aw#57468, github/gh-aw-firewall#7994, github/gh-aw-firewall#7998 | | A24 | On `runner.topology: arc-dind`, `docker compose up -d --pull never` fails with `error mounting "/dev/null" to rootfs at ".../gh-aw/home/.npmrc": create mountpoint for .../.npmrc mount: ... openat .npmrc: read-only file system` — distinct from A23 in that it persists even after github/gh-aw-firewall#7998 (A23's fix), specifically when the credential mountpoint is missing under a declared-`rw` home bind whose real backing directory is genuinely read-only on the ARC/DinD-staged filesystem | `pruneUnmountableCredentialOverlays` (`src/services/agent-volumes/credential-hiding.ts`) decided whether a `/dev/null` credential mask could be mounted purely from the *declared* compose bind mode (`ro`/`rw`), never checking the real filesystem when the mode was `rw`. AWF's own home-directory mount is declared `rw`, but under `--docker-host-path-prefix` it can resolve to a directory that is genuinely read-only on the runner's staged filesystem; Docker doesn't remount that case read-only inside the container (unlike a declared `ro` bind), so it touches the real host path directly when creating a missing mountpoint and hits `EROFS`, crashing the agent container before it starts. | **Fixed in AWF (PR github/gh-aw-firewall#8086, merged 2026-09-04):** an overlay whose mountpoint already exists is always kept (mounting over an existing path succeeds regardless of declared mode or real writability); a missing mountpoint under a declared-`ro` bind is still always dropped; a missing mountpoint under a declared-`rw` bind is now probed against the real filesystem (walking up to the nearest existing ancestor, mirroring how Docker creates missing intermediate directories) and the overlay is dropped if that real directory isn't writable. Upgrade AWF to include github/gh-aw-firewall#8086. | Reproduce with an absent credential path (for example, `.npmrc`) beneath a `chmod`-based real read-only home directory under `runner.topology: arc-dind` + `--docker-host-path-prefix`; on unpatched AWF (even with github/gh-aw-firewall#7998 applied) `docker compose up` fails with the EROFS mounting error above; on patched AWF the missing overlay is skipped and the agent starts. Existing credential paths remain mountable. | github/gh-aw#57468, github/gh-aw-firewall#8076, github/gh-aw-firewall#8086 | +| A25 | On `runner.topology: arc-dind`, workloads inside the AWF sandbox need a GitHub Actions `services:` container's native protocol (DB drivers, migration tools, etc.) but cannot reach it — the `services:` container runs on the runner's own bridge network while the AWF agent runs on the isolated `awf-net`, and the two bridges are unrouted; existing host-iptables service-port routing doesn't help because ARC/DinD network isolation never programs host iptables rules | No AWF mechanism previously joined a `services:` container to `awf-net`; raw-protocol clients (e.g. `psql`) have no route from the sandbox to the service | **Documented in AWF (PR github/gh-aw-firewall#8085, merged 2026-09-04)**: new `docs/arc-dind.md` section "Joining `services:` containers to `awf-net` for direct protocol access" documents a verified workaround — a pre-step waits for `awf-net` to exist, then `docker network connect --alias awf-net ` attaches the service container (never the agent) with a resolvable alias. Security invariant: only the service joins `awf-net`; joining the agent to the runner bridge would bypass the Squid egress firewall. Longer-term direction (`services..attach: true` compiler sugar) is not yet implemented. | Confirm the `services:` container, not the agent, is the one calling `docker network connect --alias awf-net `; verify the agent can resolve/reach `` after the join; confirm the agent itself never appears attached to the runner's default bridge | github/gh-aw-firewall#8075, github/gh-aw-firewall#8085 | ## Category B — Self-hosted runners @@ -322,6 +323,7 @@ Establish these facts before matching a failure mode: | B31 | Under `sandbox.agent.runtime: docker-sudo-iptables`, a toolchain version selected via a setup action (e.g. `ruby/setup-ruby` choosing Ruby 3.4.8) is shadowed by the system-installed version (e.g. `/usr/bin/ruby` 3.2.3) inside the AWF agent container, even though `--env-all`/`AWF_HOST_PATH` capture is active | `docker-sudo-iptables` invoked AWF via `sudo -E awf ...`; sudoers' `secure_path` could silently overwrite the runner's `$GITHUB_PATH`-augmented `PATH` before AWF observed `process.env.PATH`, losing hosted-toolcache bin-dir precedence | **Fixed in gh-aw (PR github/gh-aw#58625, merged 2026-09-05):** privileged AWF startup preserves the caller `PATH`. AWF's `readGitHubPathEntries()`/`recoverHostPaths()` recovery remains defense in depth, and merged PR github/gh-aw-firewall#8173 adds regression coverage in `src/services/agent-environment/host-path-recovery.test.ts`; no AWF production-code change was needed. | Confirm the setup-action hosted-toolcache bin dir remains ahead of `/usr/bin` in `AWF_HOST_PATH` under `docker-sudo-iptables`; if it does not, this is a regression. | github/gh-aw#58458, github/gh-aw#58625, github/gh-aw-firewall#8141, github/gh-aw-firewall#8173 | | B32 | A repeated/persistent-runner workflow intermittently blocks allowlisted `api.github.com`/`github.com` traffic with `403` or DNS `SERVFAIL`, recurring across otherwise-healthy runs | Squid's default `negative_dns_ttl` is 1 minute, so one transient upstream `SERVFAIL` is negatively cached and replayed for up to 60 seconds even after DNS recovers | **Fixed in AWF (PR github/gh-aw-firewall#8171, merged 2026-09-05):** `generateDnsSection()` emits `negative_dns_ttl 1 seconds`, `dns_retransmit_interval 1 seconds`, and `dns_timeout 10 seconds`. Upgrade AWF to include github/gh-aw-firewall#8171. | Inspect generated `squid.conf` for `negative_dns_ttl 1 seconds`, `dns_retransmit_interval 1 seconds`, `dns_timeout 10 seconds`, and `dns_nameservers`; correlate Squid `TCP_DENIED`/SERVFAIL bursts with concurrent startup or resolver load on unpatched AWF | github/gh-aw-firewall#8168, github/gh-aw-firewall#8171 | | B33 | `[DEBUG] Could not check Squid logs: EACCES ... access.log` during diagnostics, or `[DEBUG] Could not preserve squid logs: chmod ... Operation not permitted` during artifact preservation, although logs are intact | Squid writes logs as UID 13; the previous shutdown-time repair only changed mode bits, ran after mid-run diagnostics, and never transferred ownership to the runner before artifact preservation | **Fixed in AWF (PR github/gh-aw-firewall#8251, merged 2026-09-07):** reusable `fixSquidLogPermissions()` now `chown`s to the runner UID/GID via `docker exec -e` and `chmod`s; `runAgentCommand()` repairs permissions before `checkSquidLogs()`, and preserved-log `chmod` is reported separately after rename. Upgrade AWF to include github/gh-aw-firewall#8251. | Trigger a blocked-domain or upstream-error diagnostic and confirm `access.log` is readable; after the run, `ls -la ` should show runner-UID ownership and no `Could not check Squid logs`/`Could not preserve squid logs` messages | github/gh-aw-firewall#8249, github/gh-aw-firewall#8251 | +| B34 | `host.docker.internal` or `(host.docker.internal/redacted)` appears in `network.allowDomains`, but an Ollama or other host-side service remains unreachable from inside AWF even when its host port is allowlisted | Host-gateway trigger detection did not recognize the canonical `host.docker.internal` keyword or its redacted audit form, so AWF omitted the host-gateway mapping needed to route the request to the runner host | **Fixed in AWF (PR github/gh-aw-firewall#8172, merged 2026-09-05):** host-gateway detection recognizes `host.docker.internal` and `(host.docker.internal/redacted)` forms and emits the required host mapping. Upgrade AWF to include github/gh-aw-firewall#8172. | Inspect generated `docker-compose.yml` for the `host.docker.internal:host-gateway` mapping and verify `getent hosts host.docker.internal` plus a request to the allowlisted host service from inside the agent | github/gh-aw-firewall#8172 | ## Category C — GHES / GHEC / `ghe.com` diff --git a/.github/workflows/self-hosted-runner-doctor.lock.yml b/.github/workflows/self-hosted-runner-doctor.lock.yml index 82d502598..1154f81c4 100644 --- a/.github/workflows/self-hosted-runner-doctor.lock.yml +++ b/.github/workflows/self-hosted-runner-doctor.lock.yml @@ -1,6 +1,6 @@ -# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"be56d205ed7374bea2d8b19a93bf66a884b9079dc10be8d30dea7e85ae706fef","body_hash":"d332e854c02a7bece67314fc483ae2a2be006f957b4c63d3452988011c800b39","compiler_version":"v0.88.6","strict":true,"agent_id":"copilot","engine_versions":{"copilot":"1.0.80"}} -# gh-aw-manifest: {"version":1,"secrets":["GH_AW_DEFAULT_OTLP_HEADERS","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/cache","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/restore","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/save","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"d462c12b8f1f616b55fed699787b0280c594e592","version":"v0.88.6"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.28.14","digest":"sha256:f7df036c86575527b61f3f7df91c4412349a12b2a74988d929eafa2999230c98","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.28.14@sha256:f7df036c86575527b61f3f7df91c4412349a12b2a74988d929eafa2999230c98"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.14","digest":"sha256:6f95e2234dd9bd6333a8ff28ccea7ecf0204acd4a09108723844dbd2bf6268c5","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.14@sha256:6f95e2234dd9bd6333a8ff28ccea7ecf0204acd4a09108723844dbd2bf6268c5"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.28.14","digest":"sha256:2ce8df3abf3e9b76e9c0cf5863da41f1ab3f89b20ad14b988806ab89e7bf2cd5","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.28.14@sha256:2ce8df3abf3e9b76e9c0cf5863da41f1ab3f89b20ad14b988806ab89e7bf2cd5"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.18","digest":"sha256:85b940556a8faa4e1fdbef124bfd75f2c4ebd855a10b88a1c3b6f3e97f6f1a53","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.18@sha256:85b940556a8faa4e1fdbef124bfd75f2c4ebd855a10b88a1c3b6f3e97f6f1a53"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:0daa8971fa4732b647150cb6524a6b0804b68d5d24f6f58b5dd1af23bd63fb23","pinned_image":"ghcr.io/github/gh-aw-node@sha256:0daa8971fa4732b647150cb6524a6b0804b68d5d24f6f58b5dd1af23bd63fb23"},{"image":"ghcr.io/github/github-mcp-server:v1.11.0","digest":"sha256:fbec75de11c255213fa08d80fb166abe73d851fff631c51c0079872967720699","pinned_image":"ghcr.io/github/github-mcp-server:v1.11.0@sha256:fbec75de11c255213fa08d80fb166abe73d851fff631c51c0079872967720699"}],"mcp_servers":[{"name":"github","tools":["get_commit","get_file_contents","get_latest_release","get_me","get_pull_request","get_pull_request_comments","get_pull_request_diff","get_pull_request_files","get_pull_request_review_comments","get_pull_request_reviews","get_pull_request_status","get_release_by_tag","get_tag","issue_read","list_branches","list_commits","list_issue_types","list_issues","list_pull_requests","list_releases","list_starred_repositories","list_tags","pull_request_read","search_code","search_issues","search_pull_requests","search_repositories"]},{"name":"safeoutputs","tools":["add_comment","create_issue","missing_data","missing_tool","noop"]}]} -# This file was automatically generated by gh-aw (v0.88.6). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md +# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"be56d205ed7374bea2d8b19a93bf66a884b9079dc10be8d30dea7e85ae706fef","body_hash":"09884237d86a74a80aff88a2653eb0ca4da5b3cca3164d3337f4b9fc4819ea31","compiler_version":"v0.88.7","strict":true,"agent_id":"copilot","engine_versions":{"copilot":"1.0.80"}} +# gh-aw-manifest: {"version":1,"secrets":["GH_AW_DEFAULT_OTLP_HEADERS","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/cache","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/restore","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/cache/save","sha":"55cc8345863c7cc4c66a329aec7e433d2d1c52a9","version":"v6.1.0"},{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"v0.88.7","version":"v0.88.7"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.28.14","digest":"sha256:f7df036c86575527b61f3f7df91c4412349a12b2a74988d929eafa2999230c98","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.28.14@sha256:f7df036c86575527b61f3f7df91c4412349a12b2a74988d929eafa2999230c98"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.14","digest":"sha256:6f95e2234dd9bd6333a8ff28ccea7ecf0204acd4a09108723844dbd2bf6268c5","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.14@sha256:6f95e2234dd9bd6333a8ff28ccea7ecf0204acd4a09108723844dbd2bf6268c5"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.28.14","digest":"sha256:2ce8df3abf3e9b76e9c0cf5863da41f1ab3f89b20ad14b988806ab89e7bf2cd5","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.28.14@sha256:2ce8df3abf3e9b76e9c0cf5863da41f1ab3f89b20ad14b988806ab89e7bf2cd5"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.18","digest":"sha256:85b940556a8faa4e1fdbef124bfd75f2c4ebd855a10b88a1c3b6f3e97f6f1a53","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.18@sha256:85b940556a8faa4e1fdbef124bfd75f2c4ebd855a10b88a1c3b6f3e97f6f1a53"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:0daa8971fa4732b647150cb6524a6b0804b68d5d24f6f58b5dd1af23bd63fb23","pinned_image":"ghcr.io/github/gh-aw-node@sha256:0daa8971fa4732b647150cb6524a6b0804b68d5d24f6f58b5dd1af23bd63fb23"},{"image":"ghcr.io/github/github-mcp-server:v1.11.0","digest":"sha256:fbec75de11c255213fa08d80fb166abe73d851fff631c51c0079872967720699","pinned_image":"ghcr.io/github/github-mcp-server:v1.11.0@sha256:fbec75de11c255213fa08d80fb166abe73d851fff631c51c0079872967720699"}],"mcp_servers":[{"name":"github","tools":["get_commit","get_file_contents","get_latest_release","get_me","get_pull_request","get_pull_request_comments","get_pull_request_diff","get_pull_request_files","get_pull_request_review_comments","get_pull_request_reviews","get_pull_request_status","get_release_by_tag","get_tag","issue_read","list_branches","list_commits","list_issue_types","list_issues","list_pull_requests","list_releases","list_starred_repositories","list_tags","pull_request_read","search_code","search_issues","search_pull_requests","search_repositories"]},{"name":"safeoutputs","tools":["add_comment","create_issue","missing_data","missing_tool","noop"]}]} +# This file was automatically generated by gh-aw (v0.88.7). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md # # ___ _ _ # / _ \ | | (_) @@ -44,7 +44,7 @@ # - actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 # - actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 (source v9) # - actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 -# - github/gh-aw-actions/setup@d462c12b8f1f616b55fed699787b0280c594e592 # v0.88.6 +# - github/gh-aw-actions/setup@v0.88.7 # # Container images used: # - @@ -114,7 +114,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@d462c12b8f1f616b55fed699787b0280c594e592 # v0.88.6 + uses: github/gh-aw-actions/setup@v0.88.7 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -137,7 +137,7 @@ jobs: GH_AW_INFO_MODEL: ${{ vars.GH_AW_MODEL_AGENT_COPILOT || vars.GH_AW_DEFAULT_MODEL_COPILOT || 'auto' }} GH_AW_INFO_VERSION: "1.0.80" GH_AW_INFO_AGENT_VERSION: "1.0.80" - GH_AW_INFO_CLI_VERSION: "v0.88.6" + GH_AW_INFO_CLI_VERSION: "v0.88.7" GH_AW_INFO_WORKFLOW_NAME: "Self-Hosted Runner Doctor" GH_AW_INFO_EXPERIMENTAL: "false" GH_AW_INFO_SUPPORTS_TOOLS_ALLOWLIST: "true" @@ -257,7 +257,7 @@ jobs: - name: Check compile-agentic version uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 env: - GH_AW_COMPILED_VERSION: "v0.88.6" + GH_AW_COMPILED_VERSION: "v0.88.7" with: script: | const path = require('path'); @@ -486,7 +486,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@d462c12b8f1f616b55fed699787b0280c594e592 # v0.88.6 + uses: github/gh-aw-actions/setup@v0.88.7 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -588,7 +588,7 @@ jobs: run: bash "${RUNNER_TEMP}/gh-aw/actions/install_copilot_cli.sh" env: GH_HOST: github.com - GH_AW_COMPILED_VERSION: v0.88.6 + GH_AW_COMPILED_VERSION: v0.88.7 - name: Setup Node.js uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0 with: @@ -1060,7 +1060,7 @@ jobs: GH_AW_PROMPT: /tmp/gh-aw/aw-prompts/prompt.txt GH_AW_SAFE_OUTPUTS: ${{ steps.set-runtime-paths.outputs.GH_AW_SAFE_OUTPUTS }} GH_AW_TIMEOUT_MINUTES: 15 - GH_AW_VERSION: v0.88.6 + GH_AW_VERSION: v0.88.7 GITHUB_API_URL: ${{ github.api_url }} GITHUB_AW: true GITHUB_COPILOT_INTEGRATION_ID: agentic-workflows @@ -1299,7 +1299,6 @@ jobs: /tmp/gh-aw/agent_usage.json /tmp/gh-aw/agent-stdio.log /tmp/gh-aw/pre-agent-audit.txt - /tmp/gh-aw/agent/ /tmp/gh-aw/github_rate_limits.jsonl /tmp/gh-aw/otel.jsonl /tmp/gh-aw/otlp-export-errors.jsonl @@ -1339,7 +1338,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@d462c12b8f1f616b55fed699787b0280c594e592 # v0.88.6 + uses: github/gh-aw-actions/setup@v0.88.7 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -1605,7 +1604,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@d462c12b8f1f616b55fed699787b0280c594e592 # v0.88.6 + uses: github/gh-aw-actions/setup@v0.88.7 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} @@ -1674,7 +1673,7 @@ jobs: steps: - name: Setup Scripts id: setup - uses: github/gh-aw-actions/setup@d462c12b8f1f616b55fed699787b0280c594e592 # v0.88.6 + uses: github/gh-aw-actions/setup@v0.88.7 with: destination: ${{ runner.temp }}/gh-aw/actions job-name: ${{ github.job }} diff --git a/.github/workflows/shared/self-hosted-failure-modes.md b/.github/workflows/shared/self-hosted-failure-modes.md index 24ceaea21..af2973d9b 100644 --- a/.github/workflows/shared/self-hosted-failure-modes.md +++ b/.github/workflows/shared/self-hosted-failure-modes.md @@ -81,6 +81,7 @@ Establish these facts before matching a failure mode: | B31 | Under `sandbox.agent.runtime: docker-sudo-iptables`, a toolchain version selected via a setup action (e.g. `ruby/setup-ruby` choosing Ruby 3.4.8) is shadowed by the system-installed version (e.g. `/usr/bin/ruby` 3.2.3) inside the AWF agent container, even though `--env-all`/`AWF_HOST_PATH` capture is active | `docker-sudo-iptables` invoked AWF via `sudo -E awf ...`; sudoers' `secure_path` could silently overwrite the runner's `$GITHUB_PATH`-augmented `PATH` before AWF observed `process.env.PATH`, losing hosted-toolcache bin-dir precedence | **Fixed in gh-aw (PR github/gh-aw#58625, merged 2026-09-05):** privileged AWF startup preserves the caller `PATH`. AWF's `readGitHubPathEntries()`/`recoverHostPaths()` recovery remains defense in depth, and merged PR github/gh-aw-firewall#8173 adds regression coverage in `src/services/agent-environment/host-path-recovery.test.ts`; no AWF production-code change was needed. | Confirm the setup-action hosted-toolcache bin dir remains ahead of `/usr/bin` in `AWF_HOST_PATH` under `docker-sudo-iptables`; if it does not, this is a regression. | github/gh-aw#58458, github/gh-aw#58625, github/gh-aw-firewall#8141, github/gh-aw-firewall#8173 | | B32 | A repeated/persistent-runner workflow intermittently blocks allowlisted `api.github.com`/`github.com` traffic with `403` or DNS `SERVFAIL`, recurring across otherwise-healthy runs | Squid's default `negative_dns_ttl` is 1 minute, so one transient upstream `SERVFAIL` is negatively cached and replayed for up to 60 seconds even after DNS recovers | **Fixed in AWF (PR github/gh-aw-firewall#8171, merged 2026-09-05):** `generateDnsSection()` emits `negative_dns_ttl 1 seconds`, `dns_retransmit_interval 1 seconds`, and `dns_timeout 10 seconds`. Upgrade AWF to include github/gh-aw-firewall#8171. | Inspect generated `squid.conf` for `negative_dns_ttl 1 seconds`, `dns_retransmit_interval 1 seconds`, `dns_timeout 10 seconds`, and `dns_nameservers`; correlate Squid `TCP_DENIED`/SERVFAIL bursts with concurrent startup or resolver load on unpatched AWF | github/gh-aw-firewall#8168, github/gh-aw-firewall#8171 | | B33 | `[DEBUG] Could not check Squid logs: EACCES ... access.log` during diagnostics, or `[DEBUG] Could not preserve squid logs: chmod ... Operation not permitted` during artifact preservation, although logs are intact | Squid writes logs as UID 13; the previous shutdown-time repair only changed mode bits, ran after mid-run diagnostics, and never transferred ownership to the runner before artifact preservation | **Fixed in AWF (PR github/gh-aw-firewall#8251, merged 2026-09-07):** reusable `fixSquidLogPermissions()` now `chown`s to the runner UID/GID via `docker exec -e` and `chmod`s; `runAgentCommand()` repairs permissions before `checkSquidLogs()`, and preserved-log `chmod` is reported separately after rename. Upgrade AWF to include github/gh-aw-firewall#8251. | Trigger a blocked-domain or upstream-error diagnostic and confirm `access.log` is readable; after the run, `ls -la ` should show runner-UID ownership and no `Could not check Squid logs`/`Could not preserve squid logs` messages | github/gh-aw-firewall#8249, github/gh-aw-firewall#8251 | +| B34 | `host.docker.internal` or `(host.docker.internal/redacted)` appears in `network.allowDomains`, but an Ollama or other host-side service remains unreachable from inside AWF even when its host port is allowlisted | Host-gateway trigger detection did not recognize the canonical `host.docker.internal` keyword or its redacted audit form, so AWF omitted the host-gateway mapping needed to route the request to the runner host | **Fixed in AWF (PR github/gh-aw-firewall#8172, merged 2026-09-05):** host-gateway detection recognizes `host.docker.internal` and `(host.docker.internal/redacted)` forms and emits the required host mapping. Upgrade AWF to include github/gh-aw-firewall#8172. | Inspect generated `docker-compose.yml` for the `host.docker.internal:host-gateway` mapping and verify `getent hosts host.docker.internal` plus a request to the allowlisted host service from inside the agent | github/gh-aw-firewall#8172 | ## Category C — GHES / GHEC / `ghe.com` diff --git a/scripts/ci/self-hosted-runner-doctor-workflow.test.ts b/scripts/ci/self-hosted-runner-doctor-workflow.test.ts index 806cf4b0c..eccff4533 100644 --- a/scripts/ci/self-hosted-runner-doctor-workflow.test.ts +++ b/scripts/ci/self-hosted-runner-doctor-workflow.test.ts @@ -126,6 +126,8 @@ describe('self-hosted runner doctor workflow config', () => { expect(content).toContain('`pruneUnmountableCredentialOverlays` (`src/services/agent-volumes/credential-hiding.ts`)'); expect(content).toContain('github/gh-aw#57468, github/gh-aw-firewall#8076, github/gh-aw-firewall#8086'); expect(content).toContain('| `error mounting "/dev/null" to .../.npmrc: create mountpoint ...: read-only file system` on `arc-dind` persisting even after upgrading past github/gh-aw-firewall#7998 (A23\'s fix), where the credential mountpoint is missing under a declared-`rw` home bind backed by a genuinely read-only directory | A24'); + expect(content).toContain('| A25 | On `runner.topology: arc-dind`, workloads inside the AWF sandbox need a GitHub Actions `services:` container'); + expect(content).toContain('github/gh-aw-firewall#8075, github/gh-aw-firewall#8085'); // B23 update: PR #7245 fixes the AWF-side gap expect(content).toContain('**Fixed on the AWF side (PR github/gh-aw-firewall#7245, merged 2026-08-11):**'); expect(content).toContain('`ensure_usr_local_bin_shims()`'); @@ -193,6 +195,8 @@ describe('self-hosted runner doctor workflow config', () => { expect(content).toContain('github/gh-aw-firewall#8168, github/gh-aw-firewall#8171'); expect(content).toContain('| B33 | `[DEBUG] Could not check Squid logs: EACCES'); expect(content).toContain('github/gh-aw-firewall#8249, github/gh-aw-firewall#8251'); + expect(content).toContain('| B34 | `host.docker.internal` or `(host.docker.internal/redacted)` appears in `network.allowDomains`'); + expect(content).toContain('github/gh-aw-firewall#8172'); expect(content).toContain('| C9 | `400 bad request: Authorization header is badly formatted` specifically on the **derived GHEC data-residency Copilot target**'); expect(content).toContain('receives `token` instead of required `Bearer` prefix'); expect(content).toContain('| C10 | Fine-grained GitHub PATs (`github_pat_...`) sent to Copilot Business, Enterprise, and canonical GHEC'); @@ -211,6 +215,8 @@ describe('self-hosted runner doctor workflow config', () => { expect(source).toContain('- `invalid CapDrop: capability not supported by your kernel or not available in the current environment` → A22'); expect(source).toContain('- `error mounting "/dev/null" to .../.npmrc: create mountpoint ...: read-only file system` on `arc-dind` persisting even after upgrading past github/gh-aw-firewall#7998 (A23\'s fix), where the credential mountpoint is missing under a declared-`rw` home bind backed by a genuinely read-only directory → A24'); expect(source).toContain('- `a network with name awf-net exists but was not created for project` → B27'); + expect(source).toContain('- `docker network connect --alias awf-net ` is needed for raw-protocol GitHub Actions `services:` containers under `runner.topology: arc-dind` → A25'); + expect(source).toContain('- `host.docker.internal` or `(host.docker.internal/redacted)` appears in `network.allowDomains` but the host service still cannot be reached from inside AWF → B34'); expect(source).toContain('- TLS/certificate verification failure from api-proxy against a custom `--openai-api-target`/`--anthropic-api-target` internal endpoint using a private/corporate CA → B28 (api-proxy sidecar had no custom CA trust extension point; fixed in github/gh-aw-firewall#7816 with `apiProxy.caCert`/`--api-proxy-ca-cert`)'); expect(source).toContain('- `context-rebuild circuit breaker tripped` together with a failed `cd` into the expected workspace path → B29'); expect(source).toContain('- `awf logs summary` reports "no log sources found" after a pre-egress startup failure with no Squid `access.log` → B30'); @@ -247,6 +253,8 @@ describe('self-hosted runner doctor workflow config', () => { expect(portableAgent).toContain('- `mkdirat ... : read-only file system` at agent container startup while a `filesystem.allowWrite` policy is active (not the `chroot.binariesSourcePath`-specific A12 case) → A21; `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` followed by `Token protection will be disabled` → A21'); expect(portableAgent).toContain('- `invalid CapDrop: capability not supported by your kernel or not available in the current environment` → A22'); expect(portableAgent).toContain('- `a network with name awf-net exists but was not created for project` → B27'); + expect(portableAgent).toContain('- `docker network connect --alias awf-net ` is needed for raw-protocol GitHub Actions `services:` containers under `runner.topology: arc-dind` → A25'); + expect(portableAgent).toContain('- `host.docker.internal` or `(host.docker.internal/redacted)` appears in `network.allowDomains` but the host service still cannot be reached from inside AWF → B34'); expect(portableAgent).toContain('- `context-rebuild circuit breaker tripped` together with a failed `cd` into the expected workspace path → B29'); expect(portableAgent).toContain('- `awf logs summary` reports "no log sources found" after a pre-egress startup failure with no Squid `access.log` → B30'); expect(portableAgent).toContain('- A setup-action-selected toolchain version is shadowed by the system-default version inside the AWF agent under `sandbox.agent.runtime: docker-sudo-iptables` → B31');