diff --git a/.github/agents/self-hosted-runner-doctor.md b/.github/agents/self-hosted-runner-doctor.md index b0d4f7ce2..3d12f62c0 100644 --- a/.github/agents/self-hosted-runner-doctor.md +++ b/.github/agents/self-hosted-runner-doctor.md @@ -82,6 +82,7 @@ Prefer the narrowest match. Examples: - `copilot: command not found` inside `--container-runtime sbx` when binary is at `~/.local/bin/copilot` → D10 (`bash -lc` login init resets injected PATH in sbx; fixed by wrapping the executed command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` in github/gh-aw-firewall#6407) - `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit 139 / `Segmentation fault` on bash wrapper, often before any model or tool call → D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558) - `Model "auto" has no AI credits pricing and no default pricing is configured` together with `awf-reflect: request failed: fetch failed` under `--container-runtime gvisor` or `sbx` → D12 (isolated runtime cannot reach `/reflect` to pre-resolve `auto`; AI-credits guard rejected sentinel `auto`; fixed in github/gh-aw-firewall#6811) +- `Direct sbx egress reached 1.1.1.1 without proxy environment variables` (or a similar denied-destination reach) despite Squid healthchecks passing → D13 (`sbx daemon` restarted without `DOCKER_SANDBOXES_PROXY`; workaround/CI fix in github/gh-aw-firewall#8252) - `TCP_DENIED` in Squid access log for a topology peer or `difcProxyHost` during agent run; in-session MCP/HTTP calls to those hosts fail in network-isolation mode → B13 (topology peer hostnames and `difcProxyHost` not added to `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; if block report still flags topology peer after github/gh-aw-firewall#6473, treat as audit/policy-manifest reporting false positive tracked in github/gh-aw-firewall#6652 / github/gh-aw-firewall#6658 — runtime traffic is not blocked) - `⚠️ Firewall blocked N domain(s)` warning lists `awmgmcpg` or `172.30.0.x` as a blocked domain on every run, even with no actual external blocks → B13 (internal MCP gateway traffic counted by log aggregator as denied; fixed in github/gh-aw-firewall#6689 with `isInternalAwfDomain()` filter) - `--network-isolation is not yet supported with --enable-host-access` → B15 (compiler auto-emits both flags when `localhost` in allowlist + topology; fixed in github/gh-aw-firewall#6657) @@ -204,6 +205,8 @@ D11 / github/gh-aw-firewall#6558 — gVisor + Node.js v22 V8 ESM startup crash r D12 / github/gh-aw-firewall#6810, github/gh-aw-firewall#6811 — Copilot runs using `model: auto` under isolated runtimes (`--container-runtime gvisor` or `sbx`) could fail before agent start with `awf-reflect: request failed: fetch failed` plus `Model "auto" has no AI credits pricing and no default pricing is configured` when `apiProxy.maxAiCredits` was enabled. **Fixed in AWF (PR github/gh-aw-firewall#6811, merged 2026-08-01):** api-proxy `checkUnknownModelRejection` now allows Copilot `auto` through pre-flight (`provider === 'copilot' && model.toLowerCase() === 'auto'`) and charges credits from the response's resolved model metadata. Non-Copilot providers still reject unresolved `auto`. Workaround (older AWF): pin a concrete priced model (for example `model: claude-sonnet-4.6`). +D13 / github/gh-aw-firewall#8250, github/gh-aw-firewall#8252 — With `--container-runtime sbx` and `network.verifySbxEgress`/`--verify-sbx-egress` enabled, AWF can fail closed before agent startup with `Direct sbx egress reached 1.1.1.1 without proxy environment variables` despite healthy Squid checks. The long-lived `sbx daemon` was started or restarted without `DOCKER_SANDBOXES_PROXY` pointing to AWF's published Squid endpoint, so its own sandboxed egress bypasses Squid. This is caller/workflow daemon lifecycle, not an AWF core-code defect: start/restart the daemon with `DOCKER_SANDBOXES_PROXY=http://host.docker.internal:3128` (or the appropriate Squid gateway) before AWF creates the sandbox. This repository's smoke-workflow post-processing final restart was fixed in github/gh-aw-firewall#8252; see `docs/sbx-integration.md` for the daemon-level contract. Probe with `sbx daemon status` and inspect its environment for `DOCKER_SANDBOXES_PROXY`. + C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency is not yet implemented in the companion projects; AWF ≥ v0.27.12 provides improved diagnostics (HTTP status + targeted hint) but the underlying cause remains unresolved. C8 / github/gh-aw-firewall#5872, github/gh-aw-firewall#6237 — Copilot Business `token` prefix short-circuit on GHEC is **fixed** in AWF version including github/gh-aw-firewall#5872. **Additional fix (github/gh-aw-firewall#6237):** `gh-aw`'s offline mode sets `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` as a sentinel. In AWF before github/gh-aw-firewall#6237, this sentinel was treated as a real BYOK key, suppressing the GitHub-token auth path and producing `400` on Business/Enterprise targets. Fixed by treating `dummy-byok-key-for-offline-mode` as a non-credential sentinel (same class as AWF placeholder tokens). @@ -356,6 +359,7 @@ Establish these facts before matching a failure mode: | D10 | On `--container-runtime sbx`, Copilot CLI installed via `install_copilot_cli.sh --rootless` (redirects install to `~/.local/bin`) is **not found** even though `~/.local` is mounted into the microVM; `copilot: command not found` or `ENOENT` at agent startup | sbx executes agent commands via `bash -lc` (login shell), whose profile initialization can reset PATH and discard `--env PATH=...`. The binary is present in the VM but unreachable by name unless PATH is fixed after login init. | **Fixed in AWF (PR github/gh-aw-firewall#6407, merged 2026-07-19):** sbx wraps the executed command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` after login initialization (`withLocalBinOnPath()` in `sbx-manager.ts`). Upgrade to the AWF version including github/gh-aw-firewall#6407. Workaround (older AWF): invoke `$HOME/.local/bin/copilot` directly, or prefix the agent command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"; ...`. | Inside the sbx agent: `which copilot` or `ls ~/.local/bin/copilot` confirms binary presence; on patched AWF, inspect the executed command wrapper in sbx logs and verify it prepends `~/.local/bin` before invoking the agent command | github/gh-aw-firewall#6407 | | D11 | Copilot CLI agent starts under `--container-runtime gvisor` but exits immediately with **exit code 139** (`SIGSEGV`) or `SIGABRT` (exit 1); `[copilot-harness]` log shows all retry attempts crashing within ~90 ms (`tokenCount=0`, `stdout=0B`); the outer `bash` wrapper can also segfault, often before any model or tool call is issued. Affects `sandbox.agent.runtime: gvisor` (compose-managed gVisor) at ~8% failure rate; identical workloads on `runc` and `sbx` are unaffected. | Node.js v22 (bundled in the Copilot CLI) can trigger a V8 native assertion (`StringBytes::Encode ... Assertion failed: (written) == (u16size)`) during ESM module translation under gVisor's userspace netstack; gVisor's `mmap`/`madvise` emulation can return unexpected buffer contents to V8's UTF-8 decode path. The `SIGSEGV`/exit 139 variant can also take down the outer `bash` wrapper so the Copilot harness cannot retry. Root cause (gVisor ↔ Node.js v22 incompatibility) is **unresolved** — tracked in github/gh-aw-firewall#6558. | **Mitigated in AWF (PR github/gh-aw-firewall#6514, merged 2026-07-23):** `runAgentCommand()` detects gVisor runtime and automatically retries the agent container **once** (via `docker start awf-agent`) when the exit code is `134` (SIGABRT) or `139` (SIGSEGV) **within the first 30 s** (`GVISOR_STARTUP_CRASH_WINDOW_MS = 30_000`); `MAX_GVISOR_AGENT_RETRIES = 1`. Log reattachment uses `docker logs --since -f awf-agent` to avoid replaying the crashed attempt. Non-gVisor runtimes are unaffected. Root cause remains open; upgrade AWF to include github/gh-aw-firewall#6514 to get the retry mitigation. | Agent logs show `signal=SIGABRT duration=0s stdout=0B` on attempt 1; with github/gh-aw-firewall#6514, a second `docker start awf-agent` log block appears and usually succeeds. Without the fix, all harness retries fail identically. Confirm `containerRuntime: gvisor` in the resolved `docker-compose.redacted.yml`. | github/gh-aw-firewall#6513, github/gh-aw-firewall#6514, github/gh-aw-firewall#6558 | | D12 | Copilot workflow with `model: auto` (or no explicit top-level `model`, where gh-aw v0.84.1+ emits `auto`) fails before the agent starts under `--container-runtime gvisor` or `sbx`; harness logs `awf-reflect: fetching (apiproxy/redacted)` then `request failed: fetch failed`, followed by `400 ... Model "auto" has no AI credits pricing and no default pricing is configured`; retries fail identically with zero tokens consumed. Same workflow succeeds under default (non-isolated) AWF runtime. | Isolated agent runtimes (gVisor, sbx) may not reach `(apiproxy/redacted)`, which the harness uses to pre-resolve `auto` to a concrete priced model. Without that resolution, api-proxy `maxAiCredits` pre-flight guard (`checkUnknownModelRejection` in `guards/ai-credits-guard.js`) had no pricing for literal `auto` and rejected the request with HTTP 400, even though Copilot resolves `auto` server-side and returns priced resolved-model metadata post-response. | **Fixed in AWF (PR github/gh-aw-firewall#6811, merged 2026-08-01):** `checkUnknownModelRejection` now allows `provider === 'copilot' && model.toLowerCase() === 'auto'` to pass pre-flight; AI-credit accounting then uses the response's resolved model. Non-Copilot providers still reject unresolved `auto`. Upgrade AWF to include github/gh-aw-firewall#6811. Workaround (older AWF): pin a concrete priced model in workflow frontmatter (for example `model: claude-sonnet-4.6`) to avoid catalog-based `auto` resolution under isolated runtimes. | Confirm `sandbox.agent.runtime: gvisor` or `sbx` in resolved AWF config; check api-proxy logs for `400 ... Model "auto" has no AI credits pricing` alongside harness `awf-reflect: request failed: fetch failed`; verify whether `apiProxy.maxAiCredits` is enabled (guard only fires when enabled) | github/gh-aw-firewall#6810, github/gh-aw-firewall#6811 | +| D13 | Under `--container-runtime sbx` with `network.verifySbxEgress`/`--verify-sbx-egress` enabled, AWF reports `Direct sbx egress reached 1.1.1.1 without proxy environment variables` and aborts before the agent starts, even though Squid is healthy (`squid host.docker.internal:3128 -> 200`). | The long-lived `sbx daemon` was started or restarted without `DOCKER_SANDBOXES_PROXY` pointing to AWF's published Squid endpoint, allowing its sandboxed egress to bypass Squid. AWF correctly detects this and fails closed. | The caller/workflow owns daemon lifecycle: start or restart `sbx daemon` with `DOCKER_SANDBOXES_PROXY=http://host.docker.internal:3128` (or the appropriate Squid gateway) before AWF creates the sandbox. PR github/gh-aw-firewall#8252 fixes this repository's smoke-workflow post-processing final daemon restart. See `docs/sbx-integration.md` for the general contract. | Run `sbx daemon status` and inspect its environment for `DOCKER_SANDBOXES_PROXY`; restart it without that variable and confirm AWF's direct-egress probe reaches `1.1.1.1` or a denied destination without proxy variables. | github/gh-aw-firewall#8250, github/gh-aw-firewall#8252 | ## Error-string quick lookup @@ -398,6 +402,7 @@ Establish these facts before matching a failure mode: | `copilot: command not found` inside sbx microVM when binary is at `~/.local/bin/copilot` | D10 (`bash -lc` login init resets injected PATH; fixed by command wrapper `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` in github/gh-aw-firewall#6407) | | `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit code 139 with `Segmentation fault` on bash wrapper | D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558) | | `Model "auto" has no AI credits pricing and no default pricing is configured` together with `awf-reflect: request failed: fetch failed` under `--container-runtime gvisor` or `sbx` | D12 (isolated runtime cannot reach `/reflect` to pre-resolve `auto`; AI-credits guard rejected sentinel `auto`; fixed in github/gh-aw-firewall#6811) | +| `Direct sbx egress reached 1.1.1.1 without proxy environment variables` (or a similar denied-destination reach) despite Squid healthchecks passing | D13 (`sbx daemon` restarted without `DOCKER_SANDBOXES_PROXY`; workaround/CI fix in github/gh-aw-firewall#8252) | | `TCP_DENIED` in Squid access log for topology peer or DIFC proxy host during agent run; MCP tool calls silently fail or return connection errors in network-isolation mode | B13 (topology peers and `difcProxyHost` missing from `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; if block report still flags topology peer after github/gh-aw-firewall#6473, treat as audit/policy-manifest reporting false positive tracked in github/gh-aw-firewall#6652 / github/gh-aw-firewall#6658 — runtime traffic is not blocked) | | `⚠️ Firewall blocked N domain(s)` warning lists `awmgmcpg` or `172.30.0.x` as a blocked domain on every run, even with no actual external blocks | B13 (internal MCP gateway traffic counted by log aggregator as denied; fixed in github/gh-aw-firewall#6689 with `isInternalAwfDomain()` filter) | | `--network-isolation is not yet supported with --enable-host-access` | B15 (compiler auto-emits both flags when `localhost` in allowlist + topology; fixed in github/gh-aw-firewall#6657) | diff --git a/.github/workflows/self-hosted-runner-doctor.md b/.github/workflows/self-hosted-runner-doctor.md index 21f5aab49..9cbb6a7fc 100644 --- a/.github/workflows/self-hosted-runner-doctor.md +++ b/.github/workflows/self-hosted-runner-doctor.md @@ -111,6 +111,7 @@ Prefer the narrowest match. Examples: - `copilot: command not found` inside `--container-runtime sbx` when binary is at `~/.local/bin/copilot` → D10 (`bash -lc` login init resets injected PATH in sbx; fixed by wrapping the executed command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` in github/gh-aw-firewall#6407) - `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit 139 / `Segmentation fault` on bash wrapper, often before any model or tool call → D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558) - `Model "auto" has no AI credits pricing and no default pricing is configured` together with `awf-reflect: request failed: fetch failed` under `--container-runtime gvisor` or `sbx` → D12 (isolated runtime cannot reach `/reflect` to pre-resolve `auto`; AI-credits guard rejected sentinel `auto`; fixed in github/gh-aw-firewall#6811) +- `Direct sbx egress reached 1.1.1.1 without proxy environment variables` (or a similar denied-destination reach) despite Squid healthchecks passing → D13 (`sbx daemon` restarted without `DOCKER_SANDBOXES_PROXY`; workaround/CI fix in github/gh-aw-firewall#8252) - `TCP_DENIED` in Squid access log for a topology peer or `difcProxyHost` during agent run; in-session MCP/HTTP calls to those hosts fail in network-isolation mode → B13 (topology peer hostnames and `difcProxyHost` not added to `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; if block report still flags topology peer after github/gh-aw-firewall#6473, treat as audit/policy-manifest reporting false positive tracked in github/gh-aw-firewall#6652 / github/gh-aw-firewall#6658 — runtime traffic is not blocked) - `⚠️ Firewall blocked N domain(s)` warning lists `awmgmcpg` or `172.30.0.x` as a blocked domain on every run, even with no actual external blocks → B13 (internal MCP gateway traffic counted by log aggregator as denied; fixed in github/gh-aw-firewall#6689 with `isInternalAwfDomain()` filter) - `--network-isolation is not yet supported with --enable-host-access` → B15 (compiler auto-emits both flags when `localhost` in allowlist + topology; fixed in github/gh-aw-firewall#6657) @@ -227,6 +228,8 @@ D10 / github/gh-aw-firewall#6407 — On `--container-runtime sbx`, the Copilot C D12 / github/gh-aw-firewall#6810, github/gh-aw-firewall#6811 — Copilot runs using `model: auto` under isolated runtimes (`--container-runtime gvisor` or `sbx`) could fail before agent start with `awf-reflect: request failed: fetch failed` plus `Model "auto" has no AI credits pricing and no default pricing is configured` when `apiProxy.maxAiCredits` was enabled. **Fixed in AWF (PR github/gh-aw-firewall#6811, merged 2026-08-01):** api-proxy `checkUnknownModelRejection` now allows Copilot `auto` through pre-flight (`provider === 'copilot' && model.toLowerCase() === 'auto'`) and charges credits from the response's resolved model metadata. Non-Copilot providers still reject unresolved `auto`. Workaround (older AWF): pin a concrete priced model (for example `model: claude-sonnet-4.6`). +D13 / github/gh-aw-firewall#8250, github/gh-aw-firewall#8252 — With `--container-runtime sbx` and `network.verifySbxEgress`/`--verify-sbx-egress` enabled, AWF can fail closed before agent startup with `Direct sbx egress reached 1.1.1.1 without proxy environment variables` despite healthy Squid checks. The long-lived `sbx daemon` was started or restarted without `DOCKER_SANDBOXES_PROXY` pointing to AWF's published Squid endpoint, so its own sandboxed egress bypasses Squid. This is caller/workflow daemon lifecycle, not an AWF core-code defect: start/restart the daemon with `DOCKER_SANDBOXES_PROXY=http://host.docker.internal:3128` (or the appropriate Squid gateway) before AWF creates the sandbox. This repository's smoke-workflow post-processing final restart was fixed in github/gh-aw-firewall#8252; see `docs/sbx-integration.md` for the daemon-level contract. Probe with `sbx daemon status` and inspect its environment for `DOCKER_SANDBOXES_PROXY`. + C7 / #5615 — DIFC proxy enterprise-host awareness for `*.ghe.com` data-residency is not yet implemented in the companion projects; AWF ≥ v0.27.12 provides improved diagnostics (HTTP status + targeted hint) but the underlying cause remains unresolved. C8 / github/gh-aw-firewall#5872 — Copilot Business `token` prefix short-circuit on GHEC is **fixed** in AWF version including github/gh-aw-firewall#5872. **Additional fix (github/gh-aw-firewall#6237):** `gh-aw`'s offline mode sets `COPILOT_PROVIDER_API_KEY=dummy-byok-key-for-offline-mode` as a sentinel. In AWF before github/gh-aw-firewall#6237, this sentinel was treated as a real BYOK key, suppressing the GitHub-token auth path and producing `400` on Business/Enterprise targets. Fixed by treating `dummy-byok-key-for-offline-mode` as a non-credential sentinel (same class as AWF placeholder tokens). diff --git a/.github/workflows/shared/self-hosted-failure-modes.md b/.github/workflows/shared/self-hosted-failure-modes.md index af2973d9b..9fd71fa8e 100644 --- a/.github/workflows/shared/self-hosted-failure-modes.md +++ b/.github/workflows/shared/self-hosted-failure-modes.md @@ -114,6 +114,7 @@ Establish these facts before matching a failure mode: | D10 | On `--container-runtime sbx`, Copilot CLI installed via `install_copilot_cli.sh --rootless` (redirects install to `~/.local/bin`) is **not found** even though `~/.local` is mounted into the microVM; `copilot: command not found` or `ENOENT` at agent startup | sbx executes agent commands via `bash -lc` (login shell), whose profile initialization can reset PATH and discard `--env PATH=...`. The binary is present in the VM but unreachable by name unless PATH is fixed after login init. | **Fixed in AWF (PR github/gh-aw-firewall#6407, merged 2026-07-19):** sbx wraps the executed command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` after login initialization (`withLocalBinOnPath()` in `sbx-manager.ts`). Upgrade to the AWF version including github/gh-aw-firewall#6407. Workaround (older AWF): invoke `$HOME/.local/bin/copilot` directly, or prefix the agent command with `export PATH="$HOME/.local/bin${PATH:+:$PATH}"; ...`. | Inside the sbx agent: `which copilot` or `ls ~/.local/bin/copilot` confirms binary presence; on patched AWF, inspect the executed command wrapper in sbx logs and verify it prepends `~/.local/bin` before invoking the agent command | github/gh-aw-firewall#6407 | | D11 | Copilot CLI agent starts under `--container-runtime gvisor` but exits immediately with **exit code 139** (`SIGSEGV`) or `SIGABRT` (exit 1); `[copilot-harness]` log shows all retry attempts crashing within ~90 ms (`tokenCount=0`, `stdout=0B`); the outer `bash` wrapper can also segfault, often before any model or tool call is issued. Affects `sandbox.agent.runtime: gvisor` (compose-managed gVisor) at ~8% failure rate; identical workloads on `runc` and `sbx` are unaffected. | Node.js v22 (bundled in the Copilot CLI) can trigger a V8 native assertion (`StringBytes::Encode ... Assertion failed: (written) == (u16size)`) during ESM module translation under gVisor's userspace netstack; gVisor's `mmap`/`madvise` emulation can return unexpected buffer contents to V8's UTF-8 decode path. The `SIGSEGV`/exit 139 variant can also take down the outer `bash` wrapper so the Copilot harness cannot retry. Root cause (gVisor ↔ Node.js v22 incompatibility) is **unresolved** — tracked in github/gh-aw-firewall#6558. | **Mitigated in AWF (PR github/gh-aw-firewall#6514, merged 2026-07-23):** `runAgentCommand()` detects gVisor runtime and automatically retries the agent container **once** (via `docker start awf-agent`) when the exit code is `134` (SIGABRT) or `139` (SIGSEGV) **within the first 30 s** (`GVISOR_STARTUP_CRASH_WINDOW_MS = 30_000`); `MAX_GVISOR_AGENT_RETRIES = 1`. Log reattachment uses `docker logs --since -f awf-agent` to avoid replaying the crashed attempt. Non-gVisor runtimes are unaffected. Root cause remains open; upgrade AWF to include github/gh-aw-firewall#6514 to get the retry mitigation. | Agent logs show `signal=SIGABRT duration=0s stdout=0B` on attempt 1; with github/gh-aw-firewall#6514, a second `docker start awf-agent` log block appears and usually succeeds. Without the fix, all harness retries fail identically. Confirm `containerRuntime: gvisor` in the resolved `docker-compose.redacted.yml`. | github/gh-aw-firewall#6513, github/gh-aw-firewall#6514, github/gh-aw-firewall#6558 | | D12 | Copilot workflow with `model: auto` (or no explicit top-level `model`, where gh-aw v0.84.1+ emits `auto`) fails before the agent starts under `--container-runtime gvisor` or `sbx`; harness logs `awf-reflect: fetching (apiproxy/redacted)` then `request failed: fetch failed`, followed by `400 ... Model "auto" has no AI credits pricing and no default pricing is configured`; retries fail identically with zero tokens consumed. Same workflow succeeds under default (non-isolated) AWF runtime. | Isolated agent runtimes (gVisor, sbx) may not reach `(apiproxy/redacted)`, which the harness uses to pre-resolve `auto` to a concrete priced model. Without that resolution, api-proxy `maxAiCredits` pre-flight guard (`checkUnknownModelRejection` in `guards/ai-credits-guard.js`) had no pricing for literal `auto` and rejected the request with HTTP 400, even though Copilot resolves `auto` server-side and returns priced resolved-model metadata post-response. | **Fixed in AWF (PR github/gh-aw-firewall#6811, merged 2026-08-01):** `checkUnknownModelRejection` now allows `provider === 'copilot' && model.toLowerCase() === 'auto'` to pass pre-flight; AI-credit accounting then uses the response's resolved model. Non-Copilot providers still reject unresolved `auto`. Upgrade AWF to include github/gh-aw-firewall#6811. Workaround (older AWF): pin a concrete priced model in workflow frontmatter (for example `model: claude-sonnet-4.6`) to avoid catalog-based `auto` resolution under isolated runtimes. | Confirm `sandbox.agent.runtime: gvisor` or `sbx` in resolved AWF config; check api-proxy logs for `400 ... Model "auto" has no AI credits pricing` alongside harness `awf-reflect: request failed: fetch failed`; verify whether `apiProxy.maxAiCredits` is enabled (guard only fires when enabled) | github/gh-aw-firewall#6810, github/gh-aw-firewall#6811 | +| D13 | Under `--container-runtime sbx` with `network.verifySbxEgress`/`--verify-sbx-egress` enabled, AWF reports `Direct sbx egress reached 1.1.1.1 without proxy environment variables` and aborts before the agent starts, even though Squid is healthy (`squid host.docker.internal:3128 -> 200`). | The long-lived `sbx daemon` was started or restarted without `DOCKER_SANDBOXES_PROXY` pointing to AWF's published Squid endpoint, allowing its sandboxed egress to bypass Squid. AWF correctly detects this and fails closed. | The caller/workflow owns daemon lifecycle: start or restart `sbx daemon` with `DOCKER_SANDBOXES_PROXY=http://host.docker.internal:3128` (or the appropriate Squid gateway) before AWF creates the sandbox. PR github/gh-aw-firewall#8252 fixes this repository's smoke-workflow post-processing final daemon restart. See `docs/sbx-integration.md` for the general contract. | Run `sbx daemon status` and inspect its environment for `DOCKER_SANDBOXES_PROXY`; restart it without that variable and confirm AWF's direct-egress probe reaches `1.1.1.1` or a denied destination without proxy variables. | github/gh-aw-firewall#8250, github/gh-aw-firewall#8252 | ## Error-string quick lookup @@ -163,6 +164,7 @@ Establish these facts before matching a failure mode: | `copilot: command not found` inside sbx microVM when binary is at `~/.local/bin/copilot` | D10 (`bash -lc` login init resets injected PATH; fixed by command wrapper `export PATH="$HOME/.local/bin${PATH:+:$PATH}"` in github/gh-aw-firewall#6407) | | `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit code 139 with `Segmentation fault` on bash wrapper | D11 (Node.js v22 V8 ESM decode assertion under gVisor; one-shot restart mitigation in github/gh-aw-firewall#6514; underlying Node/gVisor incompatibility unresolved in github/gh-aw-firewall#6558) | | `Model "auto" has no AI credits pricing and no default pricing is configured` together with `awf-reflect: request failed: fetch failed` under `--container-runtime gvisor` or `sbx` | D12 (isolated runtime cannot reach `/reflect` to pre-resolve `auto`; AI-credits guard rejected sentinel `auto`; fixed in github/gh-aw-firewall#6811) | +| `Direct sbx egress reached 1.1.1.1 without proxy environment variables` (or a similar denied-destination reach) despite Squid healthchecks passing | D13 (`sbx daemon` restarted without `DOCKER_SANDBOXES_PROXY`; workaround/CI fix in github/gh-aw-firewall#8252) | | `TCP_DENIED` in Squid access log for topology peer or DIFC proxy host during agent run; MCP tool calls silently fail or return connection errors in network-isolation mode | B13 (topology peers and `difcProxyHost` missing from `NO_PROXY`; fixed in github/gh-aw-firewall#6189 and github/gh-aw-firewall#6438; if block report still flags topology peer after github/gh-aw-firewall#6473, treat as audit/policy-manifest reporting false positive tracked in github/gh-aw-firewall#6652 / github/gh-aw-firewall#6658 — runtime traffic is not blocked) | | `⚠️ Firewall blocked N domain(s)` warning lists `awmgmcpg` or `172.30.0.x` as a blocked domain on every run, even with no actual external blocks | B13 (internal MCP gateway traffic counted by log aggregator as denied; fixed in github/gh-aw-firewall#6689 with `isInternalAwfDomain()` filter) | | `--network-isolation is not yet supported with --enable-host-access` | B15 (compiler auto-emits both flags when `localhost` in allowlist + topology; fixed in github/gh-aw-firewall#6657) | diff --git a/scripts/ci/self-hosted-runner-doctor-workflow.test.ts b/scripts/ci/self-hosted-runner-doctor-workflow.test.ts index eccff4533..595d36201 100644 --- a/scripts/ci/self-hosted-runner-doctor-workflow.test.ts +++ b/scripts/ci/self-hosted-runner-doctor-workflow.test.ts @@ -203,6 +203,10 @@ describe('self-hosted runner doctor workflow config', () => { expect(content).toContain('github/gh-aw-firewall#8035, github/gh-aw-firewall#8038'); } + expect(shared).toContain('| D13 | Under `--container-runtime sbx` with `network.verifySbxEgress`/`--verify-sbx-egress` enabled, AWF reports `Direct sbx egress reached 1.1.1.1 without proxy environment variables`'); + expect(shared).toContain('github/gh-aw-firewall#8250, github/gh-aw-firewall#8252'); + expect(shared).toContain('| `Direct sbx egress reached 1.1.1.1 without proxy environment variables` (or a similar denied-destination reach) despite Squid healthchecks passing | D13'); + expect(source).toContain('- `unknown shorthand flag: \'d\' in -d` from `docker compose up -d` → A14 (DinD sidecar missing `docker-compose-plugin`)'); expect(source).toContain('- `Rootless artifact permission repair failed` on ARC/DinD squid logs → A15 (`dockerHostPathPrefix` not applied to repair bind mount)'); expect(source).toContain('- `EAI_AGAIN` / `ENOTFOUND` resolving a topology-attached DIFC proxy (for example `awmg-cli-proxy`) in network-isolation + topology-attach: if DinD `nslookup` fails, match B12; otherwise B5'); @@ -210,6 +214,7 @@ describe('self-hosted runner doctor workflow config', () => { expect(source).toContain('- credential files such as `~/.aws/credentials`, `~/.ssh/id_rsa`, or `~/.docker/config.json` are visible inside an `--container-runtime sbx` microVM → D9'); expect(source).toContain('- `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit 139 / `Segmentation fault` on bash wrapper, often before any model or tool call → D11'); expect(source).toContain('- `Model "auto" has no AI credits pricing and no default pricing is configured` together with `awf-reflect: request failed: fetch failed` under `--container-runtime gvisor` or `sbx` → D12'); + expect(source).toContain('- `Direct sbx egress reached 1.1.1.1 without proxy environment variables` (or a similar denied-destination reach) despite Squid healthchecks passing → D13'); expect(source).toContain('- `awf-agent` fails to start under `runner.topology: arc-dind` (runc cannot create the `/dev/null` credential-hiding overlay mountpoints under `/host$HOME`), or the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` → A20 (sysroot filter dropped every mount targeting `/host$HOME`, including a caller-supplied writable home; fixed in github/gh-aw-firewall#7244)'); expect(source).toContain('- `mkdirat ... : read-only file system` at agent container startup while a `filesystem.allowWrite` policy is active (not the `chroot.binariesSourcePath`-specific A12 case) → A21; `[entrypoint][WARN] Could not copy one-shot-token library to /tmp/awf-lib` followed by `Token protection will be disabled` → A21'); expect(source).toContain('- `invalid CapDrop: capability not supported by your kernel or not available in the current environment` → A22'); @@ -234,6 +239,7 @@ describe('self-hosted runner doctor workflow config', () => { expect(source).toContain('D9 / github/gh-aw-firewall#6336 — sbx microVMs previously mounted the entire host `$HOME`, exposing credentials such as `~/.aws/credentials`, `~/.ssh/id_rsa`, and `~/.docker/config.json`.'); expect(source).toContain('D11 / github/gh-aw-firewall#6558 — gVisor + Node.js v22 V8 ESM startup crash root cause remains unresolved (`SIGABRT` `StringBytes::Encode` assertion and occasional exit 139).'); expect(source).toContain('D12 / github/gh-aw-firewall#6810, github/gh-aw-firewall#6811 — Copilot runs using `model: auto` under isolated runtimes (`--container-runtime gvisor` or `sbx`) could fail before agent start with `awf-reflect: request failed: fetch failed` plus `Model "auto" has no AI credits pricing and no default pricing is configured` when `apiProxy.maxAiCredits` was enabled.'); + expect(source).toContain('D13 / github/gh-aw-firewall#8250, github/gh-aw-firewall#8252 — With `--container-runtime sbx` and `network.verifySbxEgress`/`--verify-sbx-egress` enabled, AWF can fail closed before agent startup with `Direct sbx egress reached 1.1.1.1 without proxy environment variables` despite healthy Squid checks.'); expect(source).toContain('A20 / github/gh-aw-firewall#7239, github/gh-aw-firewall#7244 — Under `runner.topology: arc-dind`, `filterAgentVolumesForSysroot()` (`src/services/optional-services.ts`) dropped every mount targeting `/host$HOME`'); expect(source).toContain('A21 / github/gh-aw-firewall#7678, github/gh-aw-firewall#7679, github/gh-aw-firewall#7681, github/gh-aw-firewall#7728 — When a `filesystem.allowWrite` policy narrows `/tmp` to read-only, `awf-agent` startup can fail with `runc create failed: ... mkdirat ... read-only file system`'); expect(source).toContain('A22 / github/gh-aw#56127, github/gh-aw-firewall#7788, github/gh-aw-firewall#7795 — `arc-dind` topology fails to start when Docker rejects AWF\'s compose `cap_drop` list'); @@ -246,6 +252,10 @@ describe('self-hosted runner doctor workflow config', () => { expect(portableAgent).toContain('- credential files such as `~/.aws/credentials`, `~/.ssh/id_rsa`, or `~/.docker/config.json` are visible inside an `--container-runtime sbx` microVM → D9'); expect(portableAgent).toContain('- `SIGABRT` / `signal=SIGABRT duration=0s stdout=0B` for Copilot CLI all retries under `--container-runtime gvisor`; or exit 139 / `Segmentation fault` on bash wrapper, often before any model or tool call → D11'); expect(portableAgent).toContain('- `Model "auto" has no AI credits pricing and no default pricing is configured` together with `awf-reflect: request failed: fetch failed` under `--container-runtime gvisor` or `sbx` → D12'); + expect(portableAgent).toContain('- `Direct sbx egress reached 1.1.1.1 without proxy environment variables` (or a similar denied-destination reach) despite Squid healthchecks passing → D13'); + expect(portableAgent).toContain('| D13 | Under `--container-runtime sbx` with `network.verifySbxEgress`/`--verify-sbx-egress` enabled, AWF reports `Direct sbx egress reached 1.1.1.1 without proxy environment variables`'); + expect(portableAgent).toContain('| `Direct sbx egress reached 1.1.1.1 without proxy environment variables` (or a similar denied-destination reach) despite Squid healthchecks passing | D13'); + expect(portableAgent).toContain('D13 / github/gh-aw-firewall#8250, github/gh-aw-firewall#8252 — With `--container-runtime sbx` and `network.verifySbxEgress`/`--verify-sbx-egress` enabled, AWF can fail closed before agent startup with `Direct sbx egress reached 1.1.1.1 without proxy environment variables` despite healthy Squid checks.'); expect(portableAgent).toContain('D11 / github/gh-aw-firewall#6558 — gVisor + Node.js v22 V8 ESM startup crash root cause remains unresolved (`SIGABRT` `StringBytes::Encode` assertion and occasional exit 139).'); expect(portableAgent).toContain('D12 / github/gh-aw-firewall#6810, github/gh-aw-firewall#6811 — Copilot runs using `model: auto` under isolated runtimes (`--container-runtime gvisor` or `sbx`) could fail before agent start with `awf-reflect: request failed: fetch failed` plus `Model "auto" has no AI credits pricing and no default pricing is configured` when `apiProxy.maxAiCredits` was enabled.'); expect(portableAgent).toContain('- `awf-agent` fails to start under `runner.topology: arc-dind` (runc cannot create the `/dev/null` credential-hiding overlay mountpoints under `/host$HOME`), or the entrypoint aborts with `mkdir -p /host$HOME/.m2` failing under `set -e` → A20 (sysroot filter dropped every mount targeting `/host$HOME`, including a caller-supplied writable home; fixed in github/gh-aw-firewall#7244)');