diff --git a/AGENTS.md b/AGENTS.md index cc294a69..7ece836a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -342,10 +342,11 @@ index to jump to the right page. issues from dogfood pipelines). NOT a regular safe-output. - [`docs/supply-chain.md`](docs/supply-chain.md) — optional `supply-chain:` front-matter section that mirrors the compiler, AWF binary, ado-script - bundle, and AWF/MCPG images from an internal Azure DevOps Artifacts feed - and/or container registry (NuGet `DownloadPackage@1` + ACR `az acr login`), - with asymmetric auth (feed defaults to `$(System.AccessToken)`; registry - requires a service connection). + bundle, and AWF/MCPG images from an internal Azure DevOps Artifacts feed, + an exact provenance-checked pipeline artifact, and/or a container registry. + Feed and pipeline artifact are mutually exclusive; registry is independent. + Feed defaults to `$(System.AccessToken)`, while registry requires a service + connection and the pipeline artifact uses the build identity. ### Compiler internals & operations @@ -442,8 +443,8 @@ Following the gh-aw security model: `src/compile/agentic_pipeline.rs::run_agent_step`). AWF runs rootlessly in strict topology mode; the Agent reaches trusted MCPG through `--topology-attach awmg-mcpg`, not through host access. All other ADO steps — - binary/bundle downloads, `docker pull`, ACR/NuGet auth (including the - `supply-chain:` mirror fetches) — run *outside* the sandbox with the build + binary/bundle downloads, `docker pull`, ACR/NuGet auth (including feed and + pipeline-artifact `supply-chain:` fetches) — run *outside* the sandbox with the build agent pool's normal network, so they do **not** need entries in the AWF allowlist. Air-gapping the build agent itself from GitHub/GHCR is the agent pool's network policy, not AWF. @@ -496,6 +497,17 @@ anything it flags. If a finding is genuinely intentional, add a `# shellcheck disable=SCxxxx` comment immediately above the offending line in the bash body — shellcheck honours the directive and it's inert at runtime. +### Release-owned smoke lock files + +`tests/safe-outputs/*.lock.yml` are the latest-release customer contract. Do +not regenerate them with `cargo run -- compile` from an unreleased checkout: +their runtime integrity step downloads the released compiler, so development +output can drift even while Cargo still reports the same semver. Compiler PRs +and nightly `main` are exercised by `tests/compiler-smoke-e2e/`, which +recompiles all five workflows in a temporary worktree and stages them on an +ephemeral `ado-aw-mirror` ref. The release workflow updates the checked-in +locks only after matching release assets exist. + ## Common Tasks ### Compile a markdown pipeline diff --git a/README.md b/README.md index 01ce7735..b00bfe42 100644 --- a/README.md +++ b/README.md @@ -729,7 +729,8 @@ index to jump to the right page. - [`docs/safe-output-permissions.md`](docs/safe-output-permissions.md) — diagnosis and fix reference for Stage 3 permission failures (401/403). - [`docs/supply-chain.md`](docs/supply-chain.md) — optional `supply-chain:` - front-matter section for internal feed/registry mirrors. + front-matter section for internal feed/registry mirrors and exact + provenance-checked pipeline artifacts. - [`docs/ado-aw-debug.md`](docs/ado-aw-debug.md) — debug-only `ado-aw-debug:` front-matter section (`skip-integrity`, `create-issue`). diff --git a/docs/front-matter.md b/docs/front-matter.md index c8865592..ee7bb27c 100644 --- a/docs/front-matter.md +++ b/docs/front-matter.md @@ -217,10 +217,15 @@ supply-chain: # optional internal supply-chain mirror (see docs feed: # mirror binaries (compiler, AWF, ado-script) from an ADO Artifacts feed name: my-project/my-feed # feed name or project/feed; scalar `feed: my-feed` shorthand also works service-connection: feed-conn # optional; omit for same-org feeds (uses $(System.AccessToken)) + # pipeline-artifact: # alternative binary source; mutually exclusive with feed + # project: AgentPlayground # validated ADO project name or GUID + # definition-id: 2560 # positive producer pipeline definition ID + # run-id: 630001 # positive, exact producer build ID + # artifact: ado-aw-candidate # complete payload/checksum/provenance artifact registry: # mirror AWF/MCPG images from an internal ACR name: myacr.azurecr.io/mirror # registry host or base path (artifact names kept under it) service-connection: acr-conn # REQUIRED when registry is set (ACR has no System.AccessToken path) - service-connection: shared-conn # optional shared fallback for whichever target omits its own + service-connection: shared-conn # optional feed/registry fallback; never applies to pipeline-artifact # ado-aw-debug: # debug-only knobs; see docs/ado-aw-debug.md # skip-integrity: false # omit generated pipeline integrity verification # create-issue: false # dogfood-only GitHub issue filing for debug reports diff --git a/docs/supply-chain.md b/docs/supply-chain.md index 5ddba291..008860b3 100644 --- a/docs/supply-chain.md +++ b/docs/supply-chain.md @@ -11,10 +11,10 @@ GitHub / GHCR at run time: | 4 | AWF + MCPG container images | `ghcr.io/github/...` | The optional `supply-chain:` front-matter section reroutes these fetches to an -**internal Azure DevOps Artifacts feed** (for the binaries #1–#3) and/or an -**internal container registry** (for the images #4). This is intended for -supply-chain-hardened environments where the build agent pool cannot reach -GitHub / GHCR. +**internal Azure DevOps Artifacts feed** or one exact **Azure DevOps pipeline +artifact** (for the binaries #1–#3), and/or an **internal container registry** +(for the images #4). This is intended for supply-chain-hardened environments +where the build agent pool cannot reach GitHub / GHCR. When `supply-chain:` is omitted, the generated pipeline is byte-for-byte identical to before — there is no behavioural change for existing agents. @@ -26,6 +26,12 @@ supply-chain: feed: # mirrors binaries #1, #2, #3 name: my-project/my-feed # feed name or "project/feed" service-connection: feed-conn # optional (see Authentication) + # Alternative to feed (the two are mutually exclusive): + # pipeline-artifact: + # project: AgentPlayground + # definition-id: 2560 + # run-id: 630001 + # artifact: ado-aw-candidate registry: # mirrors images #4 name: myacr.azurecr.io/mirror # registry host or base path service-connection: acr-conn # required when registry is set @@ -35,10 +41,12 @@ supply-chain: | Field | Type | Required | Purpose | |-------|------|----------|---------| | `feed` | scalar **or** `{ name, service-connection }` | optional | Enables the binary mirror (#1–#3). A bare string is shorthand for `{ name: }`. | -| `registry` | scalar **or** `{ name, service-connection }` | optional | Enables the image mirror (#4). | -| `service-connection` | string | optional | Shared fallback connection used by whichever target does not declare its own. | +| `pipeline-artifact` | `{ project, definition-id, run-id, artifact }` | optional | Uses one exact producer run as the complete binary source (#1–#3). Mutually exclusive with `feed`. | +| `registry` | scalar **or** `{ name, service-connection }` | optional | Enables the image mirror (#4), independently of either binary source. | +| `service-connection` | string | optional | Shared fallback connection for `feed` and `registry`; it does not apply to `pipeline-artifact`. | -`feed` and `registry` are **independent** — set either, both, or neither. +`feed` and `pipeline-artifact` are mutually exclusive. `registry` is +independent and may accompany either binary source. ### Scalar shorthand @@ -71,6 +79,28 @@ role and `NuGetAuthenticate@1` authenticates automatically via `$(System.AccessToken)`. Set a `service-connection` only for cross-org or external feeds. +### Pipeline artifact (binaries) + +The pipeline-artifact source is immutable configuration: the generated +`DownloadPipelineArtifact@2` tasks always use `source: specific`, +`runVersion: specific`, the configured project, definition ID (`pipeline`), +run ID (`runId`), and artifact name. The compiler never selects latest runs, +tags, or triggering pipelines. + +`project` must be an Azure DevOps project name or canonical GUID; +`definition-id` and `run-id` must be positive integers; and `artifact` follows +Azure DevOps artifact-name rules. The current build identity must have +permission to read the configured project, pipeline run, and artifact. +The producer must be in the same Azure DevOps organization; cross-organization +pipeline-artifact downloads are not supported by this source. +`supply-chain.service-connection` is not used for this source, and no +`NuGetAuthenticate@1` step is emitted. + +With an exact run ID, the artifact is consumable as soon as its publish task +has completed even if later jobs keep the producer run active. Consumers +queued before the named artifact exists fail; producers should confirm artifact +visibility before queueing them. + ### Registry (images) The image mirror authenticates with `az acr login` (`AzureCLI@2`) using the @@ -96,7 +126,7 @@ used by `permissions:`), passed to `AzureCLI@2` as `azureSubscription`. > `registry.name` to the canonical `*.azurecr.io` login server so the derived > registry name is correct. -## What the feed and registry must contain +## What the binary source and registry must contain Versions stay **pinned by the generating compiler** — the internal mirror must host the exact pinned versions. A mirror that is a few days behind is fine: use @@ -119,6 +149,47 @@ payload, so the package simply needs to carry the same files (and matching `sha256sum -c checksums.txt` is preserved**, so the mirror must ship the matching `checksums.txt`. +### Pipeline artifact + +The single configured artifact is a complete source and must contain exactly +one file with each of these names (directory nesting is allowed): + +- `ado-aw-linux-x64` +- `awf-linux-x64` +- `ado-script.zip` +- `checksums.txt` +- `provenance.json` + +`checksums.txt` must have exactly one standard sha256sum entry for each payload, +with the exact filename. Each consumer locates its payload plus the shared +manifest and provenance document, verifies the exact checksum entry, then +preserves the normal chmod/move/unzip/PATH behavior. + +`provenance.json` must be a JSON object using this Phase 1 contract: + +```json +{ + "schema": "ado-aw/candidate-artifact/1", + "producer_definition_id": 2560, + "producer_build_id": 630001, + "repository": "githubnext/ado-aw", + "source_ref": "refs/heads/main", + "source_version": "0123456789abcdef0123456789abcdef01234567", + "reason": "IndividualCI", + "compiler_version": "0.45.1", + "awf_version": "0.27.32" +} +``` + +The required identity fields are exactly `schema`, `producer_definition_id`, +and `producer_build_id`; both producer IDs are JSON numbers. Compilation embeds +the configured IDs, and every consumer fails closed if the schema or either +producer ID is missing, non-numeric, or mismatched. `repository`, `source_ref`, +`source_version`, `reason`, `compiler_version`, and `awf_version` are optional +diagnostic fields: when present, the generated step includes them in the +validated, non-secret provenance output. Their absence does not invalidate an +otherwise valid artifact. + ### Registry images `registry.name` is a registry **host or base path** — teams generally cannot @@ -175,6 +246,31 @@ supply-chain: feed: my-internal-feed ``` +All binaries from one exact candidate pipeline run: + +```yaml +supply-chain: + pipeline-artifact: + project: AgentPlayground + definition-id: 2560 + run-id: 630001 + artifact: ado-aw-candidate +``` + +Candidate binaries plus mirrored images: + +```yaml +supply-chain: + pipeline-artifact: + project: AgentPlayground + definition-id: 2560 + run-id: 630001 + artifact: ado-aw-candidate + registry: + name: myacr.azurecr.io + service-connection: acr-conn +``` + Images only: ```yaml @@ -186,9 +282,10 @@ supply-chain: ## Network isolation note -The mirror fetches (`NuGetAuthenticate@1`, `DownloadPackage@1`, `docker pull`, -`az acr login`) run as ordinary ADO steps on the build agent — **outside** the -AWF network-isolation sandbox, which wraps only the copilot agent command. +The mirror fetches (`NuGetAuthenticate@1`, `DownloadPackage@1`, +`DownloadPipelineArtifact@2`, `docker pull`, `az acr login`) run as ordinary +ADO steps on the build agent — **outside** the AWF network-isolation sandbox, +which wraps only the copilot agent command. Consequently: - The feed/registry hosts are **not** added to the agent's AWF diff --git a/rotate-agentplayground-secrets.ps1 b/rotate-agentplayground-secrets.ps1 new file mode 100644 index 00000000..fe4888ba --- /dev/null +++ b/rotate-agentplayground-secrets.ps1 @@ -0,0 +1,98 @@ +[CmdletBinding()] +param( + [string] $Organization = "msazuresphere", + [string] $Project = "AgentPlayground" +) + +Set-StrictMode -Version Latest +$ErrorActionPreference = "Stop" + +$copilotDefinitionIds = @(2545, 2546, 2547, 2548, 2549, 2554, 2555, 2556, 2557, 2558) +$reporterDefinitionIds = @(2549, 2558) +$executorDefinitionIds = @(2550) +$triggerDefinitionIds = @(2551) +$allDefinitionIds = @( + $copilotDefinitionIds + $executorDefinitionIds + $triggerDefinitionIds +) | Sort-Object -Unique + +function Set-AdoAwSecret { + param( + [Parameter(Mandatory)] + [string] $Name, + + [Parameter(Mandatory)] + [string] $Value, + + [Parameter(Mandatory)] + [int[]] $DefinitionIds + ) + + Write-Host "Setting $Name on definitions $($DefinitionIds -join ',')..." + $Value | & cargo run --quiet -- secrets set $Name --value-stdin ` + --org $Organization ` + --project $Project ` + --definition-ids ($DefinitionIds -join ",") + + if ($LASTEXITCODE -ne 0) { + throw "Failed to set $Name on definitions $($DefinitionIds -join ',')." + } +} + +Push-Location $PSScriptRoot +try { + $copilotToken = Read-Host "New Copilot GITHUB_TOKEN" -MaskInput + if ([string]::IsNullOrWhiteSpace($copilotToken)) { + throw "The Copilot GITHUB_TOKEN cannot be empty." + } + + Set-AdoAwSecret ` + -Name "GITHUB_TOKEN" ` + -Value $copilotToken ` + -DefinitionIds $copilotDefinitionIds + + Remove-Variable copilotToken + + $issuesToken = Read-Host ` + "New issues-only GitHub PAT (leave blank to preserve existing issue tokens)" ` + -MaskInput + + if ([string]::IsNullOrWhiteSpace($issuesToken)) { + Write-Host "Preserving existing issue-reporting tokens." + } + else { + Set-AdoAwSecret ` + -Name "ADO_AW_DEBUG_GITHUB_TOKEN" ` + -Value $issuesToken ` + -DefinitionIds $reporterDefinitionIds + + Set-AdoAwSecret ` + -Name "EXECUTOR_E2E_GITHUB_TOKEN" ` + -Value $issuesToken ` + -DefinitionIds $executorDefinitionIds + + Set-AdoAwSecret ` + -Name "TRIGGER_E2E_GITHUB_TOKEN" ` + -Value $issuesToken ` + -DefinitionIds $triggerDefinitionIds + } + + Remove-Variable issuesToken + + Write-Host "" + Write-Host "Verifying variable names and secret flags (values are never printed)..." + & cargo run --quiet -- secrets list ` + --org $Organization ` + --project $Project ` + --definition-ids ($allDefinitionIds -join ",") + + if ($LASTEXITCODE -ne 0) { + throw "Secret rotation succeeded, but verification failed." + } +} +finally { + Remove-Variable copilotToken -ErrorAction SilentlyContinue + Remove-Variable issuesToken -ErrorAction SilentlyContinue + Pop-Location +} diff --git a/scripts/ado-script/package-lock.json b/scripts/ado-script/package-lock.json index 46f8cc39..2e153160 100644 --- a/scripts/ado-script/package-lock.json +++ b/scripts/ado-script/package-lock.json @@ -9,7 +9,8 @@ "version": "0.0.0", "dependencies": { "azure-devops-node-api": "^14.1.0", - "picomatch": "^4.0.4" + "picomatch": "^4.0.4", + "yaml": "^2.9.0" }, "devDependencies": { "@types/node": "^20.19.39", @@ -1799,6 +1800,21 @@ "engines": { "node": ">=8" } + }, + "node_modules/yaml": { + "version": "2.9.0", + "resolved": "https://registry.npmjs.org/yaml/-/yaml-2.9.0.tgz", + "integrity": "sha512-2AvhNX3mb8zd6Zy7INTtSpl1F15HW6Wnqj0srWlkKLcpYl/gMIMJiyuGq2KeI2YFxUPjdlB+3Lc10seMLtL4cA==", + "license": "ISC", + "bin": { + "yaml": "bin.mjs" + }, + "engines": { + "node": ">= 14.6" + }, + "funding": { + "url": "https://github.com/sponsors/eemeli" + } } } } diff --git a/scripts/ado-script/package.json b/scripts/ado-script/package.json index d52b039a..e95f33e9 100644 --- a/scripts/ado-script/package.json +++ b/scripts/ado-script/package.json @@ -26,6 +26,7 @@ "build:prepare-pr-base": "ncc build src/prepare-pr-base/index.ts -o .ado-build/prepare-pr-base -m -t && node -e \"const fs=require('node:fs'); fs.copyFileSync('.ado-build/prepare-pr-base/index.js','prepare-pr-base.js'); fs.rmSync('.ado-build/prepare-pr-base',{recursive:true,force:true});\"", "build:executor-e2e": "ncc build src/executor-e2e/index.ts -o .ado-build/executor-e2e -m -t && node -e \"const fs=require('node:fs'); fs.mkdirSync('test-bin',{recursive:true}); fs.copyFileSync('.ado-build/executor-e2e/index.js','test-bin/executor-e2e.js'); fs.rmSync('.ado-build/executor-e2e',{recursive:true,force:true});\"", "build:trigger-e2e": "ncc build src/trigger-e2e/index.ts -o .ado-build/trigger-e2e -m -t && node -e \"const fs=require('node:fs'); fs.mkdirSync('test-bin',{recursive:true}); fs.copyFileSync('.ado-build/trigger-e2e/index.js','test-bin/trigger-e2e.js'); fs.rmSync('.ado-build/trigger-e2e',{recursive:true,force:true});\"", + "build:compiler-smoke-e2e": "ncc build src/compiler-smoke-e2e/index.ts -o .ado-build/compiler-smoke-e2e -m -t && node -e \"const fs=require('node:fs'); fs.mkdirSync('test-bin',{recursive:true}); fs.copyFileSync('.ado-build/compiler-smoke-e2e/index.js','test-bin/compiler-smoke-e2e.js'); fs.rmSync('.ado-build/compiler-smoke-e2e',{recursive:true,force:true});\"", "build:check": "ls -lh gate.js && wc -c gate.js", "codegen": "node -e \"require('node:fs').mkdirSync('schema', { recursive: true })\" && cargo run --quiet --manifest-path ../../Cargo.toml -- export-gate-schema --output schema/gate-spec.schema.json && npx json2ts schema/gate-spec.schema.json -o src/shared/types.gen.ts --bannerComment \"// AUTO-GENERATED from Rust IR via cargo run -- export-gate-schema. Do not edit; run npm run codegen.\" && cargo run --quiet --manifest-path ../../Cargo.toml -- export-fact-catalog --output src/trigger-e2e/fact-catalog.gen.json", "test": "vitest run", @@ -35,7 +36,8 @@ }, "dependencies": { "azure-devops-node-api": "^14.1.0", - "picomatch": "^4.0.4" + "picomatch": "^4.0.4", + "yaml": "^2.9.0" }, "devDependencies": { "@types/node": "^20.19.39", diff --git a/scripts/ado-script/src/__tests__/bundle-coverage.test.ts b/scripts/ado-script/src/__tests__/bundle-coverage.test.ts index 4e6e89ce..f0e31ebd 100644 --- a/scripts/ado-script/src/__tests__/bundle-coverage.test.ts +++ b/scripts/ado-script/src/__tests__/bundle-coverage.test.ts @@ -34,8 +34,21 @@ const packageJsonPath = join(here, "..", "..", "package.json"); * `trigger-e2e` is the analogous deterministic trigger-condition (gate / * synth-PR) E2E harness. Same treatment: its own `build:trigger-e2e` emits to * `test-bin/`, so it is never packaged in `ado-script.zip`. + * + * `compiler-smoke-e2e` is the deterministic compiler-candidate smoke E2E + * harness: it stages the `supply-chain.pipeline-artifact` candidate compiler + * across the real `tests/safe-outputs/*.md` fixtures and drives the fixed + * "candidate lane" pipeline definitions. Same treatment: its own + * `build:compiler-smoke-e2e` emits to `test-bin/`, so it is never packaged in + * `ado-script.zip`. */ -const NON_BUNDLE_DIRS = new Set(["shared", "__tests__", "executor-e2e", "trigger-e2e"]); +const NON_BUNDLE_DIRS = new Set([ + "shared", + "__tests__", + "executor-e2e", + "trigger-e2e", + "compiler-smoke-e2e", +]); function listBundleDirs(): string[] { return readdirSync(srcDir) diff --git a/scripts/ado-script/src/compiler-smoke-e2e/__tests__/ado-rest.test.ts b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/ado-rest.test.ts new file mode 100644 index 00000000..565e1acb --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/ado-rest.test.ts @@ -0,0 +1,173 @@ +import { describe, expect, it, vi } from "vitest"; + +import { AdoRest, redactToken } from "../ado-rest.js"; + +function jsonResponse(status: number, body: unknown): Response { + return new Response(JSON.stringify(body), { + status, + headers: { "content-type": "application/json" }, + }); +} + +function emptyResponse(status: number): Response { + return new Response(null, { status }); +} + +function makeRest(fetchImpl: typeof fetch, extra: Partial[0]> = {}) { + return new AdoRest({ + orgUrl: "https://dev.azure.com/org/", + project: "AgentPlayground", + token: "secret-token", + fetchImpl, + sleepImpl: async () => {}, + log: () => {}, + ...extra, + }); +} + +describe("AdoRest.getArtifact", () => { + it("returns the artifact on the first successful attempt", async () => { + const fetchImpl = vi.fn(async () => jsonResponse(200, { name: "ado-aw-candidate" })); + const rest = makeRest(fetchImpl as unknown as typeof fetch); + const artifact = await rest.getArtifact(100, "ado-aw-candidate"); + expect(artifact.name).toBe("ado-aw-candidate"); + expect(fetchImpl).toHaveBeenCalledTimes(1); + }); + + it("retries a 404 up to the configured bound, then succeeds", async () => { + let calls = 0; + const fetchImpl = vi.fn(async () => { + calls++; + if (calls < 3) return emptyResponse(404); + return jsonResponse(200, { name: "ado-aw-candidate" }); + }); + const rest = makeRest(fetchImpl as unknown as typeof fetch); + const artifact = await rest.getArtifact(100, "ado-aw-candidate", { retries: 5, retryDelayMs: 1 }); + expect(artifact.name).toBe("ado-aw-candidate"); + expect(calls).toBe(3); + }); + + it("throws after exhausting all retries", async () => { + const fetchImpl = vi.fn(async () => emptyResponse(404)); + const rest = makeRest(fetchImpl as unknown as typeof fetch); + await expect( + rest.getArtifact(100, "ado-aw-candidate", { retries: 2, retryDelayMs: 1 }), + ).rejects.toThrow(/not visible/); + expect(fetchImpl).toHaveBeenCalledTimes(2); + }); + + it("scopes the request to the exact same project as the producer build", async () => { + const fetchImpl = vi.fn(async (input: RequestInfo | URL) => { + expect(String(input)).toContain("/AgentPlayground/"); + return jsonResponse(200, { name: "a" }); + }); + const rest = makeRest(fetchImpl as unknown as typeof fetch); + await rest.getArtifact(100, "a"); + }); +}); + +describe("AdoRest.queueBuild", () => { + it("always sends both sourceBranch and sourceVersion", async () => { + let sentBody: unknown; + const fetchImpl = vi.fn(async (_input: RequestInfo | URL, init?: RequestInit) => { + sentBody = JSON.parse(String(init?.body)); + return jsonResponse(200, { id: 555 }); + }); + const rest = makeRest(fetchImpl as unknown as typeof fetch); + const result = await rest.queueBuild(2560, { + sourceBranch: "refs/heads/ado-aw-smoke-candidate/1", + sourceVersion: "deadbeef", + }); + expect(result.id).toBe(555); + expect(sentBody).toMatchObject({ + definition: { id: 2560 }, + sourceBranch: "refs/heads/ado-aw-smoke-candidate/1", + sourceVersion: "deadbeef", + }); + }); + + it("throws with a descriptive error on a non-2xx response", async () => { + const fetchImpl = vi.fn(async () => new Response("nope", { status: 500 })); + const rest = makeRest(fetchImpl as unknown as typeof fetch); + await expect( + rest.queueBuild(2560, { sourceBranch: "refs/heads/x", sourceVersion: "sha" }), + ).rejects.toThrow(/HTTP 500/); + }); +}); + +describe("AdoRest.getBuild / cancelBuild", () => { + it("getBuild returns the parsed build summary", async () => { + const fetchImpl = vi.fn(async () => jsonResponse(200, { id: 1, status: "completed", result: "succeeded" })); + const rest = makeRest(fetchImpl as unknown as typeof fetch); + const build = await rest.getBuild(1); + expect(build.status).toBe("completed"); + expect(build.result).toBe("succeeded"); + }); + + it("cancelBuild PATCHes the build to status=cancelling", async () => { + let method: string | undefined; + let body: unknown; + const fetchImpl = vi.fn(async (_input: RequestInfo | URL, init?: RequestInit) => { + method = init?.method; + body = JSON.parse(String(init?.body)); + return emptyResponse(204); + }); + const rest = makeRest(fetchImpl as unknown as typeof fetch); + await rest.cancelBuild(1); + expect(method).toBe("PATCH"); + expect(body).toEqual({ status: "cancelling" }); + }); +}); + +describe("AdoRest.listBuildsForDefinitionBranch", () => { + it("queries a single definition + exact branch and returns every build regardless of status", async () => { + let requestedPath = ""; + const fetchImpl = vi.fn(async (input: RequestInfo | URL) => { + requestedPath = String(input); + return jsonResponse(200, { + value: [ + { id: 1, status: "completed", result: "succeeded" }, + { id: 2, status: "inProgress" }, + ], + }); + }); + const rest = makeRest(fetchImpl as unknown as typeof fetch); + const builds = await rest.listBuildsForDefinitionBranch(3001, "refs/heads/ado-aw-smoke-candidate/1"); + expect(builds).toHaveLength(2); + expect(builds[1]?.status).toBe("inProgress"); + // The query is scoped to exactly one definition id + the exact branch — + // never a comma-separated statusFilter (status is inspected client-side + // instead, per the stale-scan safety requirement). + expect(requestedPath).toContain("definitions=3001"); + expect(requestedPath).toContain(encodeURIComponent("refs/heads/ado-aw-smoke-candidate/1")); + expect(requestedPath).not.toContain("statusFilter"); + }); + + it("returns an empty array when there are no builds on that branch", async () => { + const fetchImpl = vi.fn(async () => jsonResponse(200, { value: [] })); + const rest = makeRest(fetchImpl as unknown as typeof fetch); + const builds = await rest.listBuildsForDefinitionBranch(3001, "refs/heads/ado-aw-smoke-candidate/2"); + expect(builds).toEqual([]); + }); +}); + +describe("AdoRest.buildUrl", () => { + it("builds a human-facing build results URL", () => { + const rest = makeRest((async () => emptyResponse(200)) as unknown as typeof fetch); + expect(rest.buildUrl(123)).toBe( + "https://dev.azure.com/org/AgentPlayground/_build/results?buildId=123", + ); + }); +}); + +describe("redactToken", () => { + it("replaces the token with ***", () => { + expect(redactToken("Bearer secret-token in header", "secret-token")).toBe( + "Bearer *** in header", + ); + }); + + it("is a no-op for an undefined token", () => { + expect(redactToken("nothing to redact", undefined)).toBe("nothing to redact"); + }); +}); diff --git a/scripts/ado-script/src/compiler-smoke-e2e/__tests__/assertions.test.ts b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/assertions.test.ts new file mode 100644 index 00000000..65ad1d3b --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/assertions.test.ts @@ -0,0 +1,119 @@ +import { describe, expect, it } from "vitest"; + +import { assertNoForbiddenReleaseUrls, assertPipelineArtifactValues } from "../assertions.js"; + +const EXPECTED = { + project: "AgentPlayground", + pipeline: "2560", + runId: "630001", + artifact: "ado-aw-candidate", +}; + +function specificRunYaml(overrides: Partial = {}): string { + const values = { ...EXPECTED, ...overrides }; + return ` +steps: + - task: DownloadPipelineArtifact@2 + displayName: Download Pipeline Artifact + inputs: + targetPath: $(Pipeline.Workspace)/in + source: specific + project: ${values.project} + pipeline: '${values.pipeline}' + runVersion: specific + runId: '${values.runId}' + artifact: ${values.artifact} +`; +} + +describe("assertNoForbiddenReleaseUrls", () => { + it("passes for YAML with no forbidden release URL", () => { + expect(() => assertNoForbiddenReleaseUrls(specificRunYaml(), "canary")).not.toThrow(); + }); + + it("throws when the compiler release URL is present", () => { + const yaml = `${specificRunYaml()}\n# https://github.com/githubnext/ado-aw/releases/download/v1/ado-aw\n`; + expect(() => assertNoForbiddenReleaseUrls(yaml, "canary")).toThrow(/release URL/); + }); + + it("throws when the AWF firewall release URL is present", () => { + const yaml = `${specificRunYaml()}\n# https://github.com/github/gh-aw-firewall/releases/download/v1/awf\n`; + expect(() => assertNoForbiddenReleaseUrls(yaml, "canary")).toThrow(/release URL/); + }); +}); + +describe("assertPipelineArtifactValues", () => { + it("passes when the DownloadPipelineArtifact step matches exactly", () => { + expect(() => assertPipelineArtifactValues(specificRunYaml(), "canary", EXPECTED)).not.toThrow(); + }); + + it("throws when there is no specific-run DownloadPipelineArtifact step at all", () => { + const yaml = "steps:\n - task: Bash@3\n inputs:\n script: echo hi\n"; + expect(() => assertPipelineArtifactValues(yaml, "canary", EXPECTED)).toThrow(/no 'specific run'/); + }); + + it("throws on a project mismatch", () => { + expect(() => + assertPipelineArtifactValues(specificRunYaml({ project: "WrongProject" }), "canary", EXPECTED), + ).toThrow(/mismatch/); + }); + + it("throws on a pipeline (definition id) mismatch", () => { + expect(() => + assertPipelineArtifactValues(specificRunYaml({ pipeline: "9999" }), "canary", EXPECTED), + ).toThrow(/mismatch/); + }); + + it("throws on a runId mismatch", () => { + expect(() => + assertPipelineArtifactValues(specificRunYaml({ runId: "1" }), "canary", EXPECTED), + ).toThrow(/mismatch/); + }); + + it("throws on an artifact name mismatch", () => { + expect(() => + assertPipelineArtifactValues(specificRunYaml({ artifact: "wrong-name" }), "canary", EXPECTED), + ).toThrow(/mismatch/); + }); + + it("ignores a DownloadPipelineArtifact step whose source is 'current' (not our pinned source)", () => { + const yaml = ` +steps: + - task: DownloadPipelineArtifact@2 + inputs: + targetPath: $(Pipeline.Workspace)/in + source: current + artifact: safe_outputs + - task: DownloadPipelineArtifact@2 + inputs: + targetPath: $(Pipeline.Workspace)/in2 + source: specific + project: ${EXPECTED.project} + pipeline: '${EXPECTED.pipeline}' + runVersion: specific + runId: '${EXPECTED.runId}' + artifact: ${EXPECTED.artifact} +`; + expect(() => assertPipelineArtifactValues(yaml, "canary", EXPECTED)).not.toThrow(); + }); + + it("finds a DownloadPipelineArtifact step nested inside stages/jobs", () => { + const yaml = ` +stages: + - stage: Agent + jobs: + - job: run + steps: + - task: DownloadPipelineArtifact@2 + inputs: + targetPath: in + source: specific + project: ${EXPECTED.project} + pipeline: '${EXPECTED.pipeline}' + runVersion: specific + runId: '${EXPECTED.runId}' + artifact: ${EXPECTED.artifact} +`; + expect(() => assertPipelineArtifactValues(yaml, "canary", EXPECTED)).not.toThrow(); + }); +}); diff --git a/scripts/ado-script/src/compiler-smoke-e2e/__tests__/compile-cli.test.ts b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/compile-cli.test.ts new file mode 100644 index 00000000..9a49ffbf --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/compile-cli.test.ts @@ -0,0 +1,126 @@ +import { describe, expect, it, vi } from "vitest"; + +const spawnCalls: { cmd: string; args: string[]; env?: NodeJS.ProcessEnv }[] = []; +let scriptedOutcomes: Array<{ status: number | null; stdout?: string; stderr?: string; timedOut?: boolean }> = []; + +vi.mock("../process.js", async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + safeSpawn: vi.fn( + async (request: { cmd: string; args: string[]; env?: NodeJS.ProcessEnv }) => { + spawnCalls.push({ cmd: request.cmd, args: request.args, env: request.env }); + const next = scriptedOutcomes.shift(); + if (!next) throw new Error("no scripted outcome left"); + return { + status: next.status, + stdout: next.stdout ?? "", + stderr: next.stderr ?? "", + timedOut: next.timedOut ?? false, + stdoutTruncated: false, + stderrTruncated: false, + }; + }, + ), + }; +}); + +const { compileAndCheck } = await import("../compile-cli.js"); + +function reset(outcomes: typeof scriptedOutcomes): void { + spawnCalls.length = 0; + scriptedOutcomes = [...outcomes]; +} + +describe("compileAndCheck", () => { + it("runs 'compile --force ' then 'check ' in order and reports success", async () => { + reset([{ status: 0, stdout: "compiled\n" }, { status: 0, stdout: "checked\n" }]); + const result = await compileAndCheck({ + adoAwBin: "C:\\bin\\ado-aw.exe", + worktreeDir: "/wt", + metadataRemoteUrl: + "https://dev.azure.com/msazuresphere/AgentPlayground/_git/ado-aw-mirror", + relMd: "tests/safe-outputs/canary.md", + relLock: "tests/safe-outputs/canary.lock.yml", + timeoutMs: 1000, + }); + expect(result.ok).toBe(true); + expect(spawnCalls.map(({ cmd, args }) => ({ cmd, args }))).toEqual([ + { + cmd: "C:\\bin\\ado-aw.exe", + args: ["compile", "--force", "tests/safe-outputs/canary.md"], + }, + { + cmd: "C:\\bin\\ado-aw.exe", + args: ["check", "tests/safe-outputs/canary.lock.yml"], + }, + ]); + for (const call of spawnCalls) { + expect(call.env).toEqual({ + ADO_AW_COMPILE_REMOTE_URL: + "https://dev.azure.com/msazuresphere/AgentPlayground/_git/ado-aw-mirror", + }); + } + }); + + it("reports phase 'compile' and never runs check when compile fails", async () => { + reset([{ status: 1, stderr: "parse error" }]); + const result = await compileAndCheck({ + adoAwBin: "ado-aw", + worktreeDir: "/wt", + metadataRemoteUrl: "https://dev.azure.com/o/p/_git/r", + relMd: "tests/safe-outputs/canary.md", + relLock: "tests/safe-outputs/canary.lock.yml", + timeoutMs: 1000, + }); + expect(result.ok).toBe(false); + expect(result.phase).toBe("compile"); + expect(result.message).toMatch(/exited 1/); + expect(spawnCalls).toHaveLength(1); + }); + + it("reports phase 'check' when compile succeeds but check fails", async () => { + reset([{ status: 0, stdout: "compiled\n" }, { status: 2, stderr: "check failed" }]); + const result = await compileAndCheck({ + adoAwBin: "ado-aw", + worktreeDir: "/wt", + metadataRemoteUrl: "https://dev.azure.com/o/p/_git/r", + relMd: "tests/safe-outputs/canary.md", + relLock: "tests/safe-outputs/canary.lock.yml", + timeoutMs: 1000, + }); + expect(result.ok).toBe(false); + expect(result.phase).toBe("check"); + expect(spawnCalls).toHaveLength(2); + }); + + it("reports a timeout as a compile-phase failure without throwing", async () => { + reset([{ status: null, timedOut: true }]); + const result = await compileAndCheck({ + adoAwBin: "ado-aw", + worktreeDir: "/wt", + metadataRemoteUrl: "https://dev.azure.com/o/p/_git/r", + relMd: "tests/safe-outputs/canary.md", + relLock: "tests/safe-outputs/canary.lock.yml", + timeoutMs: 1000, + }); + expect(result.ok).toBe(false); + expect(result.phase).toBe("compile"); + expect(result.message).toMatch(/timed out/); + }); + + it("redacts configured secrets from captured stdout/stderr", async () => { + reset([{ status: 1, stderr: "token=super-secret-value leaked" }]); + const result = await compileAndCheck({ + adoAwBin: "ado-aw", + worktreeDir: "/wt", + metadataRemoteUrl: "https://dev.azure.com/o/p/_git/r", + relMd: "tests/safe-outputs/canary.md", + relLock: "tests/safe-outputs/canary.lock.yml", + timeoutMs: 1000, + secrets: ["super-secret-value"], + }); + expect(result.stderr).not.toContain("super-secret-value"); + expect(result.stderr).toContain("***"); + }); +}); diff --git a/scripts/ado-script/src/compiler-smoke-e2e/__tests__/config.test.ts b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/config.test.ts new file mode 100644 index 00000000..e701d631 --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/config.test.ts @@ -0,0 +1,192 @@ +import { describe, expect, it } from "vitest"; + +import { candidateRef, loadConfig } from "../config.js"; + +function baseEnv(overrides: Record = {}): NodeJS.ProcessEnv { + return { + SYSTEM_COLLECTIONURI: "https://dev.azure.com/org/", + SYSTEM_TEAMPROJECT: "AgentPlayground", + SYSTEM_ACCESSTOKEN: "tok", + BUILD_BUILDID: "42", + BUILD_SOURCEBRANCH: "refs/heads/main", + BUILD_SOURCEVERSION: "abc123", + BUILD_SOURCESDIRECTORY: "C:\\repo", + SYSTEM_DEFINITIONID: "99", + COMPILER_SMOKE_ADO_AW_BIN: "C:\\bin\\ado-aw.exe", + COMPILER_SMOKE_ARTIFACT_NAME: "ado-aw-candidate", + COMPILER_SMOKE_MIRROR_REPO: "ado-aw-mirror", + COMPILER_SMOKE_CANARY_DEFINITION_ID: "2601", + COMPILER_SMOKE_AZURE_CLI_DEFINITION_ID: "2602", + COMPILER_SMOKE_NOOP_TARGET_DEFINITION_ID: "2603", + COMPILER_SMOKE_JANITOR_DEFINITION_ID: "2604", + COMPILER_SMOKE_REPORTER_DEFINITION_ID: "2605", + ...overrides, + }; +} + +describe("loadConfig", () => { + it("parses a fully valid environment", () => { + const config = loadConfig(baseEnv()); + expect(config.orgUrl).toBe("https://dev.azure.com/org/"); + expect(config.project).toBe("AgentPlayground"); + expect(config.buildId).toBe(42); + expect(config.definitionId).toBe(99); + expect(config.definitionIds).toEqual({ + canary: 2601, + "azure-cli": 2602, + "noop-target": 2603, + janitor: 2604, + "smoke-failure-reporter": 2605, + }); + expect(config.concurrency).toBe(5); + expect(config.childTimeoutMs).toBe(7_200_000); + expect(config.pollMs).toBe(10_000); + expect(config.staleRefHours).toBe(24); + }); + + for (const name of [ + "SYSTEM_COLLECTIONURI", + "SYSTEM_TEAMPROJECT", + "SYSTEM_ACCESSTOKEN", + "BUILD_BUILDID", + "BUILD_SOURCEBRANCH", + "BUILD_SOURCEVERSION", + "BUILD_SOURCESDIRECTORY", + "SYSTEM_DEFINITIONID", + "COMPILER_SMOKE_ADO_AW_BIN", + "COMPILER_SMOKE_ARTIFACT_NAME", + "COMPILER_SMOKE_MIRROR_REPO", + "COMPILER_SMOKE_CANARY_DEFINITION_ID", + "COMPILER_SMOKE_AZURE_CLI_DEFINITION_ID", + "COMPILER_SMOKE_NOOP_TARGET_DEFINITION_ID", + "COMPILER_SMOKE_JANITOR_DEFINITION_ID", + "COMPILER_SMOKE_REPORTER_DEFINITION_ID", + ]) { + it(`rejects a missing ${name}`, () => { + expect(() => loadConfig(baseEnv({ [name]: undefined }))).toThrow(); + }); + + it(`rejects an unexpanded ADO macro for ${name}`, () => { + expect(() => loadConfig(baseEnv({ [name]: "$(Some.Macro)" }))).toThrow(/unexpanded|not set/); + }); + } + + it("rejects a malformed (non-numeric) BUILD_BUILDID", () => { + expect(() => loadConfig(baseEnv({ BUILD_BUILDID: "abc" }))).toThrow(/positive integer/); + }); + + it("rejects a zero BUILD_BUILDID", () => { + expect(() => loadConfig(baseEnv({ BUILD_BUILDID: "0" }))).toThrow(/positive integer/); + }); + + it("rejects a negative SYSTEM_DEFINITIONID", () => { + expect(() => loadConfig(baseEnv({ SYSTEM_DEFINITIONID: "-5" }))).toThrow(/positive integer/); + }); + + it("rejects a non-integer fixture definition id", () => { + expect(() => loadConfig(baseEnv({ COMPILER_SMOKE_JANITOR_DEFINITION_ID: "12.5" }))).toThrow( + /positive integer/, + ); + }); + + it("rejects duplicate fixture definition ids", () => { + expect(() => + loadConfig( + baseEnv({ + COMPILER_SMOKE_AZURE_CLI_DEFINITION_ID: "2601", + }), + ), + ).toThrow(/distinct/); + }); + + it("reports every duplicated fixture in the error message", () => { + expect(() => + loadConfig( + baseEnv({ + COMPILER_SMOKE_AZURE_CLI_DEFINITION_ID: "2601", + COMPILER_SMOKE_NOOP_TARGET_DEFINITION_ID: "2604", + COMPILER_SMOKE_JANITOR_DEFINITION_ID: "2604", + }), + ), + ).toThrow(/canary/); + }); + + describe("COMPILER_SMOKE_CONCURRENCY bounds", () => { + it("defaults to 5 when unset", () => { + expect(loadConfig(baseEnv()).concurrency).toBe(5); + }); + + it("accepts the lower bound (1)", () => { + expect(loadConfig(baseEnv({ COMPILER_SMOKE_CONCURRENCY: "1" })).concurrency).toBe(1); + }); + + it("accepts the upper bound (5)", () => { + expect(loadConfig(baseEnv({ COMPILER_SMOKE_CONCURRENCY: "5" })).concurrency).toBe(5); + }); + + it("rejects 0", () => { + expect(() => loadConfig(baseEnv({ COMPILER_SMOKE_CONCURRENCY: "0" }))).toThrow(/range/); + }); + + it("rejects 6", () => { + expect(() => loadConfig(baseEnv({ COMPILER_SMOKE_CONCURRENCY: "6" }))).toThrow(/range/); + }); + + it("rejects a non-integer value", () => { + expect(() => loadConfig(baseEnv({ COMPILER_SMOKE_CONCURRENCY: "2.5" }))).toThrow(/integer/); + }); + }); + + describe("COMPILER_SMOKE_CHILD_TIMEOUT_MS", () => { + it("defaults to 7200000ms", () => { + expect(loadConfig(baseEnv()).childTimeoutMs).toBe(7_200_000); + }); + + it("accepts an explicit override", () => { + expect(loadConfig(baseEnv({ COMPILER_SMOKE_CHILD_TIMEOUT_MS: "60000" })).childTimeoutMs).toBe(60_000); + }); + }); + + describe("COMPILER_SMOKE_POLL_MS", () => { + it("defaults to 10000ms", () => { + expect(loadConfig(baseEnv()).pollMs).toBe(10_000); + }); + + it("accepts an explicit override", () => { + expect(loadConfig(baseEnv({ COMPILER_SMOKE_POLL_MS: "5000" })).pollMs).toBe(5_000); + }); + }); + + describe("COMPILER_SMOKE_STALE_REF_HOURS bounds", () => { + it("defaults to 24", () => { + expect(loadConfig(baseEnv()).staleRefHours).toBe(24); + }); + + it("accepts the minimum (6)", () => { + expect(loadConfig(baseEnv({ COMPILER_SMOKE_STALE_REF_HOURS: "6" })).staleRefHours).toBe(6); + }); + + it("rejects below the minimum (5)", () => { + expect(() => loadConfig(baseEnv({ COMPILER_SMOKE_STALE_REF_HOURS: "5" }))).toThrow(/range/); + }); + }); + + it("trims surrounding whitespace from string env vars", () => { + const config = loadConfig(baseEnv({ SYSTEM_TEAMPROJECT: " AgentPlayground " })); + expect(config.project).toBe("AgentPlayground"); + }); + + it("treats an empty string the same as unset", () => { + expect(() => loadConfig(baseEnv({ SYSTEM_TEAMPROJECT: "" }))).toThrow(); + }); +}); + +describe("candidateRef", () => { + it("builds the deterministic per-run candidate ref", () => { + expect(candidateRef(42)).toBe("refs/heads/ado-aw-smoke-candidate/42"); + }); + + it("never collides with a plausible base ref name", () => { + expect(candidateRef(1)).not.toBe("refs/heads/main"); + }); +}); diff --git a/scripts/ado-script/src/compiler-smoke-e2e/__tests__/fixtures.test.ts b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/fixtures.test.ts new file mode 100644 index 00000000..1e1f84f9 --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/fixtures.test.ts @@ -0,0 +1,50 @@ +import { describe, expect, it } from "vitest"; + +import { ALL_FIXTURES, allowedChangedPaths, fixturePaths, FIXTURE_DIR } from "../fixtures.js"; + +describe("fixturePaths", () => { + it("builds repo-relative md/lock paths under tests/safe-outputs", () => { + expect(fixturePaths("canary")).toEqual({ + name: "canary", + relMd: "tests/safe-outputs/canary.md", + relLock: "tests/safe-outputs/canary.lock.yml", + }); + }); +}); + +describe("ALL_FIXTURES", () => { + it("has exactly the five fixtures in the required stable order", () => { + expect(ALL_FIXTURES.map((f) => f.name)).toEqual([ + "canary", + "azure-cli", + "noop-target", + "janitor", + "smoke-failure-reporter", + ]); + }); + + it("every fixture path lives under the shared FIXTURE_DIR", () => { + for (const f of ALL_FIXTURES) { + expect(f.relMd.startsWith(`${FIXTURE_DIR}/`)).toBe(true); + expect(f.relLock.startsWith(`${FIXTURE_DIR}/`)).toBe(true); + } + }); +}); + +describe("allowedChangedPaths", () => { + it("contains exactly the five md files, five lock files, and .gitattributes", () => { + const allowed = allowedChangedPaths(); + expect(allowed.size).toBe(11); + expect(allowed.has(".gitattributes")).toBe(true); + for (const f of ALL_FIXTURES) { + expect(allowed.has(f.relMd)).toBe(true); + expect(allowed.has(f.relLock)).toBe(true); + } + }); + + it("does not allow an arbitrary unrelated path", () => { + const allowed = allowedChangedPaths(); + expect(allowed.has("src/main.rs")).toBe(false); + expect(allowed.has("tests/safe-outputs/README.md")).toBe(false); + }); +}); diff --git a/scripts/ado-script/src/compiler-smoke-e2e/__tests__/git-runner.test.ts b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/git-runner.test.ts new file mode 100644 index 00000000..7e2acd94 --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/git-runner.test.ts @@ -0,0 +1,59 @@ +import { describe, expect, it, vi } from "vitest"; + +const spawnCalls: { cmd: string; args: string[]; env?: NodeJS.ProcessEnv }[] = []; + +vi.mock("../process.js", async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + safeSpawn: vi.fn(async (request: { cmd: string; args: string[]; env?: NodeJS.ProcessEnv }) => { + spawnCalls.push({ cmd: request.cmd, args: request.args, env: request.env }); + return { + status: 0, + stdout: "", + stderr: "", + timedOut: false, + stdoutTruncated: false, + stderrTruncated: false, + }; + }), + }; +}); + +const { defaultGitRunner } = await import("../git.js"); + +function reset(): void { + spawnCalls.length = 0; +} + +describe("defaultGitRunner", () => { + it("forces GIT_TERMINAL_PROMPT=0 on every invocation so a bad token/unreachable mirror never hangs on a prompt", async () => { + reset(); + await defaultGitRunner(["status"], { cwd: "/repo", timeoutMs: 1000 }); + expect(spawnCalls[0]?.cmd).toBe("git"); + expect(spawnCalls[0]?.env?.GIT_TERMINAL_PROMPT).toBe("0"); + }); + + it("layers caller-supplied env (e.g. a bearer token) on top without it being shadowed", async () => { + reset(); + await defaultGitRunner(["ls-remote", "https://example/_git/r"], { + cwd: "/repo", + timeoutMs: 1000, + env: { GIT_CONFIG_COUNT: "1", GIT_CONFIG_KEY_0: "http.extraheader", GIT_CONFIG_VALUE_0: "Authorization: bearer t" }, + }); + expect(spawnCalls[0]?.env?.GIT_TERMINAL_PROMPT).toBe("0"); + expect(spawnCalls[0]?.env?.GIT_CONFIG_VALUE_0).toBe("Authorization: bearer t"); + }); + + it("never lets a caller-supplied env override GIT_TERMINAL_PROMPT back on", async () => { + reset(); + await defaultGitRunner(["fetch"], { + cwd: "/repo", + timeoutMs: 1000, + env: { GIT_TERMINAL_PROMPT: "1" }, + }); + // GIT_TERMINAL_PROMPT=0 is a true floor: applied after any caller env, + // so it can never be silently reopened. + expect(spawnCalls[0]?.env?.GIT_TERMINAL_PROMPT).toBe("0"); + }); +}); diff --git a/scripts/ado-script/src/compiler-smoke-e2e/__tests__/git.test.ts b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/git.test.ts new file mode 100644 index 00000000..b38bbd9b --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/git.test.ts @@ -0,0 +1,389 @@ +import { describe, expect, it } from "vitest"; + +import type { GitRunner, GitRunOptions } from "../git.js"; +import { + commitAll, + commitMessage, + COMMIT_IDENTITY, + createDetachedWorktree, + deleteRemoteRef, + disallowedChanges, + listCandidateRefs, + mirrorRepoUrl, + parseCandidateBuildId, + pushCandidate, + removeWorktree, + verifyLocalCommit, + verifyRemoteRef, + worktreeChangedFiles, +} from "../git.js"; + +function fakeRunner( + handler: ( + args: string[], + opts: GitRunOptions, + ) => { status: number | null; stdout?: string; stderr?: string; timedOut?: boolean }, +): { runner: GitRunner; calls: { args: string[]; opts: GitRunOptions }[] } { + const calls: { args: string[]; opts: GitRunOptions }[] = []; + const runner: GitRunner = async (args, opts) => { + calls.push({ args, opts }); + const r = handler(args, opts); + return { + status: r.status, + stdout: r.stdout ?? "", + stderr: r.stderr ?? "", + timedOut: r.timedOut ?? false, + stdoutTruncated: false, + stderrTruncated: false, + }; + }; + return { runner, calls }; +} + +describe("mirrorRepoUrl", () => { + it("builds the ADO git remote URL, percent-encoding project/repo", () => { + expect(mirrorRepoUrl("https://dev.azure.com/org/", "My Project", "ado-aw-mirror")).toBe( + "https://dev.azure.com/org/My%20Project/_git/ado-aw-mirror", + ); + }); + + it("strips a trailing slash from orgUrl", () => { + expect(mirrorRepoUrl("https://dev.azure.com/org", "P", "R")).toBe( + "https://dev.azure.com/org/P/_git/R", + ); + }); +}); + +describe("commitMessage", () => { + it("matches the required exact format", () => { + expect(commitMessage(42)).toBe("test(smoke): stage compiler candidate 42"); + }); +}); + +describe("COMMIT_IDENTITY", () => { + it("is a deterministic, non-empty identity", () => { + expect(COMMIT_IDENTITY.name).toBeTruthy(); + expect(COMMIT_IDENTITY.email).toBeTruthy(); + }); +}); + +describe("disallowedChanges", () => { + it("returns an empty array when every changed path is allowed", () => { + const allowed = new Set(["a.md", "b.lock.yml"]); + expect(disallowedChanges(["a.md", "b.lock.yml"], allowed)).toEqual([]); + }); + + it("returns exactly the unexpected paths", () => { + const allowed = new Set(["a.md"]); + expect(disallowedChanges(["a.md", "unexpected.txt"], allowed)).toEqual(["unexpected.txt"]); + }); + + it("is order-preserving", () => { + const allowed = new Set(); + expect(disallowedChanges(["z", "a", "m"], allowed)).toEqual(["z", "a", "m"]); + }); +}); + +describe("parseCandidateBuildId", () => { + it("parses the numeric build id from a well-formed candidate ref", () => { + expect(parseCandidateBuildId("refs/heads/ado-aw-smoke-candidate/123")).toBe(123); + }); + + it("returns undefined for a ref with the wrong prefix", () => { + expect(parseCandidateBuildId("refs/heads/main")).toBeUndefined(); + }); + + it("returns undefined for a non-numeric suffix", () => { + expect(parseCandidateBuildId("refs/heads/ado-aw-smoke-candidate/abc")).toBeUndefined(); + }); + + it("returns undefined for a zero or negative-looking suffix", () => { + expect(parseCandidateBuildId("refs/heads/ado-aw-smoke-candidate/0")).toBeUndefined(); + expect(parseCandidateBuildId("refs/heads/ado-aw-smoke-candidate/-5")).toBeUndefined(); + }); +}); + +describe("worktreeChangedFiles", () => { + it("parses git status --porcelain=v1 output, one path per line", async () => { + const { runner } = fakeRunner(() => ({ + status: 0, + stdout: " M tests/safe-outputs/canary.md\n?? tests/safe-outputs/canary.lock.yml\n", + })); + const files = await worktreeChangedFiles({ worktreeDir: "/wt", timeoutMs: 1000 }, runner); + expect(files).toEqual([ + "tests/safe-outputs/canary.md", + "tests/safe-outputs/canary.lock.yml", + ]); + }); + + it("expands a rename line into both the old and new path", async () => { + const { runner } = fakeRunner(() => ({ + status: 0, + stdout: "R old/path.md -> new/path.md\n", + })); + const files = await worktreeChangedFiles({ worktreeDir: "/wt", timeoutMs: 1000 }, runner); + expect(files).toEqual(["old/path.md", "new/path.md"]); + }); + + it("returns an empty array for a clean worktree", async () => { + const { runner } = fakeRunner(() => ({ status: 0, stdout: "" })); + const files = await worktreeChangedFiles({ worktreeDir: "/wt", timeoutMs: 1000 }, runner); + expect(files).toEqual([]); + }); +}); + +describe("verifyLocalCommit", () => { + it("resolves without fetching anything when HEAD already matches expectedSha", async () => { + const { runner, calls } = fakeRunner((args) => { + if (args[0] === "rev-parse" && args[1] === "HEAD") return { status: 0, stdout: "deadbeef\n" }; + throw new Error(`unexpected args: ${args.join(" ")}`); + }); + await expect( + verifyLocalCommit({ cwd: "/repo", expectedSha: "deadbeef", timeoutMs: 1000 }, runner), + ).resolves.toBeUndefined(); + expect(calls.map((c) => c.args[0])).toEqual(["rev-parse"]); + expect(calls.every((c) => c.args[0] !== "fetch")).toBe(true); + }); + + it("falls back to an object-existence check when HEAD differs (e.g. a PR synthetic merge commit)", async () => { + const { runner, calls } = fakeRunner((args) => { + if (args[0] === "rev-parse" && args[1] === "HEAD") return { status: 0, stdout: "mergecommit\n" }; + if (args[0] === "cat-file") return { status: 0 }; + throw new Error(`unexpected args: ${args.join(" ")}`); + }); + await expect( + verifyLocalCommit({ cwd: "/repo", expectedSha: "prheadsha", timeoutMs: 1000 }, runner), + ).resolves.toBeUndefined(); + expect(calls.some((c) => c.args[0] === "fetch")).toBe(false); + expect(calls.some((c) => c.args.join(" ") === "cat-file -e prheadsha^{commit}")).toBe(true); + }); + + it("throws (never fetches from the mirror) when the commit is not present locally at all", async () => { + const { runner, calls } = fakeRunner((args) => { + if (args[0] === "rev-parse" && args[1] === "HEAD") return { status: 0, stdout: "mergecommit\n" }; + if (args[0] === "cat-file") return { status: 1 }; + throw new Error(`unexpected args: ${args.join(" ")}`); + }); + await expect( + verifyLocalCommit({ cwd: "/repo", expectedSha: "missingsha", timeoutMs: 1000 }, runner), + ).rejects.toThrow(/not found as a commit object/); + expect(calls.some((c) => c.args[0] === "fetch")).toBe(false); + }); + + it("never attempts to resolve a GitHub PR ref like refs/pull//merge against the mirror", async () => { + // This is the regression this function exists to prevent: a PR build's + // BUILD_SOURCEBRANCH (refs/pull/123/merge) does not exist on the ADO + // mirror repo at all. verifyLocalCommit never even accepts a `ref` or + // `mirrorUrl` parameter, so there is no way for a caller to pass one in. + const { runner, calls } = fakeRunner((args) => { + if (args[0] === "rev-parse" && args[1] === "HEAD") return { status: 0, stdout: "prmergecommit\n" }; + return { status: 0 }; + }); + await verifyLocalCommit({ cwd: "/repo", expectedSha: "prmergecommit", timeoutMs: 1000 }, runner); + expect(calls.every((c) => !c.args.includes("refs/pull/123/merge"))).toBe(true); + }); +}); + +describe("createDetachedWorktree / removeWorktree", () => { + it("adds a detached worktree at the given commitish", async () => { + const { runner, calls } = fakeRunner(() => ({ status: 0 })); + await createDetachedWorktree({ cwd: "/repo", worktreeDir: "/tmp/wt", commitish: "deadbeef", timeoutMs: 1000 }, runner); + expect(calls[0]?.args).toEqual(["worktree", "add", "--detach", "/tmp/wt", "deadbeef"]); + }); + + it("force-removes the worktree", async () => { + const { runner, calls } = fakeRunner(() => ({ status: 0 })); + await removeWorktree({ cwd: "/repo", worktreeDir: "/tmp/wt", timeoutMs: 1000 }, runner); + expect(calls[0]?.args).toEqual(["worktree", "remove", "--force", "/tmp/wt"]); + }); +}); + +describe("commitAll", () => { + it("stages everything, commits with the deterministic identity/message, returns the new sha", async () => { + const { runner, calls } = fakeRunner((args) => { + if (args[0] === "add") return { status: 0 }; + if (args[0] === "-c") return { status: 0 }; + if (args[0] === "rev-parse") return { status: 0, stdout: "cafebabe\n" }; + throw new Error(`unexpected args: ${args.join(" ")}`); + }); + const sha = await commitAll({ worktreeDir: "/wt", buildId: 42, timeoutMs: 1000 }, runner); + expect(sha).toBe("cafebabe"); + expect(calls[0]?.args).toEqual(["add", "-A"]); + const commitCall = calls[1]?.args ?? []; + expect(commitCall).toContain(`user.name=${COMMIT_IDENTITY.name}`); + expect(commitCall).toContain(`user.email=${COMMIT_IDENTITY.email}`); + expect(commitCall).toContain("test(smoke): stage compiler candidate 42"); + }); +}); + +describe("pushCandidate / verifyRemoteRef / deleteRemoteRef", () => { + it("pushes HEAD to the ref without --force", async () => { + const { runner, calls } = fakeRunner(() => ({ status: 0 })); + await pushCandidate( + { worktreeDir: "/wt", mirrorUrl: "https://example/_git/r", ref: "refs/heads/x/1", token: "t", timeoutMs: 1000 }, + runner, + ); + expect(calls[0]?.args).toEqual(["push", "--porcelain", "https://example/_git/r", "HEAD:refs/heads/x/1"]); + expect(calls[0]?.args).not.toContain("--force"); + expect(calls[0]?.args).not.toContain("-f"); + }); + + it("verifyRemoteRef succeeds when ls-remote returns the expected sha", async () => { + const { runner } = fakeRunner(() => ({ status: 0, stdout: "deadbeef\trefs/heads/x/1\n" })); + await expect( + verifyRemoteRef( + { + cwd: "/wt", + mirrorUrl: "https://example/_git/r", + ref: "refs/heads/x/1", + expectedSha: "deadbeef", + token: "secret-token", + timeoutMs: 1000, + }, + runner, + ), + ).resolves.toBeUndefined(); + }); + + it("verifyRemoteRef authenticates ls-remote with a bearer env (private mirror reads need auth too)", async () => { + const { runner, calls } = fakeRunner(() => ({ status: 0, stdout: "deadbeef\trefs/heads/x/1\n" })); + await verifyRemoteRef( + { + cwd: "/wt", + mirrorUrl: "https://example/_git/r", + ref: "refs/heads/x/1", + expectedSha: "deadbeef", + token: "secret-token", + timeoutMs: 1000, + }, + runner, + ); + expect(calls[0]?.args).toEqual(["ls-remote", "https://example/_git/r", "refs/heads/x/1"]); + expect(calls[0]?.opts.env?.GIT_CONFIG_VALUE_0).toContain("secret-token"); + }); + + it("verifyRemoteRef redacts the token from a thrown failure message", async () => { + const { runner } = fakeRunner(() => ({ status: 1, stderr: "fatal: auth failed for secret-token" })); + await expect( + verifyRemoteRef( + { + cwd: "/wt", + mirrorUrl: "https://example/_git/r", + ref: "refs/heads/x/1", + expectedSha: "deadbeef", + token: "secret-token", + timeoutMs: 1000, + }, + runner, + ), + ).rejects.toThrow(/\*\*\*/); + }); + + it("verifyRemoteRef throws on a sha mismatch", async () => { + const { runner } = fakeRunner(() => ({ status: 0, stdout: "other-sha\trefs/heads/x/1\n" })); + await expect( + verifyRemoteRef( + { + cwd: "/wt", + mirrorUrl: "https://example/_git/r", + ref: "refs/heads/x/1", + expectedSha: "deadbeef", + token: "t", + timeoutMs: 1000, + }, + runner, + ), + ).rejects.toThrow(/verification failed/); + }); + + it("verifyRemoteRef throws when the ref is missing entirely", async () => { + const { runner } = fakeRunner(() => ({ status: 0, stdout: "" })); + await expect( + verifyRemoteRef( + { + cwd: "/wt", + mirrorUrl: "https://example/_git/r", + ref: "refs/heads/x/1", + expectedSha: "deadbeef", + token: "t", + timeoutMs: 1000, + }, + runner, + ), + ).rejects.toThrow(); + }); + + it("deleteRemoteRef pushes a --delete for exactly the given ref", async () => { + const { runner, calls } = fakeRunner(() => ({ status: 0 })); + await deleteRemoteRef( + { cwd: "/repo", mirrorUrl: "https://example/_git/r", ref: "refs/heads/x/1", token: "t", timeoutMs: 1000 }, + runner, + ); + expect(calls[0]?.args).toEqual(["push", "--porcelain", "https://example/_git/r", "--delete", "refs/heads/x/1"]); + }); +}); + +describe("listCandidateRefs", () => { + it("lists only refs under the exact candidate prefix", async () => { + const { runner } = fakeRunner(() => ({ + status: 0, + stdout: [ + "aaa1\trefs/heads/ado-aw-smoke-candidate/1", + "bbb2\trefs/heads/ado-aw-smoke-candidate/2", + ].join("\n"), + })); + const refs = await listCandidateRefs( + { cwd: "/repo", mirrorUrl: "https://example/_git/r", token: "secret-token", timeoutMs: 1000 }, + runner, + ); + expect(refs).toEqual([ + { ref: "refs/heads/ado-aw-smoke-candidate/1", sha: "aaa1" }, + { ref: "refs/heads/ado-aw-smoke-candidate/2", sha: "bbb2" }, + ]); + }); + + it("authenticates ls-remote with a bearer env so a private mirror's stale-ref scan actually works", async () => { + const { runner, calls } = fakeRunner(() => ({ status: 0, stdout: "" })); + await listCandidateRefs( + { cwd: "/repo", mirrorUrl: "https://example/_git/r", token: "secret-token", timeoutMs: 1000 }, + runner, + ); + expect(calls[0]?.opts.env?.GIT_CONFIG_VALUE_0).toContain("secret-token"); + }); + + it("redacts the token from a thrown failure message", async () => { + const { runner } = fakeRunner(() => ({ status: 1, stderr: "fatal: auth failed for secret-token" })); + await expect( + listCandidateRefs( + { cwd: "/repo", mirrorUrl: "https://example/_git/r", token: "secret-token", timeoutMs: 1000 }, + runner, + ), + ).rejects.toThrow(/\*\*\*/); + }); + + it("excludes any ref whose glob match is not an exact-prefix match", async () => { + const { runner } = fakeRunner(() => ({ + status: 0, + stdout: [ + "aaa1\trefs/heads/ado-aw-smoke-candidate/1", + // A pathological server match that merely CONTAINS the pattern but + // does not start with the exact prefix must never be treated as ours. + "ccc3\trefs/heads/other/ado-aw-smoke-candidate/3", + ].join("\n"), + })); + const refs = await listCandidateRefs( + { cwd: "/repo", mirrorUrl: "https://example/_git/r", token: "auth-token", timeoutMs: 1000 }, + runner, + ); + expect(refs).toEqual([{ ref: "refs/heads/ado-aw-smoke-candidate/1", sha: "aaa1" }]); + }); + + it("returns an empty array when there are no candidate refs", async () => { + const { runner } = fakeRunner(() => ({ status: 0, stdout: "" })); + const refs = await listCandidateRefs( + { cwd: "/repo", mirrorUrl: "https://example/_git/r", token: "auth-token", timeoutMs: 1000 }, + runner, + ); + expect(refs).toEqual([]); + }); +}); diff --git a/scripts/ado-script/src/compiler-smoke-e2e/__tests__/index.test.ts b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/index.test.ts new file mode 100644 index 00000000..557281f1 --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/index.test.ts @@ -0,0 +1,308 @@ +import { describe, expect, it, vi, beforeEach } from "vitest"; + +const mockCalls: string[] = []; + +const baseEnv = { + SYSTEM_COLLECTIONURI: "https://dev.azure.com/org/", + SYSTEM_TEAMPROJECT: "AgentPlayground", + SYSTEM_ACCESSTOKEN: "secret-token", + BUILD_BUILDID: "630001", + BUILD_SOURCEBRANCH: "refs/heads/main", + BUILD_SOURCEVERSION: "basecommit", + BUILD_SOURCESDIRECTORY: "C:\\repo", + SYSTEM_DEFINITIONID: "2560", + COMPILER_SMOKE_ADO_AW_BIN: "C:\\bin\\ado-aw.exe", + COMPILER_SMOKE_ARTIFACT_NAME: "ado-aw-candidate", + COMPILER_SMOKE_MIRROR_REPO: "ado-aw-mirror", + COMPILER_SMOKE_CANARY_DEFINITION_ID: "3001", + COMPILER_SMOKE_AZURE_CLI_DEFINITION_ID: "3002", + COMPILER_SMOKE_NOOP_TARGET_DEFINITION_ID: "3003", + COMPILER_SMOKE_JANITOR_DEFINITION_ID: "3004", + COMPILER_SMOKE_REPORTER_DEFINITION_ID: "3005", + COMPILER_SMOKE_CHILD_TIMEOUT_MS: "5000", + COMPILER_SMOKE_POLL_MS: "1", +}; + +function specificRunYaml(): string { + return ` +steps: + - task: DownloadPipelineArtifact@2 + inputs: + targetPath: in + source: specific + project: AgentPlayground + pipeline: '2560' + runVersion: specific + runId: '630001' + artifact: ado-aw-candidate +`; +} + +vi.mock("../ado-rest.js", () => { + return { + AdoRest: vi.fn().mockImplementation(function FakeAdoRest() { + return { + getArtifact: vi.fn(async () => { + mockCalls.push("getArtifact"); + return { name: "ado-aw-candidate" }; + }), + getBuild: vi.fn(async () => ({ status: "completed", result: "succeeded" })), + queueBuild: vi.fn(async () => ({ id: 1 })), + cancelBuild: vi.fn(async () => {}), + buildUrl: (id: number) => `https://example/${id}`, + }; + }), + redactToken: (text: string) => text, + }; +}); + +vi.mock("../git.js", async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + mirrorRepoUrl: actual.mirrorRepoUrl, + verifyLocalCommit: vi.fn(async () => { + mockCalls.push("verifyLocalCommit"); + }), + createDetachedWorktree: vi.fn(async () => { + mockCalls.push("createDetachedWorktree"); + }), + removeWorktree: vi.fn(async () => { + mockCalls.push("removeWorktree"); + }), + worktreeChangedFiles: vi.fn(async () => { + mockCalls.push("worktreeChangedFiles"); + return [ + "tests/safe-outputs/canary.md", + "tests/safe-outputs/canary.lock.yml", + "tests/safe-outputs/azure-cli.md", + "tests/safe-outputs/azure-cli.lock.yml", + "tests/safe-outputs/noop-target.md", + "tests/safe-outputs/noop-target.lock.yml", + "tests/safe-outputs/janitor.md", + "tests/safe-outputs/janitor.lock.yml", + "tests/safe-outputs/smoke-failure-reporter.md", + "tests/safe-outputs/smoke-failure-reporter.lock.yml", + ]; + }), + commitAll: vi.fn(async () => { + mockCalls.push("commitAll"); + return "candidate-sha"; + }), + pushCandidate: vi.fn(async () => { + mockCalls.push("pushCandidate"); + }), + verifyRemoteRef: vi.fn(async () => { + mockCalls.push("verifyRemoteRef"); + }), + deleteRemoteRef: vi.fn(async () => { + mockCalls.push("deleteRemoteRef"); + }), + listCandidateRefs: vi.fn(async () => { + mockCalls.push("listCandidateRefs"); + return []; + }), + }; +}); + +vi.mock("../compile-cli.js", () => ({ + compileAndCheck: vi.fn(async () => { + mockCalls.push("compileAndCheck"); + return { ok: true, stdout: "", stderr: "" }; + }), +})); + +vi.mock("../runner.js", async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + runFixtures: vi.fn(async (_client: unknown, requests: { name: string }[]) => { + mockCalls.push("runFixtures"); + return { + ok: true, + allTerminal: true, + results: requests.map((r) => ({ + name: r.name, + definitionId: 0, + buildId: 1, + url: "https://example/1", + status: "succeeded" as const, + result: "succeeded", + durationMs: 1, + terminalProven: true, + })), + }; + }), + }; +}); + +vi.mock("node:fs/promises", async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + mkdtemp: vi.fn(async () => "C:\\tmp\\compiler-smoke-xyz"), + readFile: vi.fn(async (path: string) => { + if (String(path).endsWith(".lock.yml")) return specificRunYaml(); + return "---\nname: fixture\n---\nBody.\n"; + }), + writeFile: vi.fn(async () => {}), + rm: vi.fn(async () => {}), + }; +}); + +beforeEach(() => { + mockCalls.length = 0; + vi.clearAllMocks(); +}); + +describe("compiler-smoke-e2e index.main (happy path)", () => { + it("checks artifact visibility before any git work, and deletes the ref before removing the worktree", async () => { + process.env = { ...process.env, ...baseEnv, VITEST: "true" }; + const { main } = await import("../index.js"); + const code = await main(); + expect(code).toBe(0); + + expect(mockCalls.indexOf("getArtifact")).toBeGreaterThanOrEqual(0); + expect(mockCalls.indexOf("getArtifact")).toBeLessThan(mockCalls.indexOf("verifyLocalCommit")); + expect(mockCalls.indexOf("verifyLocalCommit")).toBeLessThan(mockCalls.indexOf("createDetachedWorktree")); + expect(mockCalls.indexOf("createDetachedWorktree")).toBeLessThan(mockCalls.indexOf("compileAndCheck")); + expect(mockCalls.indexOf("compileAndCheck")).toBeLessThan(mockCalls.indexOf("worktreeChangedFiles")); + expect(mockCalls.indexOf("worktreeChangedFiles")).toBeLessThan(mockCalls.indexOf("commitAll")); + expect(mockCalls.indexOf("commitAll")).toBeLessThan(mockCalls.indexOf("pushCandidate")); + expect(mockCalls.indexOf("pushCandidate")).toBeLessThan(mockCalls.indexOf("verifyRemoteRef")); + expect(mockCalls.indexOf("verifyRemoteRef")).toBeLessThan(mockCalls.indexOf("runFixtures")); + + // Cleanup ordering: the remote candidate ref must be deleted BEFORE the + // local worktree is removed (never leave the ref hanging around). + expect(mockCalls.indexOf("deleteRemoteRef")).toBeGreaterThanOrEqual(0); + expect(mockCalls.indexOf("removeWorktree")).toBeGreaterThanOrEqual(0); + expect(mockCalls.indexOf("deleteRemoteRef")).toBeLessThan(mockCalls.indexOf("removeWorktree")); + }); +}); + +describe("compiler-smoke-e2e index.main (unexpected path guard)", () => { + it("refuses to push and never deletes a ref that was never pushed, but still removes the worktree", async () => { + const gitModule = await import("../git.js"); + vi.mocked(gitModule.worktreeChangedFiles).mockResolvedValueOnce([ + "tests/safe-outputs/canary.md", + "src/main.rs", // unexpected — must abort before any commit/push + ]); + + process.env = { ...process.env, ...baseEnv, VITEST: "true" }; + const { main } = await import("../index.js"); + const code = await main(); + expect(code).toBe(1); + + expect(mockCalls).not.toContain("commitAll"); + expect(mockCalls).not.toContain("pushCandidate"); + expect(mockCalls).not.toContain("deleteRemoteRef"); + expect(mockCalls).toContain("removeWorktree"); + }); +}); + +describe("compiler-smoke-e2e index.main (stageFixtures reads from the worktree, not BUILD_SOURCESDIRECTORY)", () => { + it("reads every fixture markdown source from the detached worktree — never from BUILD_SOURCESDIRECTORY (which may sit at a different commit when verifyLocalCommit falls back to the object-existence check)", async () => { + process.env = { ...process.env, ...baseEnv, VITEST: "true" }; + const { main } = await import("../index.js"); + const fsModule = await import("node:fs/promises"); + const code = await main(); + expect(code).toBe(0); + + const mdReadPaths = vi + .mocked(fsModule.readFile) + .mock.calls.map((call) => String(call[0])) + .filter((p) => p.endsWith(".md")); + expect(mdReadPaths.length).toBeGreaterThan(0); + for (const p of mdReadPaths) { + // The worktree lives under the mocked mkdtemp() result, never under + // BUILD_SOURCESDIRECTORY ("C:\repo"). + expect(p.startsWith("C:\\tmp\\compiler-smoke-xyz")).toBe(true); + expect(p.startsWith("C:\\repo")).toBe(false); + } + }); +}); + +describe("compiler-smoke-e2e index.main (PR base-ref regression — Fix #1)", () => { + it("never fetches BUILD_SOURCEBRANCH from the mirror for a GitHub PR build; the worktree is based on the local BUILD_SOURCEVERSION", async () => { + process.env = { + ...process.env, + ...baseEnv, + BUILD_SOURCEBRANCH: "refs/pull/123/merge", // does not exist on the ADO mirror repo + BUILD_SOURCEVERSION: "pr-head-sha", + VITEST: "true", + }; + const { main } = await import("../index.js"); + const gitModule = await import("../git.js"); + const code = await main(); + expect(code).toBe(0); + + // verifyLocalCommit must be asked to verify the LOCAL BUILD_SOURCEVERSION + // — never the PR ref. + expect(vi.mocked(gitModule.verifyLocalCommit).mock.calls[0]?.[0]).toMatchObject({ + cwd: "C:\\repo", + expectedSha: "pr-head-sha", + }); + // The worktree is created directly from that same local commit; it must + // never receive `refs/pull/123/merge` as the commitish. + const worktreeArgs = vi.mocked(gitModule.createDetachedWorktree).mock.calls[0]?.[0] as + | { commitish?: string } + | undefined; + expect(worktreeArgs?.commitish).toBe("pr-head-sha"); + expect(worktreeArgs?.commitish).not.toBe("refs/pull/123/merge"); + }); +}); + +describe("compiler-smoke-e2e index.main (unproven-terminal ref retention — Fix #3)", () => { + it("retains (does not delete) the candidate ref when runFixtures cannot prove every build reached a terminal state", async () => { + const runnerModule = await import("../runner.js"); + vi.mocked(runnerModule.runFixtures).mockResolvedValueOnce({ + ok: false, + allTerminal: false, + results: [ + { + name: "canary", + definitionId: 3001, + buildId: 1, + url: "https://example/1", + status: "failed", + message: "getBuild kept failing", + durationMs: 1, + terminalProven: false, + }, + ], + }); + + process.env = { ...process.env, ...baseEnv, VITEST: "true" }; + const { main } = await import("../index.js"); + const code = await main(); + expect(code).toBe(1); + + // The push itself succeeded, but the ref must be RETAINED, not deleted, + // because this run could not prove the build actually stopped. + expect(mockCalls).toContain("pushCandidate"); + expect(mockCalls).not.toContain("deleteRemoteRef"); + expect(mockCalls).toContain("removeWorktree"); + }); + + it("retains (does not delete) the candidate ref when runFixtures itself throws unexpectedly (never trusts the fail-closed default's absence of proof)", async () => { + const runnerModule = await import("../runner.js"); + vi.mocked(runnerModule.runFixtures).mockImplementationOnce(async () => { + mockCalls.push("runFixtures"); + throw new Error("runner crashed after queueing an unknown number of builds"); + }); + + process.env = { ...process.env, ...baseEnv, VITEST: "true" }; + const { main } = await import("../index.js"); + const code = await main(); + expect(code).toBe(1); + + // The push succeeded and runFixtures was entered, but because it threw + // instead of returning a proven outcome, `allChildrenTerminal` must + // still be `false` (its pre-call fail-closed value) — the ref must be + // retained, never deleted. + expect(mockCalls).toContain("pushCandidate"); + expect(mockCalls).toContain("runFixtures"); + expect(mockCalls).not.toContain("deleteRemoteRef"); + expect(mockCalls).toContain("removeWorktree"); + }); +}); diff --git a/scripts/ado-script/src/compiler-smoke-e2e/__tests__/process.test.ts b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/process.test.ts new file mode 100644 index 00000000..4f3ecafd --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/process.test.ts @@ -0,0 +1,122 @@ +import { describe, expect, it } from "vitest"; + +import { redact, safeSpawn, sleep, stripTraceEnv } from "../process.js"; + +const isWindows = process.platform === "win32"; +const nodeCmd = process.execPath; + +function nodeEval(script: string): string[] { + return ["-e", script]; +} + +describe("stripTraceEnv", () => { + it("removes GIT_TRACE / GIT_TRACE_CURL / GIT_CURL_VERBOSE", () => { + const env = stripTraceEnv({ + GIT_TRACE: "1", + GIT_TRACE_CURL: "1", + GIT_CURL_VERBOSE: "1", + PATH: "/usr/bin", + }); + expect(env.GIT_TRACE).toBeUndefined(); + expect(env.GIT_TRACE_CURL).toBeUndefined(); + expect(env.GIT_CURL_VERBOSE).toBeUndefined(); + expect(env.PATH).toBe("/usr/bin"); + }); + + it("does not mutate the input object", () => { + const input = { GIT_TRACE: "1" }; + stripTraceEnv(input); + expect(input.GIT_TRACE).toBe("1"); + }); +}); + +describe("redact", () => { + it("replaces every occurrence of a secret with ***", () => { + expect(redact("token=abc123 again abc123", ["abc123"])).toBe("token=*** again ***"); + }); + + it("ignores empty/undefined secrets", () => { + expect(redact("hello world", [undefined, "", "world"])).toBe("hello ***"); + }); + + it("redacts multiple distinct secrets", () => { + expect(redact("a=1 b=2", ["1", "2"])).toBe("a=*** b=***"); + }); + + it("is a no-op when no secrets are given", () => { + expect(redact("unchanged", [])).toBe("unchanged"); + }); +}); + +describe("safeSpawn", () => { + it("captures stdout and a zero exit status", async () => { + const outcome = await safeSpawn({ + cmd: nodeCmd, + args: nodeEval("process.stdout.write('hello')"), + timeoutMs: 10_000, + }); + expect(outcome.status).toBe(0); + expect(outcome.stdout).toBe("hello"); + expect(outcome.timedOut).toBe(false); + }); + + it("captures a non-zero exit status", async () => { + const outcome = await safeSpawn({ + cmd: nodeCmd, + args: nodeEval("process.exit(3)"), + timeoutMs: 10_000, + }); + expect(outcome.status).toBe(3); + }); + + it("kills a child that exceeds timeoutMs and reports timedOut", async () => { + const outcome = await safeSpawn({ + cmd: nodeCmd, + args: nodeEval("setTimeout(() => {}, 60000)"), + timeoutMs: 200, + }); + expect(outcome.timedOut).toBe(true); + }, 10_000); + + it("truncates stdout past maxOutputBytes and reports stdoutTruncated", async () => { + const outcome = await safeSpawn({ + cmd: nodeCmd, + args: nodeEval("process.stdout.write('x'.repeat(1000))"), + timeoutMs: 10_000, + maxOutputBytes: 10, + }); + expect(outcome.stdout.length).toBeLessThanOrEqual(10); + expect(outcome.stdoutTruncated).toBe(true); + }); + + it("strips ambient GIT_TRACE-family env vars from the child environment", async () => { + const probe = isWindows + ? "console.log(JSON.stringify({t: process.env.GIT_TRACE ?? null}))" + : "console.log(JSON.stringify({t: process.env.GIT_TRACE ?? null}))"; + const outcome = await safeSpawn({ + cmd: nodeCmd, + args: nodeEval(probe), + env: { GIT_TRACE: "1" }, + timeoutMs: 10_000, + }); + expect(JSON.parse(outcome.stdout.trim())).toEqual({ t: null }); + }); + + it("merges caller-provided env on top of process.env", async () => { + const outcome = await safeSpawn({ + cmd: nodeCmd, + args: nodeEval("process.stdout.write(process.env.MY_CUSTOM_VAR ?? '')"), + env: { MY_CUSTOM_VAR: "custom-value" }, + timeoutMs: 10_000, + }); + expect(outcome.stdout).toBe("custom-value"); + }); +}); + +describe("sleep", () => { + it("resolves after roughly the requested delay", async () => { + const start = Date.now(); + await sleep(20); + expect(Date.now() - start).toBeGreaterThanOrEqual(10); + }); +}); diff --git a/scripts/ado-script/src/compiler-smoke-e2e/__tests__/report.test.ts b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/report.test.ts new file mode 100644 index 00000000..9f61058c --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/report.test.ts @@ -0,0 +1,54 @@ +import { describe, expect, it } from "vitest"; + +import type { FixtureBuildResult } from "../runner.js"; +import { renderResultsTable } from "../report.js"; + +function result(overrides: Partial): FixtureBuildResult { + return { + name: "canary", + definitionId: 2601, + status: "succeeded", + durationMs: 12_345, + terminalProven: true, + ...overrides, + }; +} + +describe("renderResultsTable", () => { + it("renders a header row and one row per fixture", () => { + const table = renderResultsTable([ + result({ name: "canary", buildId: 1, url: "https://x/1", result: "succeeded" }), + result({ name: "azure-cli", definitionId: 2602, buildId: 2, url: "https://x/2", result: "succeeded" }), + ]); + const lines = table.split("\n"); + expect(lines[0]).toMatch(/fixture/); + expect(lines[0]).toMatch(/definition/); + expect(lines[0]).toMatch(/result/); + expect(table).toContain("canary"); + expect(table).toContain("azure-cli"); + expect(table).toContain("2601"); + expect(table).toContain("2602"); + }); + + it("preserves the caller's declaration order", () => { + const table = renderResultsTable([ + result({ name: "janitor", definitionId: 2604 }), + result({ name: "canary", definitionId: 2601 }), + ]); + const janitorIdx = table.indexOf("janitor"); + const canaryIdx = table.indexOf("canary"); + expect(janitorIdx).toBeGreaterThan(-1); + expect(canaryIdx).toBeGreaterThan(janitorIdx); + }); + + it("renders a '-' placeholder for missing buildId/url", () => { + const table = renderResultsTable([result({ status: "queue-failed", message: "definition disabled" })]); + expect(table).toContain("queue-failed"); + expect(table).toContain("definition disabled"); + }); + + it("shows the result alongside status when present", () => { + const table = renderResultsTable([result({ status: "succeeded", result: "succeeded" })]); + expect(table).toContain("succeeded (succeeded)"); + }); +}); diff --git a/scripts/ado-script/src/compiler-smoke-e2e/__tests__/runner.test.ts b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/runner.test.ts new file mode 100644 index 00000000..6b0601cc --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/runner.test.ts @@ -0,0 +1,594 @@ +import { describe, expect, it } from "vitest"; + +import type { FixtureBuildClient, FixtureBuildRequest } from "../runner.js"; +import { runFixtures } from "../runner.js"; + +interface FakeBuild { + status: string; + result?: string; + /** Optional identity override for a specific poll — omit to auto-match the request (see `makeFakeClient`'s default-fill behavior below). */ + definition?: { id?: number }; + sourceBranch?: string; + sourceVersion?: string; +} + +/** + * A fully scripted fake ADO Build client: queue results + a per-build status + * timeline. + * + * By default every polled build's identity (definition id, sourceBranch, + * sourceVersion) is auto-filled to exactly match what was requested — via + * `req()`'s `sourceBranch: "refs/heads/x"` / `sourceVersion: "sha"` defaults + * and a definitionId reverse-derived from `queueResults` — so tests that + * aren't specifically exercising the identity-mismatch check never need to + * care about it. A timeline entry can still explicitly set any of these + * fields (including to a wrong value, or omit them, via a hand-rolled + * `FixtureBuildClient` instead of this helper) to exercise the mismatch + * path. + */ +function makeFakeClient(opts: { + queueResults: Record; + /** For each queued build id, the sequence of statuses returned on successive getBuild() polls (last value repeats). */ + timelines: Record; + onCancel?: (buildId: number) => void; +}): { client: FixtureBuildClient; cancelled: number[] } { + const cancelled: number[] = []; + const pollCounts: Record = {}; + const definitionIdByBuildId = new Map(); + for (const [definitionIdStr, result] of Object.entries(opts.queueResults)) { + if (result.ok) definitionIdByBuildId.set(result.id, Number(definitionIdStr)); + } + const client: FixtureBuildClient = { + async queueBuild(definitionId) { + const result = opts.queueResults[definitionId]; + if (!result) throw new Error(`no queue result configured for definition ${definitionId}`); + if (!result.ok) throw new Error(result.error); + return { id: result.id }; + }, + async getBuild(buildId) { + const timeline = opts.timelines[buildId]; + if (!timeline || timeline.length === 0) throw new Error(`no timeline for build ${buildId}`); + const idx = pollCounts[buildId] ?? 0; + pollCounts[buildId] = idx + 1; + const entry = timeline[Math.min(idx, timeline.length - 1)]!; + const defaultIdentity = { + definition: { id: definitionIdByBuildId.get(buildId) }, + sourceBranch: "refs/heads/x", + sourceVersion: "sha", + }; + return { ...defaultIdentity, ...entry }; + }, + async cancelBuild(buildId) { + cancelled.push(buildId); + opts.onCancel?.(buildId); + }, + buildUrl(buildId) { + return `https://example/_build/results?buildId=${buildId}`; + }, + }; + return { client, cancelled }; +} + +function req(name: FixtureBuildRequest["name"], definitionId: number): FixtureBuildRequest { + return { name, definitionId, sourceBranch: "refs/heads/x", sourceVersion: "sha" }; +} + +const noopSleep = async (): Promise => {}; + +describe("runFixtures", () => { + it("succeeds when every fixture queues and completes successfully", async () => { + const { client } = makeFakeClient({ + queueResults: { + 1: { ok: true, id: 101 }, + 2: { ok: true, id: 102 }, + }, + timelines: { + 101: [{ status: "completed", result: "succeeded" }], + 102: [{ status: "completed", result: "succeeded" }], + }, + }); + const outcome = await runFixtures(client, [req("canary", 1), req("azure-cli", 2)], { + concurrency: 2, + timeoutMs: 10_000, + pollMs: 1, + log: () => {}, + sleepImpl: noopSleep, + }); + expect(outcome.ok).toBe(true); + expect(outcome.results.map((r) => r.status)).toEqual(["succeeded", "succeeded"]); + expect(outcome.allTerminal).toBe(true); + expect(outcome.results.every((r) => r.terminalProven)).toBe(true); + }); + + it("preserves declaration order in results regardless of completion order", async () => { + const { client } = makeFakeClient({ + queueResults: { 1: { ok: true, id: 201 }, 2: { ok: true, id: 202 } }, + timelines: { + // build 201 (first in declaration order) takes longer to complete than 202. + 201: [ + { status: "inProgress" }, + { status: "inProgress" }, + { status: "completed", result: "succeeded" }, + ], + 202: [{ status: "completed", result: "succeeded" }], + }, + }); + const outcome = await runFixtures(client, [req("canary", 1), req("azure-cli", 2)], { + concurrency: 2, + timeoutMs: 10_000, + pollMs: 1, + log: () => {}, + sleepImpl: noopSleep, + }); + expect(outcome.results.map((r) => r.name)).toEqual(["canary", "azure-cli"]); + expect(outcome.results.every((r) => r.status === "succeeded")).toBe(true); + }); + + it("a partial queue failure still polls and reports the successfully queued builds", async () => { + const { client } = makeFakeClient({ + queueResults: { + 1: { ok: false, error: "definition disabled" }, + 2: { ok: true, id: 302 }, + }, + timelines: { + 302: [{ status: "completed", result: "succeeded" }], + }, + }); + const outcome = await runFixtures(client, [req("canary", 1), req("azure-cli", 2)], { + concurrency: 2, + timeoutMs: 10_000, + pollMs: 1, + log: () => {}, + sleepImpl: noopSleep, + }); + expect(outcome.ok).toBe(false); + const canary = outcome.results.find((r) => r.name === "canary")!; + const azureCli = outcome.results.find((r) => r.name === "azure-cli")!; + expect(canary.status).toBe("queue-failed"); + expect(canary.message).toMatch(/definition disabled/); + expect(azureCli.status).toBe("succeeded"); + // A queueBuild error is ambiguous (ADO may have accepted the build + // before the client observed the failure), so it can never be treated + // as proof nothing was created — the ref must be retained. + expect(canary.terminalProven).toBe(false); + expect(outcome.allTerminal).toBe(false); + }); + + it("a failed build flips the shared abort flag and cancels sibling builds", async () => { + const timelines: Record = { + 401: [{ status: "completed", result: "failed" }], + // 402 never completes on its own; only cancellation (via cancelGraceMs) ends the poll. + 402: Array.from({ length: 50 }, () => ({ status: "inProgress" as const })), + }; + const { client, cancelled } = makeFakeClient({ + queueResults: { 1: { ok: true, id: 401 }, 2: { ok: true, id: 402 } }, + timelines, + }); + + const outcome = await runFixtures(client, [req("canary", 1), req("azure-cli", 2)], { + concurrency: 2, + timeoutMs: 10_000, + pollMs: 1, + log: () => {}, + sleepImpl: noopSleep, + cancelGraceMs: 5, + }); + expect(outcome.ok).toBe(false); + const canary = outcome.results.find((r) => r.name === "canary")!; + const azureCli = outcome.results.find((r) => r.name === "azure-cli")!; + expect(canary.status).toBe("failed"); + expect(azureCli.status === "canceled" || azureCli.status === "timed-out").toBe(true); + expect(cancelled).toContain(402); + }); + + it("a per-build timeout cancels that build and flips the shared abort flag for siblings", async () => { + const timelines: Record = { + 501: Array.from({ length: 1000 }, () => ({ status: "inProgress" })), + 502: Array.from({ length: 1000 }, () => ({ status: "inProgress" })), + }; + const { client } = makeFakeClient({ + queueResults: { 1: { ok: true, id: 501 }, 2: { ok: true, id: 502 } }, + timelines, + }); + const outcome = await runFixtures(client, [req("canary", 1), req("azure-cli", 2)], { + concurrency: 2, + timeoutMs: 5, + pollMs: 1, + log: () => {}, + sleepImpl: noopSleep, + cancelGraceMs: 5, + }); + expect(outcome.ok).toBe(false); + for (const r of outcome.results) { + expect(["timed-out", "canceled", "failed"]).toContain(r.status); + } + }); + + it("never returns with a build left non-terminal (waits out the cancel grace period)", async () => { + const timelines: Record = { + 601: [{ status: "completed", result: "failed" }], + // 602 never transitions to completed even after cancellation is requested. + 602: Array.from({ length: 1000 }, () => ({ status: "inProgress" })), + }; + const { client } = makeFakeClient({ + queueResults: { 1: { ok: true, id: 601 }, 2: { ok: true, id: 602 } }, + timelines, + }); + const outcome = await runFixtures(client, [req("canary", 1), req("azure-cli", 2)], { + concurrency: 2, + timeoutMs: 10_000, + pollMs: 1, + log: () => {}, + sleepImpl: noopSleep, + cancelGraceMs: 5, + }); + const azureCli = outcome.results.find((r) => r.name === "azure-cli")!; + expect(azureCli.status).toBe("timed-out"); + expect(azureCli.message).toMatch(/cancellation grace period/); + }); + + it("respects the configured concurrency (never more than N builds polled in parallel)", async () => { + let inFlight = 0; + let maxInFlight = 0; + const fixtureNames = ["canary", "azure-cli", "noop-target", "janitor"] as const; + const buildIds = [701, 702, 703, 704]; + + const client: FixtureBuildClient = { + async queueBuild(definitionId) { + return { id: buildIds[definitionId - 1]! }; + }, + async getBuild() { + inFlight++; + maxInFlight = Math.max(maxInFlight, inFlight); + await new Promise((r) => setTimeout(r, 5)); + inFlight--; + return { status: "completed", result: "succeeded" }; + }, + async cancelBuild() {}, + buildUrl: (id) => `https://example/${id}`, + }; + + const requests = fixtureNames.map((name, i) => req(name, i + 1)); + await runFixtures(client, requests, { + concurrency: 2, + timeoutMs: 10_000, + pollMs: 1, + log: () => {}, + }); + expect(maxInFlight).toBeLessThanOrEqual(2); + }); + + // ---- Fix #3: never claim a build is terminal without positive proof ---- + + it("marks allTerminal=false when getBuild keeps failing and never confirms a terminal state, even after cancellation", async () => { + const client: FixtureBuildClient = { + async queueBuild(definitionId) { + return { id: 800 + definitionId }; + }, + async getBuild() { + throw new Error("transient network error"); + }, + async cancelBuild() {}, + buildUrl: (id) => `https://example/${id}`, + }; + const outcome = await runFixtures(client, [req("canary", 1)], { + concurrency: 1, + timeoutMs: 10_000, + pollMs: 1, + log: () => {}, + sleepImpl: noopSleep, + cancelGraceMs: 5, + }); + expect(outcome.allTerminal).toBe(false); + const canary = outcome.results[0]!; + expect(canary.status).toBe("failed"); + expect(canary.terminalProven).toBe(false); + expect(canary.message).toMatch(/never confirmed a terminal state/); + }); + + it("recovers from a transient getBuild error: once a later call confirms completion, terminalProven is true", async () => { + let calls = 0; + const client: FixtureBuildClient = { + async queueBuild(definitionId) { + return { id: 810 + definitionId }; + }, + async getBuild() { + calls++; + if (calls <= 2) throw new Error("transient network error"); + return { + status: "completed", + result: "canceled", + definition: { id: 1 }, + sourceBranch: "refs/heads/x", + sourceVersion: "sha", + }; + }, + async cancelBuild() {}, + buildUrl: (id) => `https://example/${id}`, + }; + const outcome = await runFixtures(client, [req("canary", 1)], { + concurrency: 1, + timeoutMs: 10_000, + pollMs: 1, + log: () => {}, + sleepImpl: noopSleep, + cancelGraceMs: 5_000, + }); + const canary = outcome.results[0]!; + expect(canary.terminalProven).toBe(true); + expect(canary.status).toBe("canceled"); + expect(outcome.allTerminal).toBe(true); + }); + + it("marks allTerminal=false when cancellation is requested (grace period) but the build never actually stops, even if cancelBuild itself throws", async () => { + const client: FixtureBuildClient = { + async queueBuild(definitionId) { + return { id: 820 + definitionId }; + }, + async getBuild() { + return { status: "inProgress" }; + }, + async cancelBuild() { + throw new Error("cancel API rejected"); + }, + buildUrl: (id) => `https://example/${id}`, + }; + const outcome = await runFixtures(client, [req("canary", 1)], { + concurrency: 1, + timeoutMs: 5, + pollMs: 1, + log: () => {}, + sleepImpl: noopSleep, + cancelGraceMs: 5, + }); + const canary = outcome.results[0]!; + expect(canary.status).toBe("timed-out"); + expect(canary.terminalProven).toBe(false); + expect(outcome.allTerminal).toBe(false); + }); + + // ---- Fix #4: verify the queued build's identity matches the request ---- + + it("treats a definitionId/sourceBranch/sourceVersion mismatch as a failure and flips the shared abort flag", async () => { + const cancelled: number[] = []; + const client: FixtureBuildClient = { + async queueBuild(definitionId) { + return { id: 900 + definitionId }; + }, + async getBuild(buildId) { + if (buildId === 901) { + // Wrong sourceVersion — as if ADO (or a racing queue) somehow + // built a different commit than what was requested. + return { + status: "completed", + result: "succeeded", + definition: { id: 1 }, + sourceBranch: "refs/heads/x", + sourceVersion: "not-the-requested-sha", + }; + } + // 902 (azure-cli) has correct, matching identity — isolates the + // mismatch assertion below to the canary build only; it is only + // cancelled because canary's mismatch flips the shared abort flag. + return { + status: "inProgress", + definition: { id: 2 }, + sourceBranch: "refs/heads/x", + sourceVersion: "sha", + }; + }, + async cancelBuild(buildId) { + cancelled.push(buildId); + }, + buildUrl: (id) => `https://example/${id}`, + }; + const outcome = await runFixtures(client, [req("canary", 1), req("azure-cli", 2)], { + concurrency: 2, + timeoutMs: 10_000, + pollMs: 1, + log: () => {}, + sleepImpl: noopSleep, + cancelGraceMs: 5, + }); + expect(outcome.ok).toBe(false); + const canary = outcome.results.find((r) => r.name === "canary")!; + expect(canary.status).toBe("failed"); + expect(canary.terminalProven).toBe(true); + expect(canary.message).toMatch(/does not match the requested queue parameters/); + expect(canary.message).toMatch(/sourceVersion/); + // The mismatch must also cancel the sibling build. + expect(cancelled).toContain(902); + }); + + it("treats a wrong definition id as a mismatch even when sourceBranch/sourceVersion are correct", async () => { + const client: FixtureBuildClient = { + async queueBuild(definitionId) { + return { id: 910 + definitionId }; + }, + async getBuild() { + return { + status: "completed", + result: "succeeded", + definition: { id: 999 }, // wrong — requested definition was 1 + sourceBranch: "refs/heads/x", + sourceVersion: "sha", + }; + }, + async cancelBuild() {}, + buildUrl: (id) => `https://example/${id}`, + }; + const outcome = await runFixtures(client, [req("canary", 1)], { + concurrency: 1, + timeoutMs: 10_000, + pollMs: 1, + log: () => {}, + sleepImpl: noopSleep, + cancelGraceMs: 5, + }); + const canary = outcome.results[0]!; + expect(canary.status).toBe("failed"); + expect(canary.message).toMatch(/does not match the requested queue parameters/); + expect(canary.message).toMatch(/definition id 999/); + }); + + it("treats a wrong sourceBranch as a mismatch even when definition id/sourceVersion are correct", async () => { + const client: FixtureBuildClient = { + async queueBuild(definitionId) { + return { id: 920 + definitionId }; + }, + async getBuild() { + return { + status: "completed", + result: "succeeded", + definition: { id: 1 }, + sourceBranch: "refs/heads/some-other-branch", // wrong + sourceVersion: "sha", + }; + }, + async cancelBuild() {}, + buildUrl: (id) => `https://example/${id}`, + }; + const outcome = await runFixtures(client, [req("canary", 1)], { + concurrency: 1, + timeoutMs: 10_000, + pollMs: 1, + log: () => {}, + sleepImpl: noopSleep, + cancelGraceMs: 5, + }); + const canary = outcome.results[0]!; + expect(canary.status).toBe("failed"); + expect(canary.message).toMatch(/does not match the requested queue parameters/); + expect(canary.message).toMatch(/sourceBranch/); + }); + + it("treats entirely missing identity fields as a mismatch — missing data is never treated as proof of the requested build", async () => { + const client: FixtureBuildClient = { + async queueBuild(definitionId) { + return { id: 930 + definitionId }; + }, + async getBuild() { + // No `definition`, `sourceBranch`, or `sourceVersion` at all. + return { status: "completed", result: "succeeded" }; + }, + async cancelBuild() {}, + buildUrl: (id) => `https://example/${id}`, + }; + const outcome = await runFixtures(client, [req("canary", 1)], { + concurrency: 1, + timeoutMs: 10_000, + pollMs: 1, + log: () => {}, + sleepImpl: noopSleep, + cancelGraceMs: 5, + }); + expect(outcome.ok).toBe(false); + const canary = outcome.results[0]!; + expect(canary.status).toBe("failed"); + // Even though ADO reported "succeeded", a build with unverifiable + // identity must never be trusted or reported as succeeded. + expect(canary.result).not.toBe(undefined); + expect(canary.message).toMatch(/does not match the requested queue parameters/); + expect(canary.message).toMatch(/definition id /); + expect(canary.message).toMatch(/sourceBranch ''/); + expect(canary.message).toMatch(/sourceVersion ''/); + // The identity check runs BEFORE the terminal-state result is trusted, + // so this is still positively proven terminal (ADO really did report + // "completed") — just correctly reported as a failure, never a + // false "succeeded". + expect(canary.terminalProven).toBe(true); + }); + + // ---- Fix #3 (queue-failure abort propagation) ---- + + it("a queue failure immediately signals the shared abort flag so an already-queued sibling is cancelled rather than run to completion", async () => { + const cancelled: number[] = []; + const client: FixtureBuildClient = { + async queueBuild(definitionId) { + if (definitionId === 1) throw new Error("definition disabled"); + return { id: 900 + definitionId }; + }, + async getBuild(buildId) { + if (cancelled.includes(buildId)) { + return { + status: "completed", + result: "canceled", + definition: { id: buildId - 900 }, + sourceBranch: "refs/heads/x", + sourceVersion: "sha", + }; + } + return { + status: "inProgress", + definition: { id: buildId - 900 }, + sourceBranch: "refs/heads/x", + sourceVersion: "sha", + }; + }, + async cancelBuild(buildId) { + cancelled.push(buildId); + }, + buildUrl: (id) => `https://example/${id}`, + }; + const outcome = await runFixtures(client, [req("canary", 1), req("azure-cli", 2)], { + concurrency: 2, + timeoutMs: 10_000, + pollMs: 1, + log: () => {}, + sleepImpl: noopSleep, + cancelGraceMs: 1_000, + }); + expect(outcome.ok).toBe(false); + expect(cancelled).toContain(902); + const azureCli = outcome.results.find((r) => r.name === "azure-cli")!; + expect(azureCli.status).toBe("canceled"); + expect(azureCli.terminalProven).toBe(true); + const canary = outcome.results.find((r) => r.name === "canary")!; + expect(canary.status).toBe("queue-failed"); + // Ambiguous queue failure: never proven, so the overall run can't + // claim every build is terminal even though the sibling was cancelled + // and confirmed. + expect(canary.terminalProven).toBe(false); + expect(outcome.allTerminal).toBe(false); + }); + + // ---- Fix #6: concurrent queueing ---- + + it("queues every fixture concurrently rather than one at a time, while preserving declaration-order results", async () => { + const startOrder: number[] = []; + const resolveOrder: number[] = []; + const client: FixtureBuildClient = { + async queueBuild(definitionId) { + startOrder.push(definitionId); + // definition 1 is deliberately the slowest to resolve. If queueing + // were sequential, definitions 2/3 could not even START until 1 + // resolved — so under sequential queueing resolveOrder would still + // be [1, 2, 3]. Under concurrent queueing, 2 and 3 (fast) resolve + // before 1 (slow) because all three started at once. + const delayMs = definitionId === 1 ? 20 : 1; + await new Promise((r) => setTimeout(r, delayMs)); + resolveOrder.push(definitionId); + return { id: 1000 + definitionId }; + }, + async getBuild(buildId) { + return { + status: "completed", + result: "succeeded", + definition: { id: buildId - 1000 }, + sourceBranch: "refs/heads/x", + sourceVersion: "sha", + }; + }, + async cancelBuild() {}, + buildUrl: (id) => `https://example/${id}`, + }; + const outcome = await runFixtures( + client, + [req("canary", 1), req("azure-cli", 2), req("noop-target", 3)], + { concurrency: 3, timeoutMs: 10_000, pollMs: 1, log: () => {} }, + ); + expect(resolveOrder.indexOf(1)).toBe(2); // definition 1 resolves LAST despite being declared first + expect(outcome.results.map((r) => r.name)).toEqual(["canary", "azure-cli", "noop-target"]); + expect(outcome.results.every((r) => r.status === "succeeded")).toBe(true); + }); +}); diff --git a/scripts/ado-script/src/compiler-smoke-e2e/__tests__/source.test.ts b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/source.test.ts new file mode 100644 index 00000000..9a14b6f1 --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/source.test.ts @@ -0,0 +1,101 @@ +import { describe, expect, it } from "vitest"; + +import { injectPipelineArtifact, splitFrontMatter } from "../source.js"; + +const VALUES = { + project: "AgentPlayground", + definitionId: 2560, + runId: 630001, + artifact: "ado-aw-candidate", +}; + +describe("splitFrontMatter", () => { + it("splits the first --- front-matter block from the body", () => { + const md = "---\nname: x\n---\n# Hello\n\nBody text.\n"; + const { yamlText, body } = splitFrontMatter(md); + expect(yamlText).toBe("name: x"); + expect(body).toBe("# Hello\n\nBody text.\n"); + }); + + it("throws when there is no leading front-matter block", () => { + expect(() => splitFrontMatter("# Just a heading\n")).toThrow(/front-matter/); + }); + + it("only matches the FIRST closing delimiter, never a later one inside the body", () => { + const md = "---\nname: x\n---\nSome body\n---\nnot-frontmatter: true\n---\nmore\n"; + const { yamlText, body } = splitFrontMatter(md); + expect(yamlText).toBe("name: x"); + expect(body).toBe("Some body\n---\nnot-frontmatter: true\n---\nmore\n"); + }); +}); + +describe("injectPipelineArtifact", () => { + it("injects literal project/definition-id/run-id/artifact under supply-chain.pipeline-artifact", () => { + const md = "---\nname: canary\non:\n schedule:\n - cron: '0 3 * * *'\n---\nBody unchanged.\n"; + const out = injectPipelineArtifact(md, VALUES); + const { yamlText, body } = splitFrontMatter(out); + expect(yamlText).toContain("supply-chain:"); + expect(yamlText).toContain("pipeline-artifact:"); + expect(yamlText).toContain("project: AgentPlayground"); + expect(yamlText).toContain("definition-id: 2560"); + expect(yamlText).toContain("run-id: 630001"); + expect(yamlText).toContain("artifact: ado-aw-candidate"); + expect(body).toBe("Body unchanged.\n"); + }); + + it("preserves the markdown body byte-for-byte, including trailing whitespace quirks", () => { + const body = "# Title\n\n- a\n- b\n\ntrailing spaces \n\n\nextra blank lines\n"; + const md = `---\nname: x\n---\n${body}`; + const out = injectPipelineArtifact(md, VALUES); + expect(out.endsWith(body)).toBe(true); + }); + + it("removes on.schedule but preserves other 'on' keys", () => { + const md = "---\nname: x\non:\n schedule:\n - cron: '0 3 * * *'\n workflow_dispatch: {}\n---\nBody.\n"; + const out = injectPipelineArtifact(md, VALUES); + const { yamlText } = splitFrontMatter(out); + expect(yamlText).not.toContain("schedule"); + expect(yamlText).toContain("workflow_dispatch"); + expect(yamlText).toContain("on:"); + }); + + it("removes the 'on' key entirely once schedule was its only member", () => { + const md = "---\nname: x\non:\n schedule:\n - cron: '0 3 * * *'\n---\nBody.\n"; + const out = injectPipelineArtifact(md, VALUES); + const { yamlText } = splitFrontMatter(out); + expect(yamlText).not.toMatch(/^on:/m); + }); + + it("is a no-op with respect to 'on' when there is no 'on' block at all", () => { + const md = "---\nname: noop-target\n---\nBody.\n"; + const out = injectPipelineArtifact(md, VALUES); + const { yamlText } = splitFrontMatter(out); + expect(yamlText).not.toMatch(/^on:/m); + expect(yamlText).toContain("supply-chain:"); + }); + + it("preserves an existing supply-chain.registry untouched", () => { + const md = + "---\nname: x\nsupply-chain:\n registry: my-registry\n---\nBody.\n"; + const out = injectPipelineArtifact(md, VALUES); + const { yamlText } = splitFrontMatter(out); + expect(yamlText).toContain("registry: my-registry"); + expect(yamlText).toContain("pipeline-artifact:"); + }); + + it("rejects a fixture that already defines supply-chain.feed", () => { + const md = "---\nname: x\nsupply-chain:\n feed: my-feed\n---\nBody.\n"; + expect(() => injectPipelineArtifact(md, VALUES)).toThrow(/supply-chain\.feed/); + }); + + it("rejects a fixture that already defines supply-chain.pipeline-artifact", () => { + const md = + "---\nname: x\nsupply-chain:\n pipeline-artifact:\n project: Other\n definition-id: 1\n run-id: 1\n artifact: a\n---\nBody.\n"; + expect(() => injectPipelineArtifact(md, VALUES)).toThrow(/pipeline-artifact/); + }); + + it("throws on malformed YAML front matter", () => { + const md = "---\nname: [unterminated\n---\nBody.\n"; + expect(() => injectPipelineArtifact(md, VALUES)).toThrow(); + }); +}); diff --git a/scripts/ado-script/src/compiler-smoke-e2e/__tests__/stale.test.ts b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/stale.test.ts new file mode 100644 index 00000000..c9ec6969 --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/__tests__/stale.test.ts @@ -0,0 +1,252 @@ +import { describe, expect, it } from "vitest"; + +import type { RemoteRef } from "../git.js"; +import { scanStaleRefs, type StaleScanBuild, type StaleScanClient } from "../stale.js"; + +const NOW = new Date("2024-06-01T00:00:00Z").getTime(); +const HOUR = 3_600_000; +const CHILD_DEFINITION_IDS = [901, 902, 903]; + +function ref(buildId: number): RemoteRef { + return { ref: `refs/heads/ado-aw-smoke-candidate/${buildId}`, sha: `sha-${buildId}` }; +} + +interface ClientOpts { + /** child (definitionId, branch) -> builds; defaults to "no builds found" (i.e. safely deletable) for every child */ + childBuilds?: Record; + /** definitionId that should throw when queried for child builds, regardless of branch */ + childLookupErrorFor?: number; +} + +function client(builds: Record, opts: ClientOpts = {}): StaleScanClient { + return { + async getBuild(buildId) { + const b = builds[buildId]; + if (!b) throw new Error(`no such build ${buildId}`); + return b; + }, + async listBuildsForDefinitionBranch(definitionId, branch) { + if (opts.childLookupErrorFor === definitionId) { + throw new Error(`child lookup failed for definition ${definitionId}`); + } + return opts.childBuilds?.[`${definitionId}:${branch}`] ?? []; + }, + }; +} + +const baseOpts = { + baseRef: "refs/heads/main", + ownRef: "refs/heads/ado-aw-smoke-candidate/999", + definitionId: 42, + childDefinitionIds: CHILD_DEFINITION_IDS, + staleRefHours: 24, +}; + +describe("scanStaleRefs", () => { + it("marks a completed, own-definition, old-enough build as eligible when no child builds are found", async () => { + const decisions = await scanStaleRefs({ + ...baseOpts, + refs: [ref(1)], + client: client({ + 1: { status: "completed", result: "succeeded", definition: { id: 42 }, finishTime: new Date(NOW - 30 * HOUR).toISOString() }, + }), + now: () => NOW, + }); + expect(decisions).toEqual([ + expect.objectContaining({ ref: ref(1).ref, outcome: "eligible" }), + ]); + }); + + it("marks a too-recently-completed build as too-recent, not eligible", async () => { + const decisions = await scanStaleRefs({ + ...baseOpts, + refs: [ref(2)], + client: client({ + 2: { status: "completed", result: "succeeded", definition: { id: 42 }, finishTime: new Date(NOW - 2 * HOUR).toISOString() }, + }), + now: () => NOW, + }); + expect(decisions[0]?.outcome).toBe("too-recent"); + }); + + it("marks a still-in-progress build as active, never eligible", async () => { + const decisions = await scanStaleRefs({ + ...baseOpts, + refs: [ref(3)], + client: client({ + 3: { status: "inProgress", definition: { id: 42 } }, + }), + now: () => NOW, + }); + expect(decisions[0]?.outcome).toBe("active"); + }); + + it("fails closed (ambiguous) for an unparseable ref name", async () => { + const decisions = await scanStaleRefs({ + ...baseOpts, + refs: [{ ref: "refs/heads/ado-aw-smoke-candidate/not-a-number", sha: "x" }], + client: client({}), + now: () => NOW, + }); + expect(decisions[0]?.outcome).toBe("ambiguous"); + }); + + it("fails closed (ambiguous) when the build lookup throws", async () => { + const decisions = await scanStaleRefs({ + ...baseOpts, + refs: [ref(4)], + client: { + getBuild: async () => { + throw new Error("network error"); + }, + listBuildsForDefinitionBranch: async () => [], + }, + now: () => NOW, + }); + expect(decisions[0]?.outcome).toBe("ambiguous"); + expect(decisions[0]?.reason).toMatch(/network error/); + }); + + it("fails closed (ambiguous) when the build belongs to a different definition", async () => { + const decisions = await scanStaleRefs({ + ...baseOpts, + refs: [ref(5)], + client: client({ + 5: { status: "completed", result: "succeeded", definition: { id: 43 }, finishTime: new Date(NOW - 30 * HOUR).toISOString() }, + }), + now: () => NOW, + }); + expect(decisions[0]?.outcome).toBe("ambiguous"); + expect(decisions[0]?.reason).toMatch(/definition 43/); + }); + + it("fails closed (ambiguous) when a completed build has no usable timestamp", async () => { + const decisions = await scanStaleRefs({ + ...baseOpts, + refs: [ref(6)], + client: client({ + 6: { status: "completed", result: "succeeded", definition: { id: 42 } }, + }), + now: () => NOW, + }); + expect(decisions[0]?.outcome).toBe("ambiguous"); + }); + + it("never treats the base ref or this run's own ref as a candidate to evaluate", async () => { + const decisions = await scanStaleRefs({ + ...baseOpts, + refs: [ + { ref: "refs/heads/main", sha: "base" }, + { ref: "refs/heads/ado-aw-smoke-candidate/999", sha: "own" }, + ], + client: client({}), + now: () => NOW, + }); + expect(decisions).toEqual([]); + }); + + it("respects a custom staleRefHours threshold at the boundary", async () => { + const builds = client({ + 7: { status: "completed", result: "succeeded", definition: { id: 42 }, finishTime: new Date(NOW - 7 * HOUR).toISOString() }, + }); + const eligible = await scanStaleRefs({ + ...baseOpts, + refs: [ref(7)], + staleRefHours: 6, + client: builds, + now: () => NOW, + }); + expect(eligible[0]?.outcome).toBe("eligible"); + + const notYet = await scanStaleRefs({ + ...baseOpts, + refs: [ref(7)], + staleRefHours: 8, + client: builds, + now: () => NOW, + }); + expect(notYet[0]?.outcome).toBe("too-recent"); + }); + + // ---- Fix #5: an old, terminal orchestrator build does NOT prove its + // queued fixture ("child") builds have also finished ---- + + it("marks a ref as active (not eligible) when a fixture/child definition still has a non-completed build on that branch", async () => { + const decisions = await scanStaleRefs({ + ...baseOpts, + refs: [ref(8)], + client: client( + { + 8: { status: "completed", result: "succeeded", definition: { id: 42 }, finishTime: new Date(NOW - 30 * HOUR).toISOString() }, + }, + { + childBuilds: { + [`902:${ref(8).ref}`]: [{ status: "inProgress" }], + }, + }, + ), + now: () => NOW, + }); + expect(decisions[0]?.outcome).toBe("active"); + expect(decisions[0]?.reason).toMatch(/902/); + }); + + it("fails closed (ambiguous) when a child-definition build lookup throws, even if the parent is old and completed", async () => { + const decisions = await scanStaleRefs({ + ...baseOpts, + refs: [ref(9)], + client: client( + { + 9: { status: "completed", result: "succeeded", definition: { id: 42 }, finishTime: new Date(NOW - 30 * HOUR).toISOString() }, + }, + { childLookupErrorFor: 903 }, + ), + now: () => NOW, + }); + expect(decisions[0]?.outcome).toBe("ambiguous"); + expect(decisions[0]?.reason).toMatch(/903/); + }); + + it("is eligible only once every fixed child definition has zero non-completed builds on the exact branch", async () => { + const decisions = await scanStaleRefs({ + ...baseOpts, + refs: [ref(10)], + client: client( + { + 10: { status: "completed", result: "succeeded", definition: { id: 42 }, finishTime: new Date(NOW - 30 * HOUR).toISOString() }, + }, + { + childBuilds: { + [`901:${ref(10).ref}`]: [{ status: "completed", result: "succeeded" }], + [`902:${ref(10).ref}`]: [{ status: "completed", result: "canceled" }], + [`903:${ref(10).ref}`]: [], + }, + }, + ), + now: () => NOW, + }); + expect(decisions[0]?.outcome).toBe("eligible"); + }); + + it("classifies an abruptly-cancelled orchestrator that completes while a child keeps running as active, never eligible", async () => { + // Regression for the exact scenario in Fix #5: the parent (orchestrator) + // build is 'completed' (e.g. cancelled) and old, but one of its queued + // fixture children is still 'inProgress' on the same branch. + const decisions = await scanStaleRefs({ + ...baseOpts, + refs: [ref(11)], + client: client( + { + 11: { status: "completed", result: "canceled", definition: { id: 42 }, finishTime: new Date(NOW - 48 * HOUR).toISOString() }, + }, + { + childBuilds: { + [`901:${ref(11).ref}`]: [{ status: "inProgress" }], + }, + }, + ), + now: () => NOW, + }); + expect(decisions[0]?.outcome).toBe("active"); + }); +}); diff --git a/scripts/ado-script/src/compiler-smoke-e2e/ado-rest.ts b/scripts/ado-script/src/compiler-smoke-e2e/ado-rest.ts new file mode 100644 index 00000000..e0dad3da --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/ado-rest.ts @@ -0,0 +1,206 @@ +/** + * Minimal, self-contained Azure DevOps Build REST client for the + * deterministic compiler-smoke E2E harness. + * + * Uses the global `fetch` (Node 20+) with Basic auth (empty user + token) — + * same posture as `executor-e2e/ado-rest.ts` and `trigger-e2e`. Kept + * self-contained (rather than importing the sibling harness's client) so + * this directory has no cross-harness coupling; `fetchImpl`/`sleepImpl` are + * injectable so every caller is testable without live network access. + * + * Test-harness module; not shipped in `ado-script.zip`. + */ +import { redact } from "./process.js"; +import { sleep as defaultSleep } from "./process.js"; + +export interface AdoRestOptions { + orgUrl: string; + project: string; + token: string; + log?: (msg: string) => void; + timeoutMs?: number; + fetchImpl?: typeof fetch; + sleepImpl?: (ms: number) => Promise; +} + +export interface BuildSummary { + id: number; + status?: string; + result?: string; + definition?: { id?: number }; + sourceBranch?: string; + sourceVersion?: string; + queueTime?: string; + finishTime?: string; +} + +export interface ArtifactInfo { + name: string; + resource?: { downloadUrl?: string; type?: string }; +} + +const DEFAULT_ARTIFACT_RETRIES = 5; +const DEFAULT_ARTIFACT_RETRY_DELAY_MS = 5_000; + +export class AdoRest { + private readonly base: string; + private readonly project: string; + private readonly authHeader: string; + private readonly log: (msg: string) => void; + private readonly timeoutMs: number; + private readonly fetchImpl: typeof fetch; + private readonly sleepImpl: (ms: number) => Promise; + + constructor(opts: AdoRestOptions) { + this.base = opts.orgUrl.replace(/\/+$/, ""); + this.project = opts.project; + this.authHeader = "Basic " + Buffer.from(":" + opts.token).toString("base64"); + this.log = opts.log ?? (() => {}); + this.timeoutMs = opts.timeoutMs ?? 30_000; + this.fetchImpl = opts.fetchImpl ?? fetch; + this.sleepImpl = opts.sleepImpl ?? defaultSleep; + } + + private static seg(value: string): string { + return encodeURIComponent(value); + } + + private projPath(rest: string): string { + return `${this.base}/${AdoRest.seg(this.project)}/${rest}`; + } + + private async request( + path: string, + opts: { method?: string; body?: unknown; allow404?: boolean } = {}, + ): Promise { + const headers: Record = { + Authorization: this.authHeader, + Accept: "application/json", + }; + let body: string | undefined; + if (opts.body !== undefined) { + body = JSON.stringify(opts.body); + headers["Content-Type"] = "application/json"; + } + const res = await this.fetchImpl(path, { + method: opts.method ?? "GET", + headers, + body, + signal: AbortSignal.timeout(this.timeoutMs), + }); + if (res.status === 404 && opts.allow404) return undefined; + if (!res.ok) { + const text = await res.text().catch(() => ""); + throw new Error(`ADO ${opts.method ?? "GET"} ${path} -> HTTP ${res.status}: ${text}`); + } + if (res.status === 204) return undefined; + const text = await res.text(); + if (!text) return undefined; + return JSON.parse(text) as T; + } + + /** Build a human-facing URL for a build (used in the final results table). */ + buildUrl(buildId: number): string { + return `${this.base}/${AdoRest.seg(this.project)}/_build/results?buildId=${buildId}`; + } + + /** + * Confirm the producer build's configured artifact is visible before any + * source/git work begins. Bounded retry: ADO can take a few seconds to + * index a just-published artifact. + */ + async getArtifact( + buildId: number, + artifactName: string, + opts: { retries?: number; retryDelayMs?: number } = {}, + ): Promise { + const retries = opts.retries ?? DEFAULT_ARTIFACT_RETRIES; + const retryDelayMs = opts.retryDelayMs ?? DEFAULT_ARTIFACT_RETRY_DELAY_MS; + const path = this.projPath( + `_apis/build/builds/${buildId}/artifacts?artifactName=${AdoRest.seg(artifactName)}&api-version=7.1`, + ); + let lastErr: unknown; + for (let attempt = 1; attempt <= retries; attempt++) { + try { + const res = await this.request(path, { allow404: true }); + if (res) return res; + lastErr = new Error( + `artifact '${artifactName}' not found on build #${buildId} (attempt ${attempt}/${retries})`, + ); + } catch (err) { + lastErr = err; + } + if (attempt < retries) { + this.log( + `[artifact-visibility] attempt ${attempt}/${retries} failed, retrying in ${retryDelayMs}ms: ${ + (lastErr as Error).message + }`, + ); + await this.sleepImpl(retryDelayMs); + } + } + throw new Error( + `artifact '${artifactName}' not visible on build #${buildId} in project '${this.project}' after ${retries} attempts: ${ + (lastErr as Error)?.message ?? "unknown error" + }`, + ); + } + + async getBuild(buildId: number): Promise { + const path = this.projPath(`_apis/build/builds/${buildId}?api-version=7.1`); + const res = await this.request(path); + if (!res) throw new Error(`getBuild(${buildId}) returned no body`); + return res; + } + + async cancelBuild(buildId: number): Promise { + const path = this.projPath(`_apis/build/builds/${buildId}?api-version=7.1`); + await this.request(path, { method: "PATCH", body: { status: "cancelling" } }); + } + + /** + * Queue a build of `definitionId`, pointed at the staged candidate branch + * + exact commit. Both are always supplied (never sourceBranch alone) so + * a slow/racing ref update on the mirror repo can never cause ADO to + * silently build the wrong commit. + */ + async queueBuild( + definitionId: number, + opts: { sourceBranch: string; sourceVersion: string }, + ): Promise<{ id: number }> { + const path = this.projPath(`_apis/build/builds?api-version=7.1`); + const body = { + definition: { id: definitionId }, + sourceBranch: opts.sourceBranch, + sourceVersion: opts.sourceVersion, + }; + const res = await this.request<{ id: number }>(path, { method: "POST", body }); + if (!res) throw new Error(`queueBuild(${definitionId}) returned no body`); + return res; + } + + /** + * List every build of `definitionId` on the exact `branch` (a full ref, + * e.g. `refs/heads/ado-aw-smoke-candidate/123`), regardless of status. + * + * Deliberately queries a single definition + exact branch and inspects + * each build's own `status` client-side, rather than asking ADO's + * `statusFilter` for a comma-separated set of "still running" states — + * whether that filter reliably matches every non-terminal status across + * ADO Build REST versions is not something this harness can assume. + * Used by the stale-ref scanner to prove NO fixture child build is still + * active on a candidate branch before it is deleted. + */ + async listBuildsForDefinitionBranch(definitionId: number, branch: string): Promise { + const path = this.projPath( + `_apis/build/builds?definitions=${definitionId}&branchName=${AdoRest.seg(branch)}&api-version=7.1&$top=50`, + ); + const res = await this.request<{ value?: BuildSummary[] }>(path); + return res?.value ?? []; + } +} + +/** Redact the ADO token from a diagnostic string before logging/reporting. */ +export function redactToken(text: string, token: string | undefined): string { + return redact(text, [token]); +} diff --git a/scripts/ado-script/src/compiler-smoke-e2e/assertions.ts b/scripts/ado-script/src/compiler-smoke-e2e/assertions.ts new file mode 100644 index 00000000..0a107d47 --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/assertions.ts @@ -0,0 +1,95 @@ +/** + * Post-compile assertions run against each fixture's freshly regenerated + * `*.lock.yml` inside the detached worktree, before anything is committed or + * pushed. + * + * Test-harness module; not shipped in `ado-script.zip`. + */ +import { parseAllDocuments } from "yaml"; + +/** + * Release URLs the candidate lane must never reference — the whole point of + * pinning `supply-chain.pipeline-artifact` is to source every binary + * (compiler, AWF, ado-script) from the current build's own artifact instead + * of a public release. + */ +const FORBIDDEN_URL_SNIPPETS = [ + "github.com/githubnext/ado-aw/releases", + "github.com/github/gh-aw-firewall/releases", +] as const; + +/** Throws if the compiled YAML still references a public release download URL. */ +export function assertNoForbiddenReleaseUrls(yamlText: string, label: string): void { + for (const snippet of FORBIDDEN_URL_SNIPPETS) { + if (yamlText.includes(snippet)) { + throw new Error( + `${label}: compiled pipeline still references a release URL ('${snippet}') — the candidate lane must source binaries exclusively from the pinned pipeline artifact`, + ); + } + } +} + +export interface ExpectedPipelineArtifact { + readonly project: string; + readonly pipeline: string; + readonly runId: string; + readonly artifact: string; +} + +function collectDownloadPipelineArtifactSteps(node: unknown, out: Record[]): void { + if (Array.isArray(node)) { + for (const item of node) collectDownloadPipelineArtifactSteps(item, out); + return; + } + if (node && typeof node === "object") { + const obj = node as Record; + if (typeof obj.task === "string" && obj.task.startsWith("DownloadPipelineArtifact")) { + out.push(obj); + } + for (const value of Object.values(obj)) { + collectDownloadPipelineArtifactSteps(value, out); + } + } +} + +/** + * Throws unless every `DownloadPipelineArtifact` "specific run" step in the + * compiled YAML carries exactly the expected project/pipeline/runId/artifact + * inputs. Throws if no such step exists at all (the transform is a no-op if + * the compiler silently dropped the pinned source). + */ +export function assertPipelineArtifactValues( + yamlText: string, + label: string, + expected: ExpectedPipelineArtifact, +): void { + const docs = parseAllDocuments(yamlText, { merge: false }).map((d) => d.toJS()); + const steps: Record[] = []; + for (const doc of docs) collectDownloadPipelineArtifactSteps(doc, steps); + + const specificRunSteps = steps.filter((step) => { + const inputs = (step.inputs ?? {}) as Record; + return inputs.source === "specific"; + }); + if (specificRunSteps.length === 0) { + throw new Error(`${label}: compiled pipeline has no 'specific run' DownloadPipelineArtifact task`); + } + + for (const step of specificRunSteps) { + const inputs = (step.inputs ?? {}) as Record; + const actual = { + project: inputs.project, + pipeline: inputs.pipeline, + runId: inputs.runId, + artifact: inputs.artifact, + }; + const mismatched = (Object.keys(expected) as (keyof ExpectedPipelineArtifact)[]).filter( + (key) => actual[key] !== expected[key], + ); + if (mismatched.length > 0) { + throw new Error( + `${label}: DownloadPipelineArtifact inputs mismatch — expected ${JSON.stringify(expected)}, got ${JSON.stringify(actual)}`, + ); + } + } +} diff --git a/scripts/ado-script/src/compiler-smoke-e2e/compile-cli.ts b/scripts/ado-script/src/compiler-smoke-e2e/compile-cli.ts new file mode 100644 index 00000000..d3f9b3cc --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/compile-cli.ts @@ -0,0 +1,91 @@ +/** + * Invokes the candidate `ado-aw` binary against one fixture inside the + * detached worktree: `compile --force ` followed by `check `. + * + * Test-harness module; not shipped in `ado-script.zip`. + */ +import { redact, safeSpawn } from "./process.js"; + +export interface CompileCheckResult { + ok: boolean; + /** Which sub-step failed ("compile" | "check"), if any. */ + phase?: "compile" | "check"; + stdout: string; + stderr: string; + message?: string; +} + +export interface CompileCheckOptions { + /** Path to the candidate ado-aw binary. */ + adoAwBin: string; + /** Detached worktree root (subprocess cwd). */ + worktreeDir: string; + /** + * Remote URL whose ADO identity must be embedded in generated metadata. + * + * The candidate is compiled in a GitHub worktree but executed from an Azure + * Repos mirror. Pass the compiler-owned context override only to these child + * processes so staging and runtime integrity compilation infer the same + * org/repo without mutating the worktree's shared Git configuration. + */ + metadataRemoteUrl: string; + /** Repo-relative path to the fixture markdown source. */ + relMd: string; + /** Repo-relative path to the compiled lock file. */ + relLock: string; + timeoutMs: number; + /** Secrets to redact from any captured output/failure message. */ + secrets?: readonly (string | undefined)[]; +} + +/** Run `compile --force ` then `check ` for one fixture. Never throws. */ +export async function compileAndCheck(opts: CompileCheckOptions): Promise { + const secrets = opts.secrets ?? []; + const metadataEnv = { + ADO_AW_COMPILE_REMOTE_URL: opts.metadataRemoteUrl, + }; + + const compileOutcome = await safeSpawn({ + cmd: opts.adoAwBin, + args: ["compile", "--force", opts.relMd], + cwd: opts.worktreeDir, + env: metadataEnv, + timeoutMs: opts.timeoutMs, + }); + if (compileOutcome.timedOut || compileOutcome.status !== 0) { + return { + ok: false, + phase: "compile", + stdout: redact(compileOutcome.stdout, secrets), + stderr: redact(compileOutcome.stderr, secrets), + message: compileOutcome.timedOut + ? `compile --force ${opts.relMd} timed out after ${opts.timeoutMs}ms` + : `compile --force ${opts.relMd} exited ${compileOutcome.status}`, + }; + } + + const checkOutcome = await safeSpawn({ + cmd: opts.adoAwBin, + args: ["check", opts.relLock], + cwd: opts.worktreeDir, + env: metadataEnv, + timeoutMs: opts.timeoutMs, + }); + if (checkOutcome.timedOut || checkOutcome.status !== 0) { + return { + ok: false, + phase: "check", + stdout: redact(checkOutcome.stdout, secrets), + stderr: redact(checkOutcome.stderr, secrets), + message: checkOutcome.timedOut + ? `check ${opts.relLock} timed out after ${opts.timeoutMs}ms` + : `check ${opts.relLock} exited ${checkOutcome.status}`, + }; + } + + return { + ok: true, + stdout: redact(compileOutcome.stdout + checkOutcome.stdout, secrets), + stderr: redact(compileOutcome.stderr + checkOutcome.stderr, secrets), + }; +} diff --git a/scripts/ado-script/src/compiler-smoke-e2e/config.ts b/scripts/ado-script/src/compiler-smoke-e2e/config.ts new file mode 100644 index 00000000..ba94e545 --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/config.ts @@ -0,0 +1,228 @@ +/** + * Environment configuration for the deterministic compiler-smoke E2E harness. + * + * This harness stages the compiler candidate produced by the current build as + * a pinned `supply-chain.pipeline-artifact` source across five real, + * registered ADO pipeline fixtures (canary, azure-cli, noop-target, janitor, + * smoke-failure-reporter), pushes the staged candidate to a per-run branch on + * a mirror repo, queues all five, and asserts they all go green. + * + * Strict, fail-closed parsing lives here so every other module can trust a + * fully validated {@link CompilerSmokeConfig} rather than re-checking env + * vars ad hoc. + * + * Test-harness module; not shipped in `ado-script.zip`. + */ + +/** Per-run candidate branch prefix (never the base ref). */ +export const CANDIDATE_BRANCH_PREFIX = "ado-aw-smoke-candidate"; + +export const DEFAULT_CONCURRENCY = 5; +export const MIN_CONCURRENCY = 1; +export const MAX_CONCURRENCY = 5; + +export const DEFAULT_CHILD_TIMEOUT_MS = 7_200_000; +export const DEFAULT_POLL_MS = 10_000; + +export const DEFAULT_STALE_REF_HOURS = 24; +export const MIN_STALE_REF_HOURS = 6; + +/** Stable declaration order for every fixture-keyed collection in the harness. */ +export const FIXTURE_NAMES = [ + "canary", + "azure-cli", + "noop-target", + "janitor", + "smoke-failure-reporter", +] as const; + +export type FixtureName = (typeof FIXTURE_NAMES)[number]; + +export interface CompilerSmokeConfig { + /** ADO collection URI, e.g. https://dev.azure.com/org/. */ + readonly orgUrl: string; + /** ADO project name (also the pinned pipeline-artifact project). */ + readonly project: string; + /** Write-capable ADO token (System.AccessToken). */ + readonly token: string; + /** Current orchestrator build id (also the pinned pipeline-artifact run-id). */ + readonly buildId: number; + /** Full ref of the checked-out base branch, e.g. refs/heads/main. Never used as the candidate ref. */ + readonly sourceBranch: string; + /** Commit SHA of the checked-out base branch — the candidate commit's parent context. */ + readonly sourceVersion: string; + /** Local checkout root (self repo), used as the base for the detached worktree. */ + readonly sourcesDirectory: string; + /** This orchestrator pipeline's own definition id (used to age-check stale candidate refs). */ + readonly definitionId: number; + /** Path to the candidate ado-aw binary under test. */ + readonly adoAwBin: string; + /** Pipeline artifact name pinned into each fixture's supply-chain config. */ + readonly artifactName: string; + /** ADO Git repo hosting the five registered fixture pipeline definitions. */ + readonly mirrorRepo: string; + /** Registered ADO pipeline definition id, keyed by fixture name. */ + readonly definitionIds: Readonly>; + /** Bounded fixture polling concurrency (1..5, default 5). */ + readonly concurrency: number; + /** Bounded per-fixture build wait, in ms (default 2h). */ + readonly childTimeoutMs: number; + /** Build poll interval, in ms (default 10s). */ + readonly pollMs: number; + /** Minimum age (hours) before a leftover candidate ref is eligible for cleanup (default 24, min 6). */ + readonly staleRefHours: number; +} + +const REQUIRED_STRING_VARS = [ + "SYSTEM_COLLECTIONURI", + "SYSTEM_TEAMPROJECT", + "SYSTEM_ACCESSTOKEN", + "BUILD_SOURCEBRANCH", + "BUILD_SOURCEVERSION", + "BUILD_SOURCESDIRECTORY", + "COMPILER_SMOKE_ADO_AW_BIN", + "COMPILER_SMOKE_ARTIFACT_NAME", + "COMPILER_SMOKE_MIRROR_REPO", +] as const; + +const DEFINITION_ID_ENV_BY_FIXTURE: Readonly> = { + canary: "COMPILER_SMOKE_CANARY_DEFINITION_ID", + "azure-cli": "COMPILER_SMOKE_AZURE_CLI_DEFINITION_ID", + "noop-target": "COMPILER_SMOKE_NOOP_TARGET_DEFINITION_ID", + janitor: "COMPILER_SMOKE_JANITOR_DEFINITION_ID", + "smoke-failure-reporter": "COMPILER_SMOKE_REPORTER_DEFINITION_ID", +}; + +/** ADO macros that failed to expand look like `$(NAME)`; treat them as unset. */ +const UNEXPANDED_MACRO_RE = /^\$\([^)]*\)$/; + +function cleanVar(raw: string | undefined): string | undefined { + const value = raw?.trim(); + if (!value) return undefined; + if (UNEXPANDED_MACRO_RE.test(value)) return undefined; + return value; +} + +function requireString(env: NodeJS.ProcessEnv, name: string): string { + const value = cleanVar(env[name]); + if (value === undefined) { + throw new Error( + `required env var ${name} is not set (or contains an unexpanded ADO macro)`, + ); + } + return value; +} + +/** Parse a required positive integer env var, rejecting malformed/zero/negative values. */ +function requirePositiveInt(env: NodeJS.ProcessEnv, name: string): number { + const raw = requireString(env, name); + const parsed = Number(raw); + if (!Number.isInteger(parsed) || parsed <= 0) { + throw new Error(`${name} must be a positive integer (got '${raw}')`); + } + return parsed; +} + +/** Parse an optional bounded integer env var: default when unset, reject when malformed/out of range. */ +function optionalBoundedInt( + env: NodeJS.ProcessEnv, + name: string, + opts: { default: number; min: number; max?: number }, +): number { + const raw = cleanVar(env[name]); + if (raw === undefined) return opts.default; + const parsed = Number(raw); + if (!Number.isInteger(parsed)) { + throw new Error(`${name} must be an integer (got '${raw}')`); + } + if (parsed < opts.min || (opts.max !== undefined && parsed > opts.max)) { + const range = opts.max !== undefined ? `${opts.min}..${opts.max}` : `>= ${opts.min}`; + throw new Error(`${name} must be in range ${range} (got '${raw}')`); + } + return parsed; +} + +/** Load and strictly validate the harness configuration. Throws on any invalid input. */ +export function loadConfig(env: NodeJS.ProcessEnv = process.env): CompilerSmokeConfig { + for (const name of REQUIRED_STRING_VARS) { + requireString(env, name); + } + + const orgUrl = requireString(env, "SYSTEM_COLLECTIONURI"); + const project = requireString(env, "SYSTEM_TEAMPROJECT"); + const token = requireString(env, "SYSTEM_ACCESSTOKEN"); + const sourceBranch = requireString(env, "BUILD_SOURCEBRANCH"); + const sourceVersion = requireString(env, "BUILD_SOURCEVERSION"); + const sourcesDirectory = requireString(env, "BUILD_SOURCESDIRECTORY"); + const adoAwBin = requireString(env, "COMPILER_SMOKE_ADO_AW_BIN"); + const artifactName = requireString(env, "COMPILER_SMOKE_ARTIFACT_NAME"); + const mirrorRepo = requireString(env, "COMPILER_SMOKE_MIRROR_REPO"); + + const buildId = requirePositiveInt(env, "BUILD_BUILDID"); + const definitionId = requirePositiveInt(env, "SYSTEM_DEFINITIONID"); + + const definitionIds = {} as Record; + for (const fixture of FIXTURE_NAMES) { + definitionIds[fixture] = requirePositiveInt(env, DEFINITION_ID_ENV_BY_FIXTURE[fixture]); + } + + const seen = new Map(); + for (const fixture of FIXTURE_NAMES) { + const id = definitionIds[fixture]; + const existing = seen.get(id); + if (existing) { + existing.push(fixture); + } else { + seen.set(id, [fixture]); + } + } + const duplicates = [...seen.entries()].filter(([, fixtures]) => fixtures.length > 1); + if (duplicates.length > 0) { + const detail = duplicates + .map(([id, fixtures]) => `${id} used by [${fixtures.join(", ")}]`) + .join("; "); + throw new Error(`fixture definition ids must be distinct; duplicates found: ${detail}`); + } + + const concurrency = optionalBoundedInt(env, "COMPILER_SMOKE_CONCURRENCY", { + default: DEFAULT_CONCURRENCY, + min: MIN_CONCURRENCY, + max: MAX_CONCURRENCY, + }); + const childTimeoutMs = optionalBoundedInt(env, "COMPILER_SMOKE_CHILD_TIMEOUT_MS", { + default: DEFAULT_CHILD_TIMEOUT_MS, + min: 1, + }); + const pollMs = optionalBoundedInt(env, "COMPILER_SMOKE_POLL_MS", { + default: DEFAULT_POLL_MS, + min: 1, + }); + const staleRefHours = optionalBoundedInt(env, "COMPILER_SMOKE_STALE_REF_HOURS", { + default: DEFAULT_STALE_REF_HOURS, + min: MIN_STALE_REF_HOURS, + }); + + return { + orgUrl, + project, + token, + buildId, + sourceBranch, + sourceVersion, + sourcesDirectory, + definitionId, + adoAwBin, + artifactName, + mirrorRepo, + definitionIds, + concurrency, + childTimeoutMs, + pollMs, + staleRefHours, + }; +} + +/** Deterministic per-run candidate ref, e.g. refs/heads/ado-aw-smoke-candidate/12345. Never the base ref. */ +export function candidateRef(buildId: number): string { + return `refs/heads/${CANDIDATE_BRANCH_PREFIX}/${buildId}`; +} diff --git a/scripts/ado-script/src/compiler-smoke-e2e/fixtures.ts b/scripts/ado-script/src/compiler-smoke-e2e/fixtures.ts new file mode 100644 index 00000000..137de4ed --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/fixtures.ts @@ -0,0 +1,57 @@ +/** + * Manifest of the five fixed compiler-smoke fixtures. + * + * These are the same five agentic-pipeline sources documented in + * `tests/safe-outputs/README.md` (canary, azure-cli, noop-target, janitor, + * smoke-failure-reporter) — this harness does not invent its own fixture + * content. It reads the exact files from the detached candidate worktree + * (an exact checkout of `BUILD_SOURCEVERSION`, at + * `/tests/safe-outputs/.md` — never from + * `BUILD_SOURCESDIRECTORY`, which may sit at a different commit), stages a + * pinned `supply-chain.pipeline-artifact` transform of each onto the mirror + * repo, recompiles, and queues the five FIXED "candidate lane" pipeline + * definitions tracked in `tests/compiler-smoke-e2e/REGISTERED.md` (distinct + * from the release-backed definitions those same sources also feed). + * + * Test-harness module; not shipped in `ado-script.zip`. + */ +import type { FixtureName } from "./config.js"; +import { FIXTURE_NAMES } from "./config.js"; + +/** Repo-relative directory containing every fixture source + compiled lock. */ +export const FIXTURE_DIR = "tests/safe-outputs"; + +export interface FixturePaths { + readonly name: FixtureName; + /** Repo-relative path to the fixture markdown source, e.g. tests/safe-outputs/canary.md. */ + readonly relMd: string; + /** Repo-relative path to the compiled lock file, e.g. tests/safe-outputs/canary.lock.yml. */ + readonly relLock: string; +} + +/** Repo-relative path for a fixture's markdown source. Always POSIX-separated (a git path, not a filesystem path). */ +export function fixturePaths(name: FixtureName): FixturePaths { + return { + name, + relMd: `${FIXTURE_DIR}/${name}.md`, + relLock: `${FIXTURE_DIR}/${name}.lock.yml`, + }; +} + +/** All five fixtures in the stable declaration order used throughout the harness. */ +export const ALL_FIXTURES: readonly FixturePaths[] = FIXTURE_NAMES.map(fixturePaths); + +/** + * The exact set of repo-relative paths the candidate-staging commit is + * allowed to touch: the five markdown sources, their five compiled locks, + * and the compiler-managed `.gitattributes` block. Any other changed path + * fails the run before it pushes anything. + */ +export function allowedChangedPaths(): Set { + const paths = new Set([".gitattributes"]); + for (const fixture of ALL_FIXTURES) { + paths.add(fixture.relMd); + paths.add(fixture.relLock); + } + return paths; +} diff --git a/scripts/ado-script/src/compiler-smoke-e2e/git.ts b/scripts/ado-script/src/compiler-smoke-e2e/git.ts new file mode 100644 index 00000000..fa1c4316 --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/git.ts @@ -0,0 +1,287 @@ +/** + * Git operations for staging a compiler candidate onto the mirror repo: + * fetch the base ref into the self checkout's object store, spin up a + * detached temp worktree, commit the transformed fixtures, push to a + * per-run candidate ref, verify, and clean up (remote ref + worktree). + * + * Also implements the startup stale-ref scanner (see {@link scanStaleRefs} + * usage in `index.ts`). + * + * Every git invocation goes through an injectable {@link GitRunner} so tests + * never need a real repository or network access — mirrors the DI pattern + * used by `trigger-e2e/mirror.ts`. + * + * Test-harness module; not shipped in `ado-script.zip`. + */ +import { bearerEnv } from "../shared/git.js"; +import { redact, safeSpawn, type SpawnOutcome } from "./process.js"; +import { CANDIDATE_BRANCH_PREFIX } from "./config.js"; + +export interface GitRunOptions { + cwd: string; + env?: NodeJS.ProcessEnv; + timeoutMs: number; +} + +export type GitRunner = (args: string[], opts: GitRunOptions) => Promise; + +export const defaultGitRunner: GitRunner = (args, opts) => + safeSpawn({ + cmd: "git", + args, + cwd: opts.cwd, + // GIT_TERMINAL_PROMPT=0 ensures a bad/expired token or unreachable + // mirror fails fast with a normal non-zero exit instead of hanging on an + // interactive credential prompt in a non-interactive pipeline agent. + // Applied AFTER `opts.env` (e.g. the bearer-token triple) so it is a + // true, non-overridable floor — no caller in this harness ever needs to + // set it itself. + env: { ...opts.env, GIT_TERMINAL_PROMPT: "0" }, + timeoutMs: opts.timeoutMs, + }); + +/** Deterministic commit identity for staged candidate commits. */ +export const COMMIT_IDENTITY = { + name: "ado-aw-smoke-e2e", + email: "ado-aw-smoke-e2e@users.noreply.github.com", +} as const; + +/** Deterministic commit message: `test(smoke): stage compiler candidate `. */ +export function commitMessage(buildId: number): string { + return `test(smoke): stage compiler candidate ${buildId}`; +} + +/** Build the ADO Git remote URL for `//_git/`. */ +export function mirrorRepoUrl(orgUrl: string, project: string, repo: string): string { + return `${orgUrl.replace(/\/+$/, "")}/${encodeURIComponent(project)}/_git/${encodeURIComponent(repo)}`; +} + +async function run( + args: string[], + opts: GitRunOptions, + runner: GitRunner, + secrets: readonly (string | undefined)[] = [], +): Promise { + const outcome = await runner(args, opts); + if (outcome.timedOut) { + throw new Error(`git ${args[0] ?? ""} timed out after ${opts.timeoutMs}ms`); + } + if (outcome.status !== 0) { + const detail = redact([outcome.stderr.trim(), outcome.stdout.trim()].filter(Boolean).join("\n"), secrets); + throw new Error(`git ${args.join(" ")} failed (exit ${outcome.status}): ${detail || "(no output)"}`); + } + // Trim only TRAILING whitespace here — a leading `.trim()` would eat the + // significant leading space of a ` M ` line in `git status + // --porcelain=v1` output (an unmodified-index/modified-worktree entry), + // silently corrupting the very first parsed path in + // `worktreeChangedFiles`. + return redact(outcome.stdout.replace(/\s+$/, ""), secrets); +} + +/** + * Verify `expectedSha` (BUILD_SOURCEVERSION) is present in the self checkout + * at `cwd` (BUILD_SOURCESDIRECTORY) so the detached worktree can be built + * directly from it — never fetched from the mirror repo. + * + * This is deliberately NOT a mirror fetch: for a GitHub PR build, + * BUILD_SOURCEBRANCH is a synthetic ref such as `refs/pull//merge` that + * only exists on the GitHub-backed remote the pipeline checked out from, not + * on the ADO `ado-aw-mirror` repo — attempting to fetch it there would fail + * (or silently resolve to nothing meaningful). The pipeline's own checkout + * step already fetched every object this build needs into `cwd`, so we only + * need to confirm `expectedSha` is actually reachable there before basing a + * worktree on it; the resulting candidate commit is the only thing pushed + * to the mirror. + */ +export async function verifyLocalCommit( + opts: { cwd: string; expectedSha: string; timeoutMs: number }, + runner: GitRunner = defaultGitRunner, +): Promise { + const headSha = await run(["rev-parse", "HEAD"], { cwd: opts.cwd, timeoutMs: opts.timeoutMs }, runner); + if (headSha.toLowerCase() === opts.expectedSha.toLowerCase()) return; + // HEAD can legitimately differ from BUILD_SOURCEVERSION (e.g. a synthetic + // PR merge commit checked out as HEAD while BUILD_SOURCEVERSION names the + // PR source-branch tip) — fall back to an object-existence check rather + // than requiring an exact HEAD match. + const outcome = await runner( + ["cat-file", "-e", `${opts.expectedSha}^{commit}`], + { cwd: opts.cwd, timeoutMs: opts.timeoutMs }, + ); + if (outcome.timedOut || outcome.status !== 0) { + throw new Error( + `BUILD_SOURCEVERSION ${opts.expectedSha} was not found as a commit object in the local checkout at ` + + `${opts.cwd} (HEAD is ${headSha}); refusing to fetch it from the mirror repo since a GitHub PR ref ` + + `(e.g. refs/pull//merge) would not exist there`, + ); + } +} + +/** Create a detached temp worktree at `worktreeDir`, checked out at `commitish`. */ +export async function createDetachedWorktree( + opts: { cwd: string; worktreeDir: string; commitish: string; timeoutMs: number }, + runner: GitRunner = defaultGitRunner, +): Promise { + await run( + ["worktree", "add", "--detach", opts.worktreeDir, opts.commitish], + { cwd: opts.cwd, timeoutMs: opts.timeoutMs }, + runner, + ); +} + +/** Remove a detached worktree (best-effort caller decides how to handle failure). */ +export async function removeWorktree( + opts: { cwd: string; worktreeDir: string; timeoutMs: number }, + runner: GitRunner = defaultGitRunner, +): Promise { + await run( + ["worktree", "remove", "--force", opts.worktreeDir], + { cwd: opts.cwd, timeoutMs: opts.timeoutMs }, + runner, + ); +} + +/** + * List working-tree changes inside `worktreeDir` as a flat list of affected + * repo-relative paths (renames contribute both their old and new path). + */ +export async function worktreeChangedFiles( + opts: { worktreeDir: string; timeoutMs: number }, + runner: GitRunner = defaultGitRunner, +): Promise { + const stdout = await run( + ["status", "--porcelain=v1"], + { cwd: opts.worktreeDir, timeoutMs: opts.timeoutMs }, + runner, + ); + if (!stdout) return []; + const files: string[] = []; + for (const line of stdout.split("\n")) { + if (!line) continue; + const rest = line.slice(3); + const arrow = rest.indexOf(" -> "); + if (arrow >= 0) { + files.push(rest.slice(0, arrow), rest.slice(arrow + 4)); + } else { + files.push(rest); + } + } + return files; +} + +/** Pure comparison: returns any changed path NOT in `allowed`. Empty = clean. */ +export function disallowedChanges(changed: readonly string[], allowed: ReadonlySet): string[] { + return changed.filter((path) => !allowed.has(path)); +} + +/** Stage all changes and commit with the deterministic identity/message. Returns the new commit SHA. */ +export async function commitAll( + opts: { worktreeDir: string; buildId: number; timeoutMs: number }, + runner: GitRunner = defaultGitRunner, +): Promise { + await run(["add", "-A"], { cwd: opts.worktreeDir, timeoutMs: opts.timeoutMs }, runner); + await run( + [ + "-c", + `user.name=${COMMIT_IDENTITY.name}`, + "-c", + `user.email=${COMMIT_IDENTITY.email}`, + "commit", + "-m", + commitMessage(opts.buildId), + ], + { cwd: opts.worktreeDir, timeoutMs: opts.timeoutMs }, + runner, + ); + return run(["rev-parse", "HEAD"], { cwd: opts.worktreeDir, timeoutMs: opts.timeoutMs }, runner); +} + +/** Push the worktree's HEAD to `ref` on the mirror repo (never force). */ +export async function pushCandidate( + opts: { worktreeDir: string; mirrorUrl: string; ref: string; token: string; timeoutMs: number }, + runner: GitRunner = defaultGitRunner, +): Promise { + const env = bearerEnv(opts.token); + await run( + ["push", "--porcelain", opts.mirrorUrl, `HEAD:${opts.ref}`], + { cwd: opts.worktreeDir, env, timeoutMs: opts.timeoutMs }, + runner, + [opts.token], + ); +} + +/** Verify the pushed ref resolves to `expectedSha` on the remote. Throws on mismatch. */ +export async function verifyRemoteRef( + opts: { cwd: string; mirrorUrl: string; ref: string; expectedSha: string; token: string; timeoutMs: number }, + runner: GitRunner = defaultGitRunner, +): Promise { + const env = bearerEnv(opts.token); + const stdout = await run( + ["ls-remote", opts.mirrorUrl, opts.ref], + { cwd: opts.cwd, env, timeoutMs: opts.timeoutMs }, + runner, + [opts.token], + ); + const remoteSha = stdout.split(/\s+/)[0]?.trim(); + if (!remoteSha || remoteSha.toLowerCase() !== opts.expectedSha.toLowerCase()) { + throw new Error( + `candidate push verification failed: expected ${opts.expectedSha} at ${opts.ref}, got '${remoteSha ?? ""}'`, + ); + } +} + +/** Delete the candidate ref on the mirror repo (best-effort; caller decides how to handle failure). */ +export async function deleteRemoteRef( + opts: { cwd: string; mirrorUrl: string; ref: string; token: string; timeoutMs: number }, + runner: GitRunner = defaultGitRunner, +): Promise { + const env = bearerEnv(opts.token); + await run( + ["push", "--porcelain", opts.mirrorUrl, "--delete", opts.ref], + { cwd: opts.cwd, env, timeoutMs: opts.timeoutMs }, + runner, + [opts.token], + ); +} + +export interface RemoteRef { + ref: string; + sha: string; +} + +/** List every remote ref under the exact `refs/heads//` prefix. */ +export async function listCandidateRefs( + opts: { cwd: string; mirrorUrl: string; token: string; timeoutMs: number }, + runner: GitRunner = defaultGitRunner, +): Promise { + const env = bearerEnv(opts.token); + const stdout = await run( + ["ls-remote", "--heads", opts.mirrorUrl, `refs/heads/${CANDIDATE_BRANCH_PREFIX}/*`], + { cwd: opts.cwd, env, timeoutMs: opts.timeoutMs }, + runner, + [opts.token], + ); + if (!stdout) return []; + const prefix = `refs/heads/${CANDIDATE_BRANCH_PREFIX}/`; + const refs: RemoteRef[] = []; + for (const line of stdout.split("\n")) { + if (!line.trim()) continue; + const [sha, ref] = line.split(/\s+/); + // Exact-prefix guard: ls-remote's glob can match unintended refs on some + // git/server implementations (e.g. a sibling branch containing the + // pattern as a substring) — never treat those as our candidate refs. + if (sha && ref && ref.startsWith(prefix)) { + refs.push({ ref, sha }); + } + } + return refs; +} + +/** Parse the numeric build id embedded in a candidate ref name, or `undefined` if malformed. */ +export function parseCandidateBuildId(ref: string): number | undefined { + const prefix = `refs/heads/${CANDIDATE_BRANCH_PREFIX}/`; + if (!ref.startsWith(prefix)) return undefined; + const suffix = ref.slice(prefix.length); + if (!/^[0-9]+$/.test(suffix)) return undefined; + const id = Number(suffix); + return Number.isSafeInteger(id) && id > 0 ? id : undefined; +} diff --git a/scripts/ado-script/src/compiler-smoke-e2e/index.ts b/scripts/ado-script/src/compiler-smoke-e2e/index.ts new file mode 100644 index 00000000..fe3c4547 --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/index.ts @@ -0,0 +1,320 @@ +/** + * Entry point for the deterministic compiler-smoke E2E orchestrator. + * + * Stages the compiler candidate produced by the current build (PR or + * nightly `main`) as a pinned `supply-chain.pipeline-artifact` source across + * the five real fixtures documented in `tests/safe-outputs/README.md` + * (canary, azure-cli, noop-target, janitor, smoke-failure-reporter), pushes + * the staged candidate to a short-lived branch on the mirror repo, queues + * the five FIXED "candidate lane" pipeline definitions (tracked in + * `tests/compiler-smoke-e2e/REGISTERED.md`), and asserts they all go green. + * + * See `config.ts` for the full required/optional env var contract. + * + * Test-harness module; not shipped in `ado-script.zip`. + */ +import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; + +import { AdoRest } from "./ado-rest.js"; +import { assertNoForbiddenReleaseUrls, assertPipelineArtifactValues } from "./assertions.js"; +import { candidateRef, FIXTURE_NAMES, loadConfig, type CompilerSmokeConfig } from "./config.js"; +import { compileAndCheck } from "./compile-cli.js"; +import { ALL_FIXTURES, allowedChangedPaths } from "./fixtures.js"; +import { + commitAll, + createDetachedWorktree, + deleteRemoteRef, + disallowedChanges, + listCandidateRefs, + mirrorRepoUrl, + pushCandidate, + removeWorktree, + verifyLocalCommit, + verifyRemoteRef, + worktreeChangedFiles, +} from "./git.js"; +import { injectPipelineArtifact } from "./source.js"; +import { renderResultsTable } from "./report.js"; +import { runFixtures, type FixtureBuildRequest, type FixtureBuildResult } from "./runner.js"; +import { scanStaleRefs } from "./stale.js"; + +function log(msg: string): void { + // Percent-encode a leading '#' so a message cannot smuggle a ##vso command. + process.stdout.write(msg.replace(/^#/gm, "%23") + "\n"); +} + +function errMessage(err: unknown): string { + return err instanceof Error ? err.message : String(err); +} + +/** Read each fixture's markdown directly from the detached candidate worktree (an exact checkout of BUILD_SOURCEVERSION — never the possibly-divergent BUILD_SOURCESDIRECTORY), then apply the pipeline-artifact transform in place. */ +async function stageFixtures(config: CompilerSmokeConfig, worktreeDir: string): Promise { + for (const fixture of ALL_FIXTURES) { + const selfContent = await readFile(join(worktreeDir, fixture.relMd), "utf8"); + const transformed = injectPipelineArtifact(selfContent, { + project: config.project, + definitionId: config.definitionId, + runId: config.buildId, + artifact: config.artifactName, + }); + await writeFile(join(worktreeDir, fixture.relMd), transformed, "utf8"); + } +} + +async function compileFixtures( + config: CompilerSmokeConfig, + worktreeDir: string, + mirrorUrl: string, +): Promise { + for (const fixture of ALL_FIXTURES) { + const result = await compileAndCheck({ + adoAwBin: config.adoAwBin, + worktreeDir, + metadataRemoteUrl: mirrorUrl, + relMd: fixture.relMd, + relLock: fixture.relLock, + timeoutMs: config.childTimeoutMs, + secrets: [config.token], + }); + if (!result.ok) { + throw new Error( + `fixture '${fixture.name}' ${result.phase} failed: ${result.message}\n--- stdout ---\n${result.stdout}\n--- stderr ---\n${result.stderr}`, + ); + } + + const yamlText = await readFile(join(worktreeDir, fixture.relLock), "utf8"); + assertNoForbiddenReleaseUrls(yamlText, fixture.name); + assertPipelineArtifactValues(yamlText, fixture.name, { + project: config.project, + pipeline: String(config.definitionId), + runId: String(config.buildId), + artifact: config.artifactName, + }); + } +} + +async function cleanupStaleRefs(config: CompilerSmokeConfig, rest: AdoRest, mirrorUrl: string, ownRef: string): Promise { + try { + const refs = await listCandidateRefs({ + cwd: config.sourcesDirectory, + mirrorUrl, + token: config.token, + timeoutMs: config.childTimeoutMs, + }); + const decisions = await scanStaleRefs({ + refs, + baseRef: config.sourceBranch, + ownRef, + definitionId: config.definitionId, + childDefinitionIds: FIXTURE_NAMES.map((name) => config.definitionIds[name]), + staleRefHours: config.staleRefHours, + client: rest, + }); + for (const decision of decisions) { + if (decision.outcome !== "eligible") { + log(`[stale-scan] ${decision.ref}: ${decision.outcome} — ${decision.reason}`); + continue; + } + try { + await deleteRemoteRef({ + cwd: config.sourcesDirectory, + mirrorUrl, + ref: decision.ref, + token: config.token, + timeoutMs: config.childTimeoutMs, + }); + log(`[stale-scan] deleted ${decision.ref}: ${decision.reason}`); + } catch (err) { + log(`[stale-scan] WARNING: failed to delete ${decision.ref}: ${errMessage(err)}`); + } + } + } catch (err) { + log(`[stale-scan] WARNING: scan failed (best-effort, continuing): ${errMessage(err)}`); + } +} + +export async function main(): Promise { + const config = loadConfig(); + const rest = new AdoRest({ orgUrl: config.orgUrl, project: config.project, token: config.token, log }); + const mirrorUrl = mirrorRepoUrl(config.orgUrl, config.project, config.mirrorRepo); + const ownRef = candidateRef(config.buildId); + + log( + `compiler-smoke-e2e: build #${config.buildId}, candidate ref ${ownRef}, mirror '${config.mirrorRepo}'`, + ); + + // ---- Artifact visibility gate — before any source/git work or queueing ---- + await rest.getArtifact(config.buildId, config.artifactName); + log(`[artifact-visibility] '${config.artifactName}' is visible on build #${config.buildId}`); + + // ---- Best-effort startup stale-ref cleanup ---- + await cleanupStaleRefs(config, rest, mirrorUrl, ownRef); + + const worktreeParent = await mkdtemp(join(tmpdir(), "ado-aw-compiler-smoke-")); + const worktreeDir = join(worktreeParent, "candidate"); + + let pushed = false; + let overallOk = true; + // Placeholder only: never trusted directly. It's forced to `false` + // immediately before `runFixtures` is invoked (see below) and only ever + // set back to `true` from that call's own returned outcome. + let allChildrenTerminal = true; + let failureMessage: string | undefined; + let results: FixtureBuildResult[] = []; + + try { + // The detached worktree is based directly on the LOCALLY checked-out + // BUILD_SOURCEVERSION — never fetched from the mirror. For a GitHub PR + // build, BUILD_SOURCEBRANCH is a synthetic ref (e.g. + // `refs/pull//merge`) that does not exist on the ADO mirror repo; the + // self checkout at BUILD_SOURCESDIRECTORY already has every object this + // build needs. Only the resulting candidate commit is pushed TO the + // mirror below. + await verifyLocalCommit({ + cwd: config.sourcesDirectory, + expectedSha: config.sourceVersion, + timeoutMs: config.childTimeoutMs, + }); + await createDetachedWorktree({ + cwd: config.sourcesDirectory, + worktreeDir, + commitish: config.sourceVersion, + timeoutMs: config.childTimeoutMs, + }); + + await stageFixtures(config, worktreeDir); + await compileFixtures(config, worktreeDir, mirrorUrl); + + const changed = await worktreeChangedFiles({ worktreeDir, timeoutMs: config.childTimeoutMs }); + const violations = disallowedChanges(changed, allowedChangedPaths()); + if (violations.length > 0) { + throw new Error(`refusing to push: unexpected path(s) changed: ${violations.join(", ")}`); + } + + const candidateSha = await commitAll({ + worktreeDir, + buildId: config.buildId, + timeoutMs: config.childTimeoutMs, + }); + await pushCandidate({ + worktreeDir, + mirrorUrl, + ref: ownRef, + token: config.token, + timeoutMs: config.childTimeoutMs, + }); + pushed = true; + await verifyRemoteRef({ + cwd: worktreeDir, + mirrorUrl, + ref: ownRef, + expectedSha: candidateSha, + token: config.token, + timeoutMs: config.childTimeoutMs, + }); + log(`[git] candidate ${candidateSha} pushed to ${ownRef}`); + + const requests: FixtureBuildRequest[] = FIXTURE_NAMES.map((name) => ({ + name, + definitionId: config.definitionIds[name], + sourceBranch: ownRef, + sourceVersion: candidateSha, + })); + // Fail-closed: flip to `false` right before the call that might queue + // builds, so an unexpected throw out of `runFixtures` itself (a runner + // bug, not a reported build failure) can never leave this at its + // initial `true` and delete the ref out from under a build that may + // have been queued. Only a normally-returned outcome is trusted to set + // this back to `true`. + allChildrenTerminal = false; + const outcome = await runFixtures(rest, requests, { + concurrency: config.concurrency, + timeoutMs: config.childTimeoutMs, + pollMs: config.pollMs, + log, + }); + results = outcome.results; + overallOk = outcome.ok; + allChildrenTerminal = outcome.allTerminal; + if (!overallOk) failureMessage = "one or more fixture builds did not succeed"; + if (!allChildrenTerminal) { + overallOk = false; + failureMessage = [ + failureMessage, + `could not confirm every fixture build reached a terminal state — retaining ${ownRef} for the startup stale-ref scanner to clean up once ADO confirms completion`, + ] + .filter(Boolean) + .join("; "); + } + } catch (err) { + overallOk = false; + failureMessage = errMessage(err); + log(`FAILED: ${failureMessage}`); + } finally { + if (pushed) { + if (allChildrenTerminal) { + try { + await deleteRemoteRef({ + cwd: config.sourcesDirectory, + mirrorUrl, + ref: ownRef, + token: config.token, + timeoutMs: config.childTimeoutMs, + }); + } catch (err) { + overallOk = false; + failureMessage ??= `failed to delete candidate ref ${ownRef}: ${errMessage(err)}`; + log(`WARNING: failed to delete candidate ref ${ownRef}: ${errMessage(err)}`); + } + } else { + // Never delete a ref while any queued build might still be + // running against it — retain it and let the fail-closed + // stale-ref scanner reclaim it on a later run once it can prove + // every child build actually terminated. + log( + `WARNING: retaining candidate ref ${ownRef} because not every fixture build's terminal state could be confirmed`, + ); + } + } + try { + await removeWorktree({ + cwd: config.sourcesDirectory, + worktreeDir, + timeoutMs: config.childTimeoutMs, + }); + } catch (err) { + overallOk = false; + failureMessage ??= `failed to remove worktree: ${errMessage(err)}`; + log(`WARNING: failed to remove worktree ${worktreeDir}: ${errMessage(err)}`); + } + await rm(worktreeParent, { recursive: true, force: true }).catch(() => {}); + } + + if (results.length > 0) { + log(""); + log("=== Compiler smoke E2E results ==="); + log(renderResultsTable(results)); + } + if (failureMessage) { + log(`Overall: FAILED — ${failureMessage}`); + } else { + log("Overall: PASSED"); + } + + return overallOk ? 0 : 1; +} + +// Run as the bundle entry point. Skipped under Vitest so unit tests can +// import these modules without launching the whole suite. +if (process.env.VITEST !== "true") { + main().then( + (code) => process.exit(code), + (err: unknown) => { + const e = err as Error; + log(`compiler-smoke-e2e crashed: ${e.stack ?? e.message}`); + process.exit(1); + }, + ); +} diff --git a/scripts/ado-script/src/compiler-smoke-e2e/process.ts b/scripts/ado-script/src/compiler-smoke-e2e/process.ts new file mode 100644 index 00000000..4c01e26c --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/process.ts @@ -0,0 +1,145 @@ +/** + * Safe subprocess execution primitive shared by every other module in this + * harness (git commands, the candidate `ado-aw` binary). + * + * Guarantees: + * - bounded wall-clock time (SIGKILL past `timeoutMs`), + * - bounded captured output (stdout/stderr truncated past `maxOutputBytes` + * so a runaway/looping child can never blow up harness memory or logs), + * - secret redaction of every string handed back to the caller (including + * thrown/failure messages), and + * - no ambient git tracing env vars (`GIT_TRACE*`, `GIT_CURL_VERBOSE`) — + * those can leak a bearer `Authorization` header into captured stderr. + * + * Test-harness module; not shipped in `ado-script.zip`. + */ +import { spawn } from "node:child_process"; + +/** Env vars that can leak secrets (bearer headers) into captured output. */ +const TRACE_ENV_VARS = ["GIT_TRACE", "GIT_TRACE_CURL", "GIT_CURL_VERBOSE"] as const; + +const DEFAULT_MAX_OUTPUT_BYTES = 1_000_000; + +export interface SpawnRequest { + cmd: string; + args: string[]; + cwd?: string; + env?: NodeJS.ProcessEnv; + timeoutMs: number; + maxOutputBytes?: number; +} + +export interface SpawnOutcome { + status: number | null; + stdout: string; + stderr: string; + timedOut: boolean; + stdoutTruncated: boolean; + stderrTruncated: boolean; +} + +/** Strip env vars that can leak secrets (bearer headers) into git trace output. */ +export function stripTraceEnv(env: NodeJS.ProcessEnv): NodeJS.ProcessEnv { + const out = { ...env }; + for (const name of TRACE_ENV_VARS) { + delete out[name]; + } + return out; +} + +/** Replace every occurrence of any secret in `text` with `***`. Empty/undefined secrets are ignored. */ +export function redact(text: string, secrets: readonly (string | undefined)[]): string { + let out = text; + for (const secret of secrets) { + if (!secret) continue; + out = out.split(secret).join("***"); + } + return out; +} + +class BoundedBuffer { + private chunks: string[] = []; + private bytes = 0; + truncated = false; + + constructor(private readonly maxBytes: number) {} + + push(data: Buffer): void { + if (this.truncated) return; + const remaining = this.maxBytes - this.bytes; + if (remaining <= 0) { + this.truncated = true; + return; + } + const text = data.toString("utf8"); + if (Buffer.byteLength(text, "utf8") <= remaining) { + this.chunks.push(text); + this.bytes += Buffer.byteLength(text, "utf8"); + return; + } + // Truncate at a safe (possibly mid-character) boundary; this is + // diagnostic-only output so exactness past the cap doesn't matter. + this.chunks.push(Buffer.from(text, "utf8").subarray(0, remaining).toString("utf8")); + this.bytes = this.maxBytes; + this.truncated = true; + } + + toString(): string { + return this.chunks.join(""); + } +} + +/** + * Spawn `cmd` with bounded output/time and no ambient trace env. Never + * throws for a non-zero exit or timeout — callers inspect `status`/ + * `timedOut`. Output is NOT pre-redacted (callers know which secrets to + * redact); use {@link redact} before logging or embedding in an error. + */ +export function safeSpawn(request: SpawnRequest): Promise { + const maxOutputBytes = request.maxOutputBytes ?? DEFAULT_MAX_OUTPUT_BYTES; + return new Promise((resolve, reject) => { + const env = stripTraceEnv({ ...process.env, ...request.env }); + const child = spawn(request.cmd, request.args, { + cwd: request.cwd, + env, + }); + + const stdout = new BoundedBuffer(maxOutputBytes); + const stderr = new BoundedBuffer(maxOutputBytes); + let timedOut = false; + let settled = false; + + const timer = setTimeout(() => { + timedOut = true; + child.kill("SIGKILL"); + }, request.timeoutMs); + + child.stdout?.on("data", (d: Buffer) => stdout.push(d)); + child.stderr?.on("data", (d: Buffer) => stderr.push(d)); + + child.on("error", (err) => { + if (settled) return; + settled = true; + clearTimeout(timer); + reject(err); + }); + + child.on("close", (status) => { + if (settled) return; + settled = true; + clearTimeout(timer); + resolve({ + status, + stdout: stdout.toString(), + stderr: stderr.toString(), + timedOut, + stdoutTruncated: stdout.truncated, + stderrTruncated: stderr.truncated, + }); + }); + }); +} + +export function sleep(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); +} diff --git a/scripts/ado-script/src/compiler-smoke-e2e/report.ts b/scripts/ado-script/src/compiler-smoke-e2e/report.ts new file mode 100644 index 00000000..e410fb25 --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/report.ts @@ -0,0 +1,34 @@ +/** + * Concise final results table: fixture / definition / build / url / result / duration. + * + * Test-harness module; not shipped in `ado-script.zip`. + */ +import type { FixtureBuildResult } from "./runner.js"; + +function pad(value: string, width: number): string { + return value.length >= width ? value : value + " ".repeat(width - value.length); +} + +/** Render the final per-fixture outcome table, in the caller's declaration order. */ +export function renderResultsTable(results: readonly FixtureBuildResult[]): string { + const headers = ["fixture", "definition", "build", "url", "result", "duration"]; + const rows = results.map((r) => [ + r.name, + String(r.definitionId), + r.buildId !== undefined ? String(r.buildId) : "-", + r.url ?? "-", + r.result ? `${r.status} (${r.result})` : r.status, + `${(r.durationMs / 1000).toFixed(1)}s`, + ]); + const widths = headers.map((h, i) => Math.max(h.length, ...rows.map((row) => row[i]?.length ?? 0))); + + const lines = [ + headers.map((h, i) => pad(h, widths[i] ?? h.length)).join(" "), + widths.map((w) => "-".repeat(w)).join(" "), + ...rows.map((row) => row.map((cell, i) => pad(cell, widths[i] ?? cell.length)).join(" ")), + ]; + for (const r of results) { + if (r.message) lines.push(` [${r.name}] ${r.message}`); + } + return lines.join("\n"); +} diff --git a/scripts/ado-script/src/compiler-smoke-e2e/runner.ts b/scripts/ado-script/src/compiler-smoke-e2e/runner.ts new file mode 100644 index 00000000..1fb6c87c --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/runner.ts @@ -0,0 +1,386 @@ +/** + * Bounded-concurrency queue/poll/cancel state machine for the five staged + * fixture builds. + * + * Contract: + * - every fixture is queued concurrently (a queue failure for one fixture + * never prevents queueing the rest — "partial queue failure still + * polls/cancels queued" — but DOES immediately signal the shared abort + * flag so already-queued siblings are cancelled rather than left to run + * to completion for a build that's already doomed to fail overall), + * - a `queueBuild` error is treated as AMBIGUOUS, never as proof nothing + * was created: ADO may have accepted and started the build before the + * client lost the response (timeout, connection reset, etc). This + * harness has no cheap way to reconcile that here, so a queue failure + * always sets `terminalProven = false` (the simpler, fail-closed + * choice) — the caller must retain the candidate ref, and the startup + * stale-ref scanner's per-child-definition build check is what + * eventually proves (or disproves) that an orphaned build exists, + * - successfully queued builds are polled with bounded concurrency, + * - the FIRST failure or timeout flips a shared abort flag; every other + * still-polling build is cancelled and polled to a terminal state + * before this function returns, + * - each polled build's identity (definition id, sourceBranch, + * sourceVersion) is verified against what was requested with exact + * equality on all three fields — a missing field is itself treated as + * a mismatch (never as "no data to compare, assume fine") — and any + * mismatch is treated as a failure and also flips the abort flag, + * - this function NEVER silently claims a build "must have stopped" + * merely because polling gave up (a `getBuild` error, a cancellation + * that never gets confirmed, or a grace-period expiry). Every result + * that isn't a positively-observed terminal state is reported via + * `RunFixturesOutcome.allTerminal = false` so the caller knows it must + * NOT delete the shared git ref out from under a build that might still + * be running, + * - results preserve the caller's declaration order regardless of + * completion order. + * + * Test-harness module; not shipped in `ado-script.zip`. + */ +import type { FixtureName } from "./config.js"; +import { sleep as defaultSleep } from "./process.js"; + +/** What a queued build looks like once polled — kept narrow (a subset of `AdoRest.BuildSummary`) so tests never need a full AdoRest fake. */ +export interface PolledBuild { + status?: string; + result?: string; + definition?: { id?: number }; + sourceBranch?: string; + sourceVersion?: string; +} + +/** The minimal ADO Build surface this state machine needs. */ +export interface FixtureBuildClient { + queueBuild(definitionId: number, opts: { sourceBranch: string; sourceVersion: string }): Promise<{ id: number }>; + getBuild(buildId: number): Promise; + cancelBuild(buildId: number): Promise; + buildUrl(buildId: number): string; +} + +export interface FixtureBuildRequest { + name: FixtureName; + definitionId: number; + sourceBranch: string; + sourceVersion: string; +} + +export type FixtureBuildStatus = + | "succeeded" + | "failed" + | "canceled" + | "timed-out" + | "queue-failed"; + +export interface FixtureBuildResult { + name: FixtureName; + definitionId: number; + buildId?: number; + url?: string; + status: FixtureBuildStatus; + result?: string; + message?: string; + durationMs: number; + /** + * Whether this fixture's terminal state was positively confirmed. + * `false` means the harness could not prove ADO actually stopped this + * build — callers must treat this as "possibly still running" and never + * delete the candidate ref. For `queue-failed`, `false` also covers the + * case where the `queueBuild` request itself may have been accepted by + * ADO despite the client observing an error (ambiguous network/timeout + * failures) — the harness never assumes "no response" means "no build". + */ + terminalProven: boolean; +} + +export interface RunFixturesOptions { + concurrency: number; + /** Bounded per-build wait before this harness gives up and cancels it. */ + timeoutMs: number; + pollMs: number; + log: (msg: string) => void; + sleepImpl?: (ms: number) => Promise; + /** Extra bounded wait for a cancelled build to actually reach 'completed' (default: max(pollMs*6, 60s)). */ + cancelGraceMs?: number; +} + +export interface RunFixturesOutcome { + ok: boolean; + /** True only when EVERY fixture's terminal state was positively proven. See {@link FixtureBuildResult.terminalProven}. */ + allTerminal: boolean; + results: FixtureBuildResult[]; +} + +function errMessage(err: unknown): string { + return err instanceof Error ? err.message : String(err); +} + +async function safeCancel( + client: FixtureBuildClient, + buildId: number, + log: (msg: string) => void, +): Promise { + try { + await client.cancelBuild(buildId); + } catch (err) { + log(`WARNING: cancelBuild(${buildId}) failed: ${errMessage(err)}`); + } +} + +interface AbortSignalLike { + aborted: boolean; + signal(): void; +} + +function makeAbortFlag(): AbortSignalLike { + return { + aborted: false, + signal(): void { + this.aborted = true; + }, + }; +} + +/** Compare a polled build's identity against what this harness requested. `undefined` is itself a mismatch — missing identity data is never treated as proof of the requested build. Returns `undefined` only when every field is present and exactly equal. */ +function describeMismatch( + build: PolledBuild, + expected: { definitionId: number; sourceBranch: string; sourceVersion: string }, +): string | undefined { + const problems: string[] = []; + if (build.definition?.id !== expected.definitionId) { + problems.push(`definition id ${build.definition?.id ?? ""} (expected ${expected.definitionId})`); + } + if (build.sourceBranch !== expected.sourceBranch) { + problems.push(`sourceBranch '${build.sourceBranch ?? ""}' (expected '${expected.sourceBranch}')`); + } + if (build.sourceVersion !== expected.sourceVersion) { + problems.push(`sourceVersion '${build.sourceVersion ?? ""}' (expected '${expected.sourceVersion}')`); + } + if (problems.length === 0) return undefined; + return `does not match the requested queue parameters: ${problems.join("; ")}`; +} + +interface PollOneResult { + status: FixtureBuildStatus; + result?: string; + message?: string; + terminalProven: boolean; +} + +/** + * Poll a single queued build to a terminal state. Only ever resolves with + * `terminalProven: true` once ADO has positively confirmed the build + * reached `status === "completed"` — a `getBuild` error, a cancellation + * that can't be confirmed, or a grace-period expiry all resolve with + * `terminalProven: false` instead of guessing. + */ +async function pollOne( + client: FixtureBuildClient, + buildId: number, + expected: { definitionId: number; sourceBranch: string; sourceVersion: string }, + opts: { + deadlineAt: number; + cancelGraceMs: number; + pollMs: number; + abort: AbortSignalLike; + sleepImpl: (ms: number) => Promise; + log: (msg: string) => void; + }, +): Promise { + let cancelRequestedAt: number | undefined; + let mismatchReason: string | undefined; + let verified = false; + + const requestCancel = async (): Promise => { + opts.abort.signal(); + if (cancelRequestedAt === undefined) { + cancelRequestedAt = Date.now(); + await safeCancel(client, buildId, opts.log); + } + }; + + for (;;) { + let build: PolledBuild; + try { + build = await client.getBuild(buildId); + } catch (err) { + // A poll error never proves the build stopped. Request cancellation + // and keep retrying (bounded by the same cancellation grace period) + // in case a LATER call confirms a genuinely terminal state; only + // give up as "unproven" once that grace period elapses. + opts.log(`WARNING: getBuild(${buildId}) failed: ${errMessage(err)}`); + const hadCancelRequest = cancelRequestedAt !== undefined; + await requestCancel(); + if (hadCancelRequest && Date.now() - cancelRequestedAt! >= opts.cancelGraceMs) { + return { + status: "failed", + message: `build #${buildId}: getBuild kept failing and never confirmed a terminal state within the cancellation grace period: ${errMessage(err)}`, + terminalProven: false, + }; + } + await opts.sleepImpl(opts.pollMs); + continue; + } + + if (!verified) { + const mismatch = describeMismatch(build, expected); + if (mismatch) { + mismatchReason = `build #${buildId} ${mismatch}`; + opts.log(`WARNING: ${mismatchReason}`); + await requestCancel(); + } + verified = true; + } + + if (build.status === "completed") { + if (mismatchReason) { + return { status: "failed", result: build.result, message: mismatchReason, terminalProven: true }; + } + if (cancelRequestedAt !== undefined) { + return { status: "canceled", result: build.result, terminalProven: true }; + } + if (build.result === "succeeded") { + return { status: "succeeded", result: build.result, terminalProven: true }; + } + opts.abort.signal(); + return { status: "failed", result: build.result, terminalProven: true }; + } + + if (cancelRequestedAt === undefined) { + const ownTimeout = Date.now() >= opts.deadlineAt; + if (ownTimeout || opts.abort.aborted) { + if (ownTimeout) opts.abort.signal(); + await requestCancel(); + } + } else if (Date.now() - cancelRequestedAt >= opts.cancelGraceMs) { + return { + status: "timed-out", + message: + mismatchReason ?? + `build #${buildId} did not reach a terminal state within the cancellation grace period`, + terminalProven: false, + }; + } + + await opts.sleepImpl(opts.pollMs); + } +} + +/** Queue and poll every fixture request. See module docstring for the abort/cancel/terminal-proof contract. */ +export async function runFixtures( + client: FixtureBuildClient, + requests: readonly FixtureBuildRequest[], + opts: RunFixturesOptions, +): Promise { + const sleepImpl = opts.sleepImpl ?? defaultSleep; + const cancelGraceMs = opts.cancelGraceMs ?? Math.max(opts.pollMs * 6, 60_000); + + const results: FixtureBuildResult[] = requests.map((r) => ({ + name: r.name, + definitionId: r.definitionId, + status: "queue-failed", + durationMs: 0, + // Placeholder until the queueing attempt below resolves one way or the + // other; a failed attempt overwrites this with `false` (see comment at + // the catch site) and a successful attempt is overwritten again once + // Phase 2 polls it to a real terminal state. + terminalProven: false, + })); + + // Shared abort flag: created up front so a Phase 1 queue failure can + // immediately cancel any sibling already queued, instead of only taking + // effect once Phase 2 starts. + const abort = makeAbortFlag(); + + // ---- Phase 1: queue every fixture concurrently (never stops on a single failure) ---- + const queued: { index: number; buildId: number; start: number }[] = []; + await Promise.all( + requests.map(async (req, i) => { + const start = Date.now(); + try { + const build = await client.queueBuild(req.definitionId, { + sourceBranch: req.sourceBranch, + sourceVersion: req.sourceVersion, + }); + queued.push({ index: i, buildId: build.id, start }); + results[i] = { + ...results[i]!, + buildId: build.id, + url: client.buildUrl(build.id), + status: "queue-failed", // overwritten once polling resolves + }; + opts.log(`[${req.name}] queued build #${build.id} on ${req.sourceBranch}`); + } catch (err) { + abort.signal(); + // A queueBuild error is ambiguous — ADO may have accepted the + // request before the client lost the response — so this is never + // treated as proof no build exists. `terminalProven: false` forces + // the caller to retain the candidate ref; the stale-ref scanner's + // per-child-definition check is what can later prove no orphaned + // build was actually created. + results[i] = { + ...results[i]!, + message: errMessage(err), + durationMs: Date.now() - start, + terminalProven: false, + }; + opts.log(`[${req.name}] queue FAILED: ${errMessage(err)}`); + } + }), + ); + + // ---- Phase 2: poll all queued builds, bounded concurrency, shared abort ---- + const deadlineAt = Date.now() + opts.timeoutMs; + let nextQueuedIdx = 0; + + const worker = async (): Promise => { + for (;;) { + const qi = nextQueuedIdx++; + if (qi >= queued.length) return; + const q = queued[qi]!; + const req = requests[q.index]!; + try { + const outcome = await pollOne( + client, + q.buildId, + { definitionId: req.definitionId, sourceBranch: req.sourceBranch, sourceVersion: req.sourceVersion }, + { + deadlineAt, + cancelGraceMs, + pollMs: opts.pollMs, + abort, + sleepImpl, + log: opts.log, + }, + ); + results[q.index] = { + ...results[q.index]!, + status: outcome.status, + result: outcome.result, + message: outcome.message, + durationMs: Date.now() - q.start, + terminalProven: outcome.terminalProven, + }; + opts.log(`[${req.name}] build #${q.buildId} -> ${outcome.status}${outcome.result ? ` (${outcome.result})` : ""}`); + } catch (err) { + // pollOne itself is designed never to throw — this is a defensive + // backstop only. Treat as unproven, never as a confirmed stop. + abort.signal(); + results[q.index] = { + ...results[q.index]!, + status: "failed", + message: errMessage(err), + durationMs: Date.now() - q.start, + terminalProven: false, + }; + opts.log(`[${req.name}] poll FAILED: ${errMessage(err)}`); + } + } + }; + + const workerCount = Math.min(opts.concurrency, queued.length); + await Promise.all(Array.from({ length: workerCount }, worker)); + + const ok = results.every((r) => r.status === "succeeded"); + const allTerminal = results.every((r) => r.terminalProven); + return { ok, allTerminal, results }; +} diff --git a/scripts/ado-script/src/compiler-smoke-e2e/source.ts b/scripts/ado-script/src/compiler-smoke-e2e/source.ts new file mode 100644 index 00000000..ca94b229 --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/source.ts @@ -0,0 +1,105 @@ +/** + * Markdown front-matter transform: pins a fixture's `supply-chain:` block to + * the compiler candidate produced by the current orchestrator run. + * + * Parses only the *first* `---` YAML front-matter block (YAML 1.2, via the + * `yaml` package) and preserves the markdown body byte-for-byte — the + * transform never touches anything after the closing `---` delimiter line. + * + * Test-harness module; not shipped in `ado-script.zip`. + */ +import { Document, parseDocument, isMap } from "yaml"; + +/** The literal fields injected under `supply-chain.pipeline-artifact:`. */ +export interface PipelineArtifactValues { + readonly project: string; + readonly definitionId: number; + readonly runId: number; + readonly artifact: string; +} + +const FRONT_MATTER_RE = /^---\r?\n([\s\S]*?)\r?\n---\r?\n/; + +interface SplitMarkdown { + yamlText: string; + /** Everything after the closing `---` delimiter line, preserved verbatim. */ + body: string; +} + +/** Split a markdown source into its first YAML front-matter block and body. */ +export function splitFrontMatter(markdown: string): SplitMarkdown { + const match = markdown.match(FRONT_MATTER_RE); + if (!match) { + throw new Error("expected a leading '---' YAML front-matter block"); + } + const yamlText = match[1] ?? ""; + const body = markdown.slice(match[0].length); + return { yamlText, body }; +} + +function parseFrontMatter(yamlText: string): Document { + const doc = parseDocument(yamlText, { merge: false, version: "1.2" }); + if (doc.errors.length > 0) { + throw new Error( + `failed to parse YAML front matter: ${doc.errors.map((e) => e.message).join("; ")}`, + ); + } + return doc; +} + +/** + * Inject `supply-chain.pipeline-artifact` (literal project/definition-id/run- + * id/artifact) into a fixture's markdown source. + * + * Fails closed: + * - throws if `supply-chain.feed` or `supply-chain.pipeline-artifact` is + * already present (this transform must never silently override an + * existing binary source), + * - preserves `supply-chain.registry` untouched when present, + * - preserves the markdown body byte-for-byte, + * - removes only `on.schedule` (and `on` entirely once it has no + * remaining keys) so a staged candidate never self-schedules. + */ +export function injectPipelineArtifact( + markdown: string, + values: PipelineArtifactValues, +): string { + const { yamlText, body } = splitFrontMatter(markdown); + const doc = parseFrontMatter(yamlText); + + if (doc.hasIn(["supply-chain", "feed"])) { + throw new Error( + "fixture already defines supply-chain.feed; refusing to override with a pinned pipeline-artifact source", + ); + } + if (doc.hasIn(["supply-chain", "pipeline-artifact"])) { + throw new Error( + "fixture already defines supply-chain.pipeline-artifact; refusing to override", + ); + } + + // setIn creates any missing intermediate maps (e.g. a wholly absent + // `supply-chain:` key), and only touches this one nested key — any sibling + // `supply-chain.registry` is left exactly as authored. + doc.setIn( + ["supply-chain", "pipeline-artifact"], + doc.createNode({ + project: values.project, + "definition-id": values.definitionId, + "run-id": values.runId, + artifact: values.artifact, + }), + ); + + if (doc.hasIn(["on", "schedule"])) { + doc.deleteIn(["on", "schedule"]); + } + const on = doc.get("on", true); + if (isMap(on) && on.items.length === 0) { + doc.delete("on"); + } + + const rendered = doc.toString({ lineWidth: 0 }); + const frontMatter = rendered.endsWith("\n") ? rendered : `${rendered}\n`; + return `---\n${frontMatter}---\n${body}`; +} diff --git a/scripts/ado-script/src/compiler-smoke-e2e/stale.ts b/scripts/ado-script/src/compiler-smoke-e2e/stale.ts new file mode 100644 index 00000000..2a31b950 --- /dev/null +++ b/scripts/ado-script/src/compiler-smoke-e2e/stale.ts @@ -0,0 +1,208 @@ +/** + * Startup stale-ref scanner for leftover `refs/heads/ado-aw-smoke-candidate/*` + * branches on the mirror repo (e.g. left behind by a run that crashed before + * its own cleanup). + * + * Deliberately conservative: a candidate ref is only ever deleted when this + * scanner can PROVE, via the ADO Build REST API, that (a) the ref name + * encodes a build id of THIS orchestrator's own definition + * (`SYSTEM_DEFINITIONID`), (b) that build is old enough + * (`COMPILER_SMOKE_STALE_REF_HOURS`), and (c) that parent build is + * terminal. Note that (c) is NOT by itself proof the orchestration it + * started is done — an abruptly canceled/killed parent process can reach a + * terminal ADO build status while the five fixture builds it queued are + * still running. The scanner therefore also queries each of the five FIXED + * child definitions on the ref's exact branch (see + * `listBuildsForDefinitionBranch`) and inspects their statuses directly; + * only when every child build found there is ALSO terminal (or none exist) + * is a ref considered `"eligible"` for deletion. Any active child, or any + * error looking one up, marks the ref `"active"`/`"ambiguous"` instead. + * + * Every other case (unparseable ref, build not found, build belongs to a + * different definition, or any lookup error) is reported but never deleted + * — a fail-closed posture is preferred over a plausible-but-unverifiable + * guess at another run's identity. + * + * Test-harness module; not shipped in `ado-script.zip`. + */ +import { parseCandidateBuildId, type RemoteRef } from "./git.js"; + +export type StaleRefOutcome = "eligible" | "too-recent" | "active" | "ambiguous"; + +export interface StaleRefDecision { + ref: string; + sha: string; + outcome: StaleRefOutcome; + reason: string; +} + +export interface StaleScanBuild { + status?: string; + result?: string; + definition?: { id?: number }; + finishTime?: string; + queueTime?: string; +} + +export interface StaleScanClient { + getBuild(buildId: number): Promise; + /** List builds of `definitionId` on the exact candidate `branch` (see {@link AdoRest.listBuildsForDefinitionBranch}). */ + listBuildsForDefinitionBranch(definitionId: number, branch: string): Promise; +} + +export interface ScanStaleRefsOptions { + refs: readonly RemoteRef[]; + /** The checked-out base ref — defensive exclusion (should already be outside the candidate prefix). */ + baseRef: string; + /** This run's own about-to-be-created candidate ref — never treated as stale. */ + ownRef: string; + /** This orchestrator pipeline's own definition id (SYSTEM_DEFINITIONID). */ + definitionId: number; + /** + * Every fixed fixture ("child") pipeline definition id the orchestrator + * queues builds against. An orchestrator run completing (even abruptly, + * e.g. cancelled) does NOT prove these have also finished — they are + * independently queued builds. A candidate ref is only ever eligible for + * deletion once none of these definitions has a still-active build on + * that ref's exact branch. + */ + childDefinitionIds: readonly number[]; + staleRefHours: number; + client: StaleScanClient; + /** Injectable clock for deterministic tests. */ + now?: () => number; +} + +function parseTimestampMs(value: string | undefined): number | undefined { + if (!value) return undefined; + const ms = Date.parse(value); + return Number.isNaN(ms) ? undefined : ms; +} + +/** Evaluate every candidate ref and classify it. Never mutates anything. */ +export async function scanStaleRefs(opts: ScanStaleRefsOptions): Promise { + const now = opts.now ?? (() => Date.now()); + const thresholdMs = opts.staleRefHours * 60 * 60 * 1000; + const decisions: StaleRefDecision[] = []; + + for (const { ref, sha } of opts.refs) { + if (ref === opts.baseRef || ref === opts.ownRef) continue; + + const buildId = parseCandidateBuildId(ref); + if (buildId === undefined) { + decisions.push({ + ref, + sha, + outcome: "ambiguous", + reason: "ref name does not match the expected / pattern", + }); + continue; + } + + let build: StaleScanBuild; + try { + build = await opts.client.getBuild(buildId); + } catch (err) { + decisions.push({ + ref, + sha, + outcome: "ambiguous", + reason: `orchestrator build #${buildId} lookup failed: ${ + err instanceof Error ? err.message : String(err) + }`, + }); + continue; + } + + if (build.definition?.id !== opts.definitionId) { + decisions.push({ + ref, + sha, + outcome: "ambiguous", + reason: `build #${buildId} belongs to definition ${build.definition?.id ?? "?"}, not this orchestrator's own definition ${opts.definitionId}`, + }); + continue; + } + + if (build.status !== "completed") { + decisions.push({ + ref, + sha, + outcome: "active", + reason: `orchestrator build #${buildId} is still ${build.status ?? "in an unknown state"}`, + }); + continue; + } + + const finishedAtMs = parseTimestampMs(build.finishTime) ?? parseTimestampMs(build.queueTime); + if (finishedAtMs === undefined) { + decisions.push({ + ref, + sha, + outcome: "ambiguous", + reason: `orchestrator build #${buildId} is completed but has no usable finishTime/queueTime`, + }); + continue; + } + + const ageMs = now() - finishedAtMs; + if (ageMs < thresholdMs) { + decisions.push({ + ref, + sha, + outcome: "too-recent", + reason: `orchestrator build #${buildId} finished ${Math.round(ageMs / 3_600_000)}h ago, below the ${opts.staleRefHours}h threshold`, + }); + continue; + } + + // The orchestrator's own run is old enough and terminal, but that alone + // does not prove the five fixture ("child") builds it queued on this + // exact branch have also finished — an abruptly cancelled orchestrator + // run can "complete" while its queued children keep running. Check + // every fixed child definition on this exact branch before declaring + // the ref deletable; any lookup failure or non-completed child build + // fails closed. + let childLookupError: string | undefined; + let activeChildDefinitionId: number | undefined; + for (const childDefinitionId of opts.childDefinitionIds) { + let childBuilds: StaleScanBuild[]; + try { + childBuilds = await opts.client.listBuildsForDefinitionBranch(childDefinitionId, ref); + } catch (err) { + childLookupError = `child definition ${childDefinitionId} build lookup on ${ref} failed: ${ + err instanceof Error ? err.message : String(err) + }`; + break; + } + if (childBuilds.some((b) => b.status !== "completed")) { + activeChildDefinitionId = childDefinitionId; + break; + } + } + + if (childLookupError) { + decisions.push({ ref, sha, outcome: "ambiguous", reason: childLookupError }); + continue; + } + + if (activeChildDefinitionId !== undefined) { + decisions.push({ + ref, + sha, + outcome: "active", + reason: `fixture definition ${activeChildDefinitionId} still has a non-completed build on ${ref}`, + }); + continue; + } + + decisions.push({ + ref, + sha, + outcome: "eligible", + reason: `orchestrator build #${buildId} completed ${Math.round(ageMs / 3_600_000)}h ago and no fixture build is active on ${ref}`, + }); + } + + return decisions; +} diff --git a/src/compile/agentic_pipeline.rs b/src/compile/agentic_pipeline.rs index 709af95f..0a978f35 100644 --- a/src/compile/agentic_pipeline.rs +++ b/src/compile/agentic_pipeline.rs @@ -63,11 +63,11 @@ use anyhow::Result; use std::path::Path; +use super::common::PerJobPools; use super::common::{ self, ADO_BUILD_ID_SUFFIX, AWF_VERSION, HEADER_MARKER, MCPG_CONTAINER_NAME, MCPG_DOMAIN, MCPG_IMAGE, MCPG_PORT, MCPG_VERSION, image_ref, }; -use super::common::PerJobPools; use super::extensions::{CompileContext, CompilerExtension, Declarations, Extension, McpgConfig}; use super::ir::condition::{Condition, Expr}; use super::ir::env::EnvValue; @@ -80,6 +80,9 @@ use super::ir::step::{ use super::ir::tasks::azure_cli::{AzureCli, ScriptLocation, ScriptType}; use super::ir::tasks::docker_installer::DockerInstaller; use super::ir::tasks::download_package::DownloadPackage; +use super::ir::tasks::download_pipeline_artifact::{ + ArtifactSource, DownloadPipelineArtifact, RunVersion, +}; use super::ir::tasks::manual_validation::{ManualValidation, OnTimeout}; use super::ir::tasks::nuget_authenticate::NuGetAuthenticate; use super::ir::{ @@ -87,8 +90,9 @@ use super::ir::{ PrTrigger, RepositoryResource, Resources, Schedule, Triggers, }; use super::types::{ - ApprovalConfig, ApprovalOnTimeout, CheckoutFetchOpts, FrontMatter, OnConfig, PrMode, - ProviderToken, Repository as RepoCfg, SELF_CHECKOUT_ALIAS, SupplyChainConfig, + ApprovalConfig, ApprovalOnTimeout, CheckoutFetchOpts, FrontMatter, OnConfig, + PipelineArtifactConfig, PrMode, ProviderToken, Repository as RepoCfg, SELF_CHECKOUT_ALIAS, + SupplyChainConfig, }; /// The `safe-outputs:` key for the create-pull-request tool. Matches the kebab @@ -449,19 +453,13 @@ pub(crate) fn build_canonical_jobs( front_matter, cfg, &p, - &SafeOutputsVariant::automatic( - &reviewed, - create_pr_configured && !create_pr_reviewed, - ), + &SafeOutputsVariant::automatic(&reviewed, create_pr_configured && !create_pr_reviewed), )?); jobs.push(build_safeoutputs_job( front_matter, cfg, &p, - &SafeOutputsVariant::reviewed( - &reviewed, - create_pr_configured && create_pr_reviewed, - ), + &SafeOutputsVariant::reviewed(&reviewed, create_pr_configured && create_pr_reviewed), )?); } if let Some(teardown) = build_teardown_job(front_matter, cfg, &p)? { @@ -1263,7 +1261,11 @@ fn build_detection_job( condition: Some(Condition::Always), })); - let mut job = Job::new(prefix.id("Detection")?, "Detection", cfg.pools.detection.clone()); + let mut job = Job::new( + prefix.id("Detection")?, + "Detection", + cfg.pools.detection.clone(), + ); job.steps = steps; Ok(job) } @@ -1385,9 +1387,7 @@ fn create_pr_prepare_repos( for alias in &front_matter.checkout { repos.push(PreparePrBaseRepo { dir: format!("{working_directory}/{alias}"), - source_ref: repo_refs - .get(alias) - .cloned(), + source_ref: repo_refs.get(alias).cloned(), target_branch: pr_cfg.resolve_target_branch(alias, &repo_refs), }); } @@ -1762,7 +1762,11 @@ fn build_teardown_job( for user_step_val in &front_matter.teardown { steps.push(Step::RawYaml(step_to_raw_yaml_string(user_step_val)?)); } - let mut job = Job::new(prefix.id("Teardown")?, "Teardown", cfg.pools.teardown.clone()); + let mut job = Job::new( + prefix.id("Teardown")?, + "Teardown", + cfg.pools.teardown.clone(), + ); job.steps = steps; Ok(Some(job)) } @@ -2001,7 +2005,11 @@ fi\n" steps.push(Step::Bash(conclusion_step)); - let mut job = Job::new(prefix.id("Conclusion")?, "Conclusion", cfg.pools.conclusion.clone()); + let mut job = Job::new( + prefix.id("Conclusion")?, + "Conclusion", + cfg.pools.conclusion.clone(), + ); job.variables = conclusion_variables; job.steps = steps; // Keep Conclusion's "run regardless of upstream result" behavior, but do @@ -2229,6 +2237,129 @@ pub(crate) fn download_package_step( .into_step() } +/// Download one pinned candidate pipeline artifact to a compiler-owned path. +/// +/// User-controlled project and artifact values remain typed task inputs; only +/// validated positive numeric IDs are converted to task-input strings. +pub(crate) fn download_candidate_artifact_step( + config: &PipelineArtifactConfig, + display: impl Into, + target_path: &str, +) -> TaskStep { + DownloadPipelineArtifact::new(target_path) + .source(ArtifactSource::Specific) + .project(config.project.as_str()) + .pipeline(config.definition_id.to_string()) + .run_version(RunVersion::Specific) + .run_id(config.run_id.to_string()) + .artifact(config.artifact.as_str()) + .with_display_name(display) + .into_step() +} + +// Keep this free of single quotes: the generated Bash passes it as one +// single-quoted `python3 -c` argument so the multiline validator stays +// readable without requiring shell escaping. +const CANDIDATE_PROVENANCE_VALIDATOR_PY: &str = r#"import json +import sys + +provenance_path, definition_arg, build_arg = sys.argv[1:] +with open(provenance_path, encoding="utf-8") as handle: + provenance = json.load(handle) + +expected_definition = int(definition_arg) +expected_build = int(build_arg) +if provenance.get("schema") != "ado-aw/candidate-artifact/1": + sys.exit("candidate provenance schema must be ado-aw/candidate-artifact/1") + +definition = provenance.get("producer_definition_id") +build = provenance.get("producer_build_id") +if type(definition) is not int or definition != expected_definition: + sys.exit( + f"candidate producer_definition_id mismatch: " + f"expected {expected_definition}, got {definition!r}" + ) +if type(build) is not int or build != expected_build: + sys.exit( + f"candidate producer_build_id mismatch: expected {expected_build}, got {build!r}" + ) + +diagnostic = { + "schema": provenance["schema"], + "producer_definition_id": definition, + "producer_build_id": build, +} +for key in ( + "repository", + "source_ref", + "source_version", + "reason", + "compiler_version", + "awf_version", +): + if key in provenance: + diagnostic[key] = provenance[key] + +print("Validated candidate provenance:") +print(json.dumps(diagnostic, indent=2, sort_keys=True)) +"#; + +/// Build the staging script for one payload from a pinned candidate artifact. +/// +/// The producer contract is `schema: ado-aw/candidate-artifact/1` with numeric +/// `producer_definition_id` and `producer_build_id` fields. The script requires +/// exactly one payload, checksum manifest, and provenance document; verifies an +/// exact filename checksum entry; and validates producer identity before +/// running `tail`. +/// +/// SAFETY: shell-interpolated path, payload, and tail arguments must be +/// compiler-owned constants. Producer IDs are validated positive `u64`s. +pub(crate) fn stage_candidate_artifact_payload_bash( + config: &PipelineArtifactConfig, + staging: &str, + dest_dir: &str, + payload: &str, + tail: &str, +) -> String { + format!( + "set -eo pipefail\n\ + STAGING=\"{staging}\"\n\ + DEST=\"{dest_dir}\"\n\ + PAYLOAD_NAME='{payload}'\n\ + mkdir -p \"$DEST\"\n\ + \n\ + locate_one() {{\n \ + local name=\"$1\"\n \ + mapfile -d '' -t matches < <(find \"$STAGING\" -type f -name \"$name\" -print0)\n \ + if [ \"${{#matches[@]}}\" -ne 1 ]; then\n \ + echo \"##vso[task.complete result=Failed]Expected exactly one $name in candidate artifact, found ${{#matches[@]}}\" >&2\n \ + exit 1\n \ + fi\n \ + printf '%s' \"${{matches[0]}}\"\n\ + }}\n\ + \n\ + PAYLOAD=\"$(locate_one \"$PAYLOAD_NAME\")\"\n\ + CHK=\"$(locate_one checksums.txt)\"\n\ + PROVENANCE=\"$(locate_one provenance.json)\"\n\ + cp \"$PAYLOAD\" \"$DEST/$PAYLOAD_NAME\"\n\ + cp \"$CHK\" \"$DEST/checksums.txt\"\n\ + cp \"$PROVENANCE\" \"$DEST/provenance.json\"\n\ + \n\ + echo \"Verifying exact checksum entry for $PAYLOAD_NAME...\"\n\ + cd \"$DEST\" || exit 1\n\ + awk -v name=\"$PAYLOAD_NAME\" '\n \ + {{ candidate=$2; sub(/^\\*/, \"\", candidate); if (candidate == name) {{ count++; line=$0 }} }}\n \ + END {{ if (count != 1) exit 1; print line }}\n\ + ' checksums.txt | sha256sum -c -\n\ + \n\ + python3 -c '{provenance_validator}' provenance.json {definition_id} {run_id}\n\ + {tail}", + provenance_validator = CANDIDATE_PROVENANCE_VALIDATOR_PY, + definition_id = config.definition_id, + run_id = config.run_id, + ) +} + /// Bash body that locates a payload file inside a `DownloadPackage@1` staging /// directory — handling both the extracted-tree and raw-`.nupkg` delivery /// shapes — copies it (plus `checksums.txt`) into `dest_dir`, then runs the @@ -2294,6 +2425,28 @@ fn download_compiler_step( compiler_version: &str, supply_chain: Option<&SupplyChainConfig>, ) -> Vec { + if let Some(artifact) = supply_chain.and_then(|sc| sc.pipeline_artifact.as_ref()) { + let dest = "$(Pipeline.Workspace)/agentic-pipeline-compiler"; + let staging = "$(Pipeline.Workspace)/ado-aw-candidate/compiler"; + let tail = "mv ado-aw-linux-x64 ado-aw\n\ + chmod +x ado-aw\n"; + let body = stage_candidate_artifact_payload_bash( + artifact, + staging, + dest, + "ado-aw-linux-x64", + tail, + ); + return vec![ + Step::Task(download_candidate_artifact_step( + artifact, + "Download candidate artifact for agentic pipeline compiler", + staging, + )), + Step::Bash(bash("Stage candidate agentic pipeline compiler", body)), + ]; + } + if let Some(feed) = supply_chain.and_then(|sc| sc.feed.as_ref()) { let dest = "$(Pipeline.Workspace)/agentic-pipeline-compiler"; let staging = "$(Pipeline.Workspace)/agentic-pipeline-compiler/_pkg"; @@ -2432,6 +2585,28 @@ fn prepare_agent_prompt_step(agent_content: &str) -> Result { } fn download_awf_step(supply_chain: Option<&SupplyChainConfig>) -> Vec { + if let Some(artifact) = supply_chain.and_then(|sc| sc.pipeline_artifact.as_ref()) { + let dest = "$(Pipeline.Workspace)/awf"; + let staging = "$(Pipeline.Workspace)/ado-aw-candidate/awf"; + let tail = "mv awf-linux-x64 awf\n\ + chmod +x awf\n\ + echo \"##vso[task.prependpath]$(Pipeline.Workspace)/awf\"\n\ + ./awf --version\n"; + let body = + stage_candidate_artifact_payload_bash(artifact, staging, dest, "awf-linux-x64", tail); + return vec![ + Step::Task(download_candidate_artifact_step( + artifact, + "Download candidate artifact for AWF", + staging, + )), + Step::Bash(bash( + "Stage candidate AWF (Agentic Workflow Firewall)", + body, + )), + ]; + } + if let Some(feed) = supply_chain.and_then(|sc| sc.feed.as_ref()) { let dest = "$(Pipeline.Workspace)/awf"; let staging = "$(Pipeline.Workspace)/awf/_pkg"; @@ -2483,10 +2658,7 @@ fn download_awf_step(supply_chain: Option<&SupplyChainConfig>) -> Vec { ))] } -fn prepull_images_step( - include_mcpg: bool, - supply_chain: Option<&SupplyChainConfig>, -) -> Vec { +fn prepull_images_step(include_mcpg: bool, supply_chain: Option<&SupplyChainConfig>) -> Vec { let registry = supply_chain.and_then(|sc| sc.registry.as_ref()); let registry_base = registry.map(|r| r.name.as_str()); @@ -3838,7 +4010,16 @@ mod tests { byom_exclude_keys: vec![], detection_provider_env: vec![], }; - build_canonical_jobs(&fm, &extensions, &cfg, &ext_setup_steps, &ext_agent_prepare, &ext_agent_conditions, None).unwrap() + build_canonical_jobs( + &fm, + &extensions, + &cfg, + &ext_setup_steps, + &ext_agent_prepare, + &ext_agent_conditions, + None, + ) + .unwrap() } fn job_pool_by_id<'a>(jobs: &'a [super::super::ir::job::Job], id: &str) -> Option<&'a Pool> { diff --git a/src/compile/extensions/ado_script.rs b/src/compile/extensions/ado_script.rs index 5eefd690..a3539325 100644 --- a/src/compile/extensions/ado_script.rs +++ b/src/compile/extensions/ado_script.rs @@ -20,7 +20,10 @@ use anyhow::Result; use super::{CompileContext, CompilerExtension, Declarations, ExtensionPhase}; -use crate::compile::agentic_pipeline::{download_package_step, nuget_authenticate_step}; +use crate::compile::agentic_pipeline::{ + download_candidate_artifact_step, download_package_step, nuget_authenticate_step, + stage_candidate_artifact_payload_bash, +}; use crate::compile::filter_ir::{ GateContext, Severity, build_gate_step_typed, lower_pipeline_filters, lower_pr_filters, validate_pipeline_filters, validate_pr_filters, @@ -86,8 +89,7 @@ pub(crate) const EXEC_CONTEXT_PR_SYNTH_PATH: &str = /// Path to the safe-outputs approval-summary bundle inside the unpacked /// `ado-script.zip`. Runs at the end of the Agent job to render the proposed /// safe outputs to a sanitized markdown summary tab. -pub(crate) const APPROVAL_SUMMARY_PATH: &str = - "/tmp/ado-aw-scripts/ado-script/approval-summary.js"; +pub(crate) const APPROVAL_SUMMARY_PATH: &str = "/tmp/ado-aw-scripts/ado-script/approval-summary.js"; /// Path to the conclusion bundle inside the unpacked `ado-script.zip`. Runs in /// the always-on Conclusion job (see [`crate::compile::agentic_pipeline`]) to /// file pipeline-failure work items and diagnostic signals. Referenced both by @@ -98,15 +100,13 @@ pub(crate) const CONCLUSION_PATH: &str = "/tmp/ado-aw-scripts/ado-script/conclus /// Runs immediately before the Copilot invocation in the Agent and Detection /// jobs (issue #1316) to mint a GitHub App installation token and expose it as /// a masked same-job `GITHUB_APP_TOKEN` variable. -pub(crate) const GITHUB_APP_TOKEN_PATH: &str = - "/tmp/ado-aw-scripts/ado-script/github-app-token.js"; +pub(crate) const GITHUB_APP_TOKEN_PATH: &str = "/tmp/ado-aw-scripts/ado-script/github-app-token.js"; /// Path to the prepare-pr-base bundle inside the unpacked `ado-script.zip`. /// Runs in the Agent job before the Copilot invocation (issue #1413) when /// `create-pull-request` is configured, to fetch/deepen the target branch so /// the containerized SafeOutputs MCP server can compute a diff base on /// shallow-default agent pools. -pub(crate) const PREPARE_PR_BASE_PATH: &str = - "/tmp/ado-aw-scripts/ado-script/prepare-pr-base.js"; +pub(crate) const PREPARE_PR_BASE_PATH: &str = "/tmp/ado-aw-scripts/ado-script/prepare-pr-base.js"; const RELEASE_BASE_URL: &str = "https://github.com/githubnext/ado-aw/releases/download"; /// Single always-on extension that owns all `ado-script` bundle wiring. @@ -377,12 +377,9 @@ impl AdoScriptExtension { } } -/// Returns the two-step bundle as typed `Step`s: a -/// `Step::Task(UseNode@1)` plus a `Step::Bash` for the curl, sha256, -/// and unzip pipeline. When an internal feed is configured the bundle is -/// pulled from the Azure DevOps Artifacts feed (NuGet) instead of GitHub -/// Releases; the `.nupkg` is unzipped and `ado-script.zip` relocated, then -/// verified and unpacked exactly as in the GitHub path. +/// Returns typed install, acquisition, verification, and unpacking steps. +/// Internal feed and pinned pipeline artifact sources replace GitHub Releases +/// while preserving the final layout. /// /// Bundles are unpacked into `/tmp/ado-aw-scripts/` so the consumer /// references `/tmp/ado-aw-scripts/ado-script/.js`. Shared by the @@ -400,6 +397,32 @@ pub(crate) fn install_and_download_steps_typed( t }; + if let Some(artifact) = supply_chain.and_then(|sc| sc.pipeline_artifact.as_ref()) { + let staging = "/tmp/ado-aw-scripts/_artifact"; + let tail = "unzip -o /tmp/ado-aw-scripts/ado-script.zip -d /tmp/ado-aw-scripts/\n"; + let script = stage_candidate_artifact_payload_bash( + artifact, + staging, + "/tmp/ado-aw-scripts", + "ado-script.zip", + tail, + ); + let download = { + let mut t = download_candidate_artifact_step( + artifact, + "Download candidate artifact for ado-aw scripts", + staging, + ); + t.timeout = Some(std::time::Duration::from_secs(300)); + t.condition = Some(Condition::Succeeded); + t + }; + let mut stage = BashStep::new("Stage candidate ado-aw scripts", script) + .with_condition(Condition::Succeeded); + stage.timeout = Some(std::time::Duration::from_secs(300)); + return vec![Step::Task(install), Step::Task(download), Step::Bash(stage)]; + } + if let Some(feed) = supply_chain.and_then(|sc| sc.feed.as_ref()) { let connection = supply_chain.and_then(|sc| sc.feed_connection()); let mut auth = nuget_authenticate_step(connection); @@ -616,10 +639,7 @@ pub struct PreparePrBaseRepo { /// Emit one of the two create-pull-request preparation modes. The Agent job /// uses `PatchBase` to recover and verify the merge-base; SafeOutputs uses /// `TargetWorktree` to fetch only the target tip required by `git worktree add`. -pub fn prepare_pr_base_step_typed( - mode: PreparePrBaseMode, - repos: &[PreparePrBaseRepo], -) -> Step { +pub fn prepare_pr_base_step_typed(mode: PreparePrBaseMode, repos: &[PreparePrBaseRepo]) -> Step { let repo_flags: String = repos .iter() .map(|repo| { @@ -874,7 +894,8 @@ impl CompilerExtension for AdoScriptExtension { || self.github_app_token_active || self.prepare_pr_base_active { - agent_prepare_steps.extend(install_and_download_steps_typed(self.supply_chain.as_ref())); + agent_prepare_steps + .extend(install_and_download_steps_typed(self.supply_chain.as_ref())); if import_active { agent_prepare_steps.push(resolver_step_typed()); } @@ -1283,7 +1304,10 @@ mod tests { let Step::Bash(step) = github_app_token_step_typed(&cfg).unwrap() else { panic!("expected a bash step"); }; - assert_eq!(step.display_name, "Mint GitHub App token (Copilot engine auth)"); + assert_eq!( + step.display_name, + "Mint GitHub App token (Copilot engine auth)" + ); assert!( step.script .contains("node '/tmp/ado-aw-scripts/ado-script/github-app-token.js'"), @@ -1303,7 +1327,8 @@ mod tests { step.script ); assert!( - step.script.contains("--repositories 'octo-repo other-repo'"), + step.script + .contains("--repositories 'octo-repo other-repo'"), "repositories must be a single-quoted argv flag:\n{}", step.script ); @@ -1466,7 +1491,8 @@ mod tests { )); // api-url is an argv flag (non-secret), not an env var. assert!( - step.script.contains("revoke --api-url 'https://ghe.example.com/api/v3'"), + step.script + .contains("revoke --api-url 'https://ghe.example.com/api/v3'"), "revoke must pass api-url as an argv flag:\n{}", step.script ); @@ -1511,8 +1537,7 @@ mod tests { source_ref: None, target_branch: "main".to_string(), }]; - let Step::Bash(step) = - prepare_pr_base_step_typed(PreparePrBaseMode::PatchBase, &repos) + let Step::Bash(step) = prepare_pr_base_step_typed(PreparePrBaseMode::PatchBase, &repos) else { panic!("expected a bash step"); }; @@ -1548,8 +1573,7 @@ mod tests { source_ref: Some("refs/heads/feature".to_string()), target_branch: "release/2.x".to_string(), }]; - let Step::Bash(step) = - prepare_pr_base_step_typed(PreparePrBaseMode::PatchBase, &repos) + let Step::Bash(step) = prepare_pr_base_step_typed(PreparePrBaseMode::PatchBase, &repos) else { panic!("expected a bash step"); }; @@ -1582,8 +1606,7 @@ mod tests { target_branch: "gh-pages".to_string(), }, ]; - let Step::Bash(step) = - prepare_pr_base_step_typed(PreparePrBaseMode::PatchBase, &repos) + let Step::Bash(step) = prepare_pr_base_step_typed(PreparePrBaseMode::PatchBase, &repos) else { panic!("expected a bash step"); }; @@ -1594,9 +1617,8 @@ mod tests { step.script ); assert!( - step.script.contains( - "--repo-dir \"$(Build.SourcesDirectory)\" --target-branch 'main'" - ) + step.script + .contains("--repo-dir \"$(Build.SourcesDirectory)\" --target-branch 'main'") ); assert!( step.script.contains( diff --git a/src/compile/extensions/mod.rs b/src/compile/extensions/mod.rs index 378b87c4..b62b6081 100644 --- a/src/compile/extensions/mod.rs +++ b/src/compile/extensions/mod.rs @@ -14,7 +14,7 @@ //! 2. Implement [`CompilerExtension`] for it //! 3. Add a variant to the [`Extension`] enum and update [`collect_extensions`] -use anyhow::Result; +use anyhow::{Context, Result}; use serde::Serialize; use std::collections::BTreeMap; use std::fmt; @@ -88,6 +88,8 @@ use crate::ado::AdoContext; use crate::engine::{self, Engine}; use std::path::Path; +const COMPILE_REMOTE_URL_ENV: &str = "ADO_AW_COMPILE_REMOTE_URL"; + /// Metadata resolved at compile time from the local environment. /// /// Built once via [`CompileContext::new`] and passed to all extension @@ -99,8 +101,9 @@ pub struct CompileContext<'a> { pub agent_name: &'a str, /// The full front matter (for cross-cutting checks like bash access level). pub front_matter: &'a FrontMatter, - /// ADO context inferred from the git remote (org URL, project, repo name). - /// `None` if the compile directory has no ADO remote. + /// ADO context inferred from the git remote (org URL, project, repo name), + /// or from the compiler-owned `ADO_AW_COMPILE_REMOTE_URL` override. + /// `None` if neither source supplies an ADO remote. pub ado_context: Option, /// Resolved engine based on the front matter `engine:` field. pub engine: Engine, @@ -121,6 +124,9 @@ impl<'a> CompileContext<'a> { /// /// Resolves the engine implementation from front matter and infers ADO /// context from the git remote in the directory containing `input_path`. + /// Internal orchestration may set `ADO_AW_COMPILE_REMOTE_URL` when output + /// will execute from a different checkout remote; an invalid override is a + /// hard error rather than a silent fallback. /// Returns an error if the engine identifier is unsupported. pub async fn new(front_matter: &'a FrontMatter, input_path: &'a Path) -> Result { // `Path::parent()` is subtle: for a bare filename like `foo.md` @@ -135,7 +141,16 @@ impl<'a> CompileContext<'a> { _ => Path::new("."), }; let engine = engine::get_engine(front_matter.engine.engine_id())?; - let ado_context = Self::infer_ado_context(compile_dir).await; + let ado_context = match Self::ado_context_override()? { + Some(ctx) => { + log::info!( + "Using ADO compile context override for repository '{}'", + ctx.repo_name + ); + Some(ctx) + } + None => Self::infer_ado_context(compile_dir).await, + }; Ok(Self { agent_name: &front_matter.name, front_matter, @@ -154,6 +169,32 @@ impl<'a> CompileContext<'a> { }) } + fn ado_context_override() -> Result> { + let Some(value) = std::env::var_os(COMPILE_REMOTE_URL_ENV) else { + return Ok(None); + }; + let value = value + .into_string() + .map_err(|_| anyhow::anyhow!("{COMPILE_REMOTE_URL_ENV} must contain valid Unicode"))?; + Self::parse_ado_context_override(Some(&value)) + } + + fn parse_ado_context_override(value: Option<&str>) -> Result> { + let Some(value) = value else { + return Ok(None); + }; + let value = value.trim(); + anyhow::ensure!( + !value.is_empty(), + "{COMPILE_REMOTE_URL_ENV} must not be empty" + ); + crate::ado::parse_ado_remote(value) + .with_context(|| { + format!("{COMPILE_REMOTE_URL_ENV} must be an Azure DevOps Git remote URL") + }) + .map(Some) + } + async fn infer_ado_context(dir: &Path) -> Option { match crate::ado::get_git_remote_url(dir).await { Ok(url) => match crate::ado::parse_ado_remote(&url) { @@ -227,6 +268,42 @@ impl<'a> CompileContext<'a> { } } +#[cfg(test)] +mod compile_context_override_tests { + use super::CompileContext; + + #[test] + fn absent_override_has_no_context() { + assert!( + CompileContext::parse_ado_context_override(None) + .unwrap() + .is_none() + ); + } + + #[test] + fn ado_remote_override_resolves_exact_context() { + let ctx = CompileContext::parse_ado_context_override(Some( + "https://dev.azure.com/msazuresphere/AgentPlayground/_git/ado-aw-mirror", + )) + .unwrap() + .unwrap(); + assert_eq!(ctx.org_url, "https://dev.azure.com/msazuresphere"); + assert_eq!(ctx.project, "AgentPlayground"); + assert_eq!(ctx.repo_name, "ado-aw-mirror"); + } + + #[test] + fn invalid_override_fails_closed() { + for value in ["", "https://github.com/githubnext/ado-aw"] { + assert!( + CompileContext::parse_ado_context_override(Some(value)).is_err(), + "override should be rejected: {value:?}" + ); + } + } +} + // ────────────────────────────────────────────────────────────────────── // CompilerExtension trait // ────────────────────────────────────────────────────────────────────── diff --git a/src/compile/ir/tasks/download_pipeline_artifact.rs b/src/compile/ir/tasks/download_pipeline_artifact.rs index 3a4b688c..6ace83cc 100644 --- a/src/compile/ir/tasks/download_pipeline_artifact.rs +++ b/src/compile/ir/tasks/download_pipeline_artifact.rs @@ -268,7 +268,7 @@ mod tests { } #[test] - fn specific_run_inputs() { + fn latest_from_branch_inputs() { let t = DownloadPipelineArtifact::new("$(Pipeline.Workspace)/in") .source(ArtifactSource::Specific) .project("$(System.TeamProject)") @@ -294,4 +294,33 @@ mod tests { Some("true") ); } + + #[test] + fn exact_specific_run_inputs() { + let t = DownloadPipelineArtifact::new("$(Pipeline.Workspace)/in") + .source(ArtifactSource::Specific) + .project("AgentPlayground") + .pipeline("2560") + .run_version(RunVersion::Specific) + .run_id("630001") + .artifact("ado-aw-candidate") + .into_step(); + assert_eq!(t.inputs.get("source").map(String::as_str), Some("specific")); + assert_eq!( + t.inputs.get("runVersion").map(String::as_str), + Some("specific") + ); + assert_eq!( + t.inputs.get("project").map(String::as_str), + Some("AgentPlayground") + ); + assert_eq!(t.inputs.get("pipeline").map(String::as_str), Some("2560")); + assert_eq!(t.inputs.get("runId").map(String::as_str), Some("630001")); + assert_eq!( + t.inputs.get("artifact").map(String::as_str), + Some("ado-aw-candidate") + ); + assert!(!t.inputs.contains_key("tags")); + assert!(!t.inputs.contains_key("preferTriggeringPipeline")); + } } diff --git a/src/compile/types.rs b/src/compile/types.rs index 70af6034..1b38b280 100644 --- a/src/compile/types.rs +++ b/src/compile/types.rs @@ -129,7 +129,10 @@ impl PoolConfig { impl SanitizeConfigTrait for PoolConfig { fn sanitize_config_fields(&mut self) { self.name = self.name.as_deref().map(crate::sanitize::sanitize_config); - self.vm_image = self.vm_image.as_deref().map(crate::sanitize::sanitize_config); + self.vm_image = self + .vm_image + .as_deref() + .map(crate::sanitize::sanitize_config); self.os = self.os.as_deref().map(crate::sanitize::sanitize_config); self.demands = self .demands @@ -1427,7 +1430,11 @@ impl FrontMatter { check("safe-outputs.require-approval", v)?; } for tool in self.safe_output_tool_names() { - if let Some(v) = self.safe_outputs.get(tool).and_then(|c| c.get("require-approval")) { + if let Some(v) = self + .safe_outputs + .get(tool) + .and_then(|c| c.get("require-approval")) + { check(&format!("safe-outputs.{tool}.require-approval"), v)?; } } @@ -1634,9 +1641,10 @@ impl SanitizeConfigTrait for AdoAwDebugConfig { /// /// Lives under the optional `supply-chain:` top-level front-matter key. When /// present, the compiler mirrors the artifacts it normally fetches from -/// GitHub Releases / GHCR from an internal Azure DevOps Artifacts feed and/or -/// an internal container registry instead. `feed` and `registry` are -/// independent — a user may set either, both, or neither. +/// GitHub Releases / GHCR from an internal Azure DevOps Artifacts feed, a +/// pinned pipeline artifact, and/or an internal container registry instead. +/// `feed` and `pipeline-artifact` are mutually exclusive binary sources; +/// `registry` is independent. /// /// See `docs/supply-chain.md`. #[derive(Debug, Deserialize, Clone, Default)] @@ -1647,12 +1655,16 @@ pub struct SupplyChainConfig { /// fetched from GitHub Releases as today. #[serde(default)] pub feed: Option, + /// A pinned Azure DevOps pipeline artifact containing all three binary + /// payloads plus checksums and provenance. + #[serde(default, rename = "pipeline-artifact")] + pub pipeline_artifact: Option, /// Internal container registry (ACR login server) for the AWF and MCPG /// images. When omitted images are pulled from GHCR as today. #[serde(default)] pub registry: Option, - /// Shared fallback service connection used by whichever target does not - /// declare its own `service-connection`. + /// Shared fallback service connection for feed and registry targets. It + /// never applies to `pipeline-artifact`. #[serde(default, rename = "service-connection")] pub service_connection: Option, } @@ -1681,6 +1693,23 @@ pub struct RegistryConfig { pub service_connection: Option, } +/// A pinned Azure DevOps pipeline artifact containing the compiler, AWF, and +/// ado-script payloads. +#[derive(Debug, Deserialize, Clone)] +#[serde(deny_unknown_fields)] +pub struct PipelineArtifactConfig { + /// Azure DevOps project name or GUID containing the producer pipeline. + pub project: crate::secure::AdoProject, + /// Producer pipeline definition ID. + #[serde(rename = "definition-id")] + pub definition_id: u64, + /// Exact producer build/run ID. + #[serde(rename = "run-id")] + pub run_id: u64, + /// Pipeline artifact name. + pub artifact: crate::secure::ArtifactName, +} + impl<'de> Deserialize<'de> for FeedConfig { fn deserialize(deserializer: D) -> Result where @@ -1747,30 +1776,47 @@ impl SupplyChainConfig { /// Effective service connection for the feed (binary) mirror. /// /// Resolution: the feed's own `service-connection` → the top-level - /// `service-connection`. `None` means "authenticate with - /// `$(System.AccessToken)`" (valid for same-org feeds). + /// `service-connection`. The top-level fallback is considered only when + /// `feed` is configured; without a feed this returns `None`. When a feed + /// is present, `None` means "authenticate with `$(System.AccessToken)`" + /// (valid for same-org feeds). pub fn feed_connection(&self) -> Option<&str> { - self.feed - .as_ref() - .and_then(|f| f.service_connection.as_deref()) + let feed = self.feed.as_ref()?; + feed.service_connection + .as_deref() .or(self.service_connection.as_deref()) } /// Effective service connection for the registry (image) mirror. /// /// Resolution: the registry's own `service-connection` → the top-level - /// `service-connection`. `None` is invalid when `registry` is set (ACR has - /// no `System.AccessToken` path) — see [`SupplyChainConfig::validate`]. + /// `service-connection`. The top-level fallback is considered only when + /// `registry` is configured; without a registry this returns `None`. + /// `None` is invalid when `registry` is set (ACR has no + /// `System.AccessToken` path) — see [`SupplyChainConfig::validate`]. pub fn registry_connection(&self) -> Option<&str> { - self.registry - .as_ref() - .and_then(|r| r.service_connection.as_deref()) + let registry = self.registry.as_ref()?; + registry + .service_connection + .as_deref() .or(self.service_connection.as_deref()) } - /// Validate cross-field rules. Errors when `registry` is configured but no - /// service connection resolves for it. + /// Validate cross-field rules and positive producer IDs. pub fn validate(&self) -> anyhow::Result<()> { + if self.feed.is_some() && self.pipeline_artifact.is_some() { + anyhow::bail!( + "supply-chain.feed and supply-chain.pipeline-artifact are mutually exclusive" + ); + } + if let Some(artifact) = &self.pipeline_artifact { + if artifact.definition_id == 0 { + anyhow::bail!("supply-chain.pipeline-artifact.definition-id must be positive"); + } + if artifact.run_id == 0 { + anyhow::bail!("supply-chain.pipeline-artifact.run-id must be positive"); + } + } if self.registry.is_some() && self.registry_connection().is_none() { anyhow::bail!( "supply-chain.registry requires a service connection: set \ @@ -1785,9 +1831,8 @@ impl SupplyChainConfig { impl SanitizeConfigTrait for SupplyChainConfig { fn sanitize_config_fields(&mut self) { - // All fields are validated newtypes (FeedRef / HostName / - // ServiceConnection) constrained at deserialization time; there is - // nothing further to sanitize. + // String fields are validated newtypes constrained at deserialization + // time; numeric fields are checked by validate(). } } @@ -2814,6 +2859,65 @@ mod tests { assert!(sc2.validate().is_ok()); } + #[test] + fn test_supply_chain_pipeline_artifact_parses_and_validates() { + let sc = parse_supply_chain( + "supply-chain:\n pipeline-artifact:\n project: AgentPlayground\n definition-id: 2560\n run-id: 630001\n artifact: ado-aw-candidate\n service-connection: must-not-apply", + ); + let artifact = sc.pipeline_artifact.as_ref().unwrap(); + assert_eq!(artifact.project.as_str(), "AgentPlayground"); + assert_eq!(artifact.definition_id, 2560); + assert_eq!(artifact.run_id, 630001); + assert_eq!(artifact.artifact.as_str(), "ado-aw-candidate"); + assert!(sc.validate().is_ok()); + assert_eq!( + sc.feed_connection(), + None, + "top-level service connections do not apply to pipeline artifacts" + ); + assert_eq!( + sc.registry_connection(), + None, + "top-level service connections do not create a registry target" + ); + } + + #[test] + fn test_supply_chain_rejects_feed_pipeline_artifact_conflict() { + let sc = parse_supply_chain( + "supply-chain:\n feed: my-feed\n pipeline-artifact:\n project: AgentPlayground\n definition-id: 2560\n run-id: 630001\n artifact: ado-aw-candidate", + ); + assert!(sc.validate().is_err()); + } + + #[test] + fn test_supply_chain_rejects_zero_pipeline_artifact_ids() { + for field in ["definition-id", "run-id"] { + let yaml = format!( + "supply-chain:\n pipeline-artifact:\n project: AgentPlayground\n definition-id: {}\n run-id: {}\n artifact: ado-aw-candidate", + if field == "definition-id" { 0 } else { 2560 }, + if field == "run-id" { 0 } else { 630001 } + ); + let sc = parse_supply_chain(&yaml); + assert!(sc.validate().is_err(), "{field}=0 must be rejected"); + } + } + + #[test] + fn test_supply_chain_rejects_invalid_pipeline_artifact_values() { + for yaml in [ + "supply-chain:\n pipeline-artifact:\n project: bad/project\n definition-id: 2560\n run-id: 630001\n artifact: ado-aw-candidate", + "supply-chain:\n pipeline-artifact:\n project: AgentPlayground\n definition-id: -1\n run-id: 630001\n artifact: ado-aw-candidate", + "supply-chain:\n pipeline-artifact:\n project: AgentPlayground\n definition-id: 2560\n run-id: 630001\n artifact: bad/artifact", + "supply-chain:\n pipeline-artifact:\n project: AgentPlayground\n definition-id: 2560\n run-id: 630001\n artifact: ado-aw-candidate\n latest: true", + ] { + let v: serde_yaml::Value = serde_yaml::from_str(yaml).unwrap(); + let result: Result = + serde_yaml::from_value(v["supply-chain"].clone()); + assert!(result.is_err(), "invalid config must be rejected:\n{yaml}"); + } + } + #[test] fn test_supply_chain_rejects_unknown_fields() { let v: serde_yaml::Value = @@ -3062,9 +3166,13 @@ github-app-token: .unwrap() .clone(); assert_eq!(gat.app_id, "1234567"); - assert_eq!(gat.api_url.as_deref(), Some("https://ghe.example.com/api/v3")); + assert_eq!( + gat.api_url.as_deref(), + Some("https://ghe.example.com/api/v3") + ); assert!(gat.skip_token_revocation); - gat.validate().expect("numeric app-id + https api-url is valid"); + gat.validate() + .expect("numeric app-id + https api-url is valid"); } #[test] @@ -3109,7 +3217,11 @@ github-app-token: let ec = EngineConfig::default(); assert!(ec.github_app_token().is_none()); let opts: EngineOptions = serde_yaml::from_str("id: copilot").unwrap(); - assert!(EngineConfig::Full(Box::new(opts)).github_app_token().is_none()); + assert!( + EngineConfig::Full(Box::new(opts)) + .github_app_token() + .is_none() + ); } #[test] @@ -3121,7 +3233,10 @@ github-app-token: owner: octo-org "#; let opts: EngineOptions = serde_yaml::from_str(yaml).unwrap(); - let gat = EngineConfig::Full(Box::new(opts)).github_app_token().unwrap().clone(); + let gat = EngineConfig::Full(Box::new(opts)) + .github_app_token() + .unwrap() + .clone(); assert!(gat.repositories.is_empty()); gat.validate().unwrap(); } @@ -3280,8 +3395,7 @@ github-app-token: // accidentally introducing a required field or a non-None serde default. let yaml = "permissions: {}"; let fm: serde_yaml::Value = serde_yaml::from_str(yaml).unwrap(); - let pc: PermissionsConfig = - serde_yaml::from_value(fm["permissions"].clone()).unwrap(); + let pc: PermissionsConfig = serde_yaml::from_value(fm["permissions"].clone()).unwrap(); assert!(pc.read.is_none()); assert!(pc.write.is_none()); } @@ -4244,8 +4358,14 @@ Body assert_eq!(report_flag, Some(false)); // noop config with flat fields let noop = fm.safe_outputs.get("noop").unwrap(); - assert_eq!(noop.get("title-prefix").and_then(|v| v.as_str()), Some("[ado-aw] Agent noop")); - assert_eq!(noop.get("area-path").and_then(|v| v.as_str()), Some("MyProject\\MyTeam")); + assert_eq!( + noop.get("title-prefix").and_then(|v| v.as_str()), + Some("[ado-aw] Agent noop") + ); + assert_eq!( + noop.get("area-path").and_then(|v| v.as_str()), + Some("MyProject\\MyTeam") + ); } #[test] @@ -4298,7 +4418,9 @@ Body let (fm, _) = super::super::common::parse_markdown(implicit).unwrap(); let default_branch = crate::safe_outputs::CreatePrConfig::default().target_branch; assert_eq!( - fm.create_pr_config().unwrap().resolve_target_branch("self", &no_refs), + fm.create_pr_config() + .unwrap() + .resolve_target_branch("self", &no_refs), default_branch ); // Guard the shared default so a future rename can't silently make the @@ -4374,7 +4496,10 @@ safe-outputs: Body "#; let (fm, _) = super::super::common::parse_markdown(content).unwrap(); - let noop = fm.safe_outputs.get("noop").expect("noop key must be present"); + let noop = fm + .safe_outputs + .get("noop") + .expect("noop key must be present"); assert!( noop.is_object(), "noop: {{}} must parse as an object, not bool or null; got: {:?}", diff --git a/src/secure.rs b/src/secure.rs index c9f52565..0490962c 100644 --- a/src/secure.rs +++ b/src/secure.rs @@ -32,6 +32,7 @@ //! - [`Identifier`] — an engine agent/model identifier. //! - [`HostName`] — a DNS-style hostname. //! - [`RegistryRef`] — a container-registry host or base path. +//! - [`AdoProject`] — an Azure DevOps project name or GUID. //! - [`Version`] — a version string (`1.2.3`, `latest`). //! //! New safe-output tools that accept paths or identifiers should type those @@ -324,6 +325,47 @@ validated_string! { } } +validated_string! { + /// An Azure DevOps project name or GUID. + AdoProject, "project", |value: &str, label: &str| { + let bytes = value.as_bytes(); + let is_guid = bytes.len() == 36 + && bytes.iter().enumerate().all(|(index, byte)| { + if matches!(index, 8 | 13 | 18 | 23) { + *byte == b'-' + } else { + byte.is_ascii_hexdigit() + } + }); + let invalid_name_char = |c: char| { + c.is_control() + || matches!( + c, + ',' | '/' | '\\' | ':' | '*' | '?' | '"' | '<' | '>' | '|' + | ';' | '#' | '$' | '{' | '}' | '+' | '=' | '[' | ']' + ) + }; + let char_count = value.chars().count(); + let is_name = char_count > 0 + && char_count <= 64 + && value.trim() == value + && !value.starts_with('_') + && !value.starts_with('.') + && !value.ends_with('.') + && !value.chars().any(invalid_name_char); + + if is_guid || is_name { + Ok(()) + } else { + anyhow::bail!( + "{label} '{value}' must be an Azure DevOps project name \ + (1-64 characters, no reserved punctuation, leading '_' or \ + leading/trailing '.') or a canonical GUID" + ) + } + } +} + validated_string! { /// An Azure resource (audience) URL passed to /// `az account get-access-token --resource`. Validated to be a shell-safe, @@ -416,6 +458,16 @@ mod tests { assert!(ArtifactName::parse("a".repeat(101).as_str()).is_err()); } + #[test] + fn ado_project_name_or_guid_rules() { + assert!(AdoProject::parse("Agent Playground").is_ok()); + assert!(AdoProject::parse("12345678-1234-1234-1234-1234567890ab").is_ok()); + assert!(AdoProject::parse("_reserved").is_err()); + assert!(AdoProject::parse("bad/project").is_err()); + assert!(AdoProject::parse("bad$(macro)").is_err()); + assert!(AdoProject::parse("x".repeat(65)).is_err()); + } + #[test] fn deserialize_validates() { // Valid value round-trips. diff --git a/tests/ado-script-e2e/README.md b/tests/ado-script-e2e/README.md index 3d20110f..7432d02d 100644 --- a/tests/ado-script-e2e/README.md +++ b/tests/ado-script-e2e/README.md @@ -41,3 +41,10 @@ failing with an unresolved merge-base. The pipeline is path-filtered for relevant GitHub PR changes, manually queueable, and scheduled daily on `main`. Because excluded PRs receive no status, it is intentionally not configured as a global required GitHub check. + +Although this pipeline does not consume the candidate-smoke write connection, +it is a GitHub-backed AgentPlayground PR definition with protected repository +access. Its live ADO trigger must keep fork builds and fork secrets disabled; +definition `2544` is included in +[`tests/compiler-smoke-e2e/trigger-policy.json`](../compiler-smoke-e2e/trigger-policy.json) +and the trusted candidate-smoke audit. diff --git a/tests/compiler-smoke-e2e/README.md b/tests/compiler-smoke-e2e/README.md new file mode 100644 index 00000000..7585db42 --- /dev/null +++ b/tests/compiler-smoke-e2e/README.md @@ -0,0 +1,157 @@ +# Candidate compiler agentic smoke + +This suite validates compiler changes before release. It builds `ado-aw` and +`ado-script` from the exact pull-request or `main` commit, recompiles every +workflow under [`tests/safe-outputs/`](../safe-outputs/), and queues five fixed +AgentPlayground definitions against the regenerated YAML. + +It complements rather than replaces the release-backed daily smoke: + +| Lane | Source | Contract | +| --- | --- | --- | +| Release smoke | Checked-in lock files and GitHub release assets | Customers can download and run the latest released compiler/runtime. | +| Candidate smoke | Exact PR or nightly `main` checkout | Current compiler output compiles, passes integrity checks, and runs end to end before release. | + +The candidate lane is deliberately ephemeral. It must not commit its generated +locks back to `tests/safe-outputs/`; those checked-in locks remain generated by +the latest released compiler so their runtime integrity check and downloaded +binary cannot drift. + +## Flow + +1. [`azure-pipelines.yml`](azure-pipelines.yml) builds the candidate compiler + and every released ado-script bundle. +2. It downloads the AWF version pinned by that compiler and verifies the + upstream release checksum. +3. It publishes `ado-aw-candidate`, containing: + `ado-aw-linux-x64`, `awf-linux-x64`, `ado-script.zip`, `checksums.txt`, and + `provenance.json`. +4. The test-only `compiler-smoke-e2e` TypeScript harness creates a detached + worktree, adds an exact `supply-chain.pipeline-artifact` source to all five + smoke files, removes their schedules, recompiles them with the candidate + compiler, and runs `ado-aw check`. Both compiler subprocesses receive + `ADO_AW_COMPILE_REMOTE_URL` set to the target `ado-aw-mirror` URL, so + integrity-sensitive org/repository metadata matches the child checkout + without changing the worktree's Git configuration. +5. The harness pushes the worktree commit to + `refs/heads/ado-aw-smoke-candidate/` in the Azure Repo + `ado-aw-mirror`. +6. Five fixed child definitions are queued concurrently with both that ref and + its exact commit SHA. Each generated pipeline downloads and verifies the + artifact from the still-running producer build. +7. The harness waits for all children, cancels non-terminal runs after a + timeout, and deletes the per-run mirror ref only after every child is + terminal. + +Candidate branches are never pushed to GitHub. + +## Triggers + +- Same-repository pull requests to `main`, path-filtered to compiler/runtime + code and these fixtures. +- Nightly at 01:00 UTC on `main`. +- Manual runs for setup and diagnosis. + +The existing release smoke remains on its own schedule. + +## Fork security boundary + +This pipeline executes PR-built code with protected AgentPlayground resources, +so fork PRs are prohibited at the ADO definition boundary. Every credentialed +GitHub-backed AgentPlayground PR definition must persist: + +```text +forks.enabled = false +forks.allowSecrets = false +forks.allowFullAccessToken = false +pipelineTriggerSettings.buildsEnabledForForks = false +``` + +The YAML also rejects `System.PullRequest.IsFork` as defense in depth, but that +check is not the security boundary because a PR can modify its YAML. The live +definition settings must be audited after registration and periodically from a +trusted `main` run. Intentional PR definitions and scheduled/manual-only +definitions are held in [`trigger-policy.json`](trigger-policy.json); the +orchestrator always audits its own `System.DefinitionId` as PR-only in addition +to that manifest. + +## Fixed child definitions + +The definitions are registered against `ado-aw-mirror`, not GitHub. Their +default branch is the permanent inert ref +`refs/heads/ado-aw-smoke-candidate-base`, whose five lock-file paths contain +hand-authored `trigger: none`, `pr: none`, schedule-free placeholders. A child +therefore runs only when the orchestrator explicitly supplies a candidate ref. +The checked-in [`inert-child.yml`](inert-child.yml) is copied to those five +paths when the base ref is created. + +See [`REGISTERED.md`](REGISTERED.md) for definition IDs and variables. + +## Required permissions + +The principal behind `agent-playground-write`, used only after artifact +publication, needs: + +- Contribute/Create branch/Delete refs on `ado-aw-mirror`; +- Queue builds and Stop builds on the five child definitions; +- Read builds and artifacts in AgentPlayground. + +Child build identities need Code Read on `ado-aw-mirror` and Build Read on the +producer definition. Authorize `agent-playground-read` and +`agent-playground-write` only on children whose generated fixture references +them. + +Every child receives its own secret `GITHUB_TOKEN` for Copilot CLI +authentication, using the same credential policy as the release-backed smoke +definitions. The candidate failure-reporter child alone additionally receives +`ADO_AW_DEBUG_GITHUB_TOKEN`, with Issues read/write limited to +`jamesadevine/ado-aw-issues`. Do not put either token in a variable group or on +the orchestrator. ADO's server-side definition clone operation does not copy +secret values; provision them explicitly. + +## Definition variables + +Set these non-secret variables on the orchestrator: + +```text +COMPILER_SMOKE_CANARY_DEFINITION_ID +COMPILER_SMOKE_AZURE_CLI_DEFINITION_ID +COMPILER_SMOKE_NOOP_TARGET_DEFINITION_ID +COMPILER_SMOKE_JANITOR_DEFINITION_ID +COMPILER_SMOKE_REPORTER_DEFINITION_ID +``` + +Optional overrides: + +```text +COMPILER_SMOKE_ARTIFACT_NAME=ado-aw-candidate +COMPILER_SMOKE_MIRROR_REPO=ado-aw-mirror +COMPILER_SMOKE_CONCURRENCY=5 +COMPILER_SMOKE_CHILD_TIMEOUT_MS=7200000 +COMPILER_SMOKE_POLL_MS=10000 +COMPILER_SMOKE_STALE_REF_HOURS=24 +``` + +## Local validation + +The harness is deterministic under unit tests; live ADO calls are injected +behind fakes. + +```bash +cd scripts/ado-script +npm ci +npm run typecheck +npx vitest run src/compiler-smoke-e2e +npm run build:compiler-smoke-e2e +``` + +The full live contract requires AgentPlayground and the fixed definitions: + +1. the producer remains in progress after publishing its artifact; +2. every child downloads the exact producer `run-id`; +3. all five children succeed; +4. child provenance identifies the producer definition, build, and source SHA; +5. the per-run mirror ref is removed. + +Do not substitute `latest` artifact selection or transient feed versions if +the exact-run contract fails. diff --git a/tests/compiler-smoke-e2e/REGISTERED.md b/tests/compiler-smoke-e2e/REGISTERED.md new file mode 100644 index 00000000..2182894c --- /dev/null +++ b/tests/compiler-smoke-e2e/REGISTERED.md @@ -0,0 +1,53 @@ +# Registered candidate compiler smoke pipelines + +These definitions live in +[AgentPlayground](https://dev.azure.com/msazuresphere/AgentPlayground) under +`\compiler-smoke-e2e`. + +| Pipeline | Repository | YAML path | Definition ID | +| --- | --- | --- | ---: | +| `ado-aw candidate compiler smoke` | `githubnext/ado-aw` | `tests/compiler-smoke-e2e/azure-pipelines.yml` | `2559` | +| `Candidate compiler smoke - canary` | `ado-aw-mirror` | `tests/safe-outputs/canary.lock.yml` | `2554` | +| `Candidate compiler smoke - azure-cli` | `ado-aw-mirror` | `tests/safe-outputs/azure-cli.lock.yml` | `2555` | +| `Candidate compiler smoke - noop-target` | `ado-aw-mirror` | `tests/safe-outputs/noop-target.lock.yml` | `2556` | +| `Candidate compiler smoke - janitor` | `ado-aw-mirror` | `tests/safe-outputs/janitor.lock.yml` | `2557` | +| `Candidate compiler smoke - failure reporter` | `ado-aw-mirror` | `tests/safe-outputs/smoke-failure-reporter.lock.yml` | `2558` | + +All five child definitions use +`refs/heads/ado-aw-smoke-candidate-base` as their default branch. The ref is +permanent and inert; the harness never deletes it. Its seed commit is +`2b5fa7c336bd1f55a867cfc281e665472730b84c`. + +## Security record + +Before protected resources are authorized, record the verified fork settings +for the GitHub-backed orchestrator: + +```text +forks.enabled=false +forks.allowSecrets=false +forks.allowFullAccessToken=false +pipelineTriggerSettings.buildsEnabledForForks=false +``` + +GitHub-backed AgentPlayground definitions that intentionally validate PRs were +explicitly hardened on 2026-07-22: + +| Definition IDs | `forks.enabled` | `allowSecrets` | `allowFullAccessToken` | Effective fork builds | +| --- | --- | --- | --- | --- | +| `2544`, `2550` | `false` | `false` | `false` | `false` | +| `2559` | `false` | `false` | `false` | `false` | + +Release-smoke definitions `2545`-`2549` and scheduled trigger E2E definition +`2551` have no CI or PR trigger metadata. Their schedules/manual queues remain +independent of the candidate compiler PR lane. + +Definition `2559` uses the `github.com_githubnext` GitHub service connection +and stores the five child definition IDs as non-secret definition variables. + +Every child definition needs its own secret `GITHUB_TOKEN` for Copilot CLI +authentication. Definition `2558` additionally needs +`ADO_AW_DEBUG_GITHUB_TOKEN`; server-side definition cloning does not copy +secret values. + +No secret values belong in this file. diff --git a/tests/compiler-smoke-e2e/azure-pipelines.yml b/tests/compiler-smoke-e2e/azure-pipelines.yml new file mode 100644 index 00000000..cabda4e8 --- /dev/null +++ b/tests/compiler-smoke-e2e/azure-pipelines.yml @@ -0,0 +1,317 @@ +# Candidate compiler agentic-smoke orchestrator. +# +# Builds ado-aw and ado-script from the exact checked-out PR/main commit, +# publishes an immutable candidate artifact, recompiles all five agentic smoke +# workflows, stages them on a short-lived ado-aw-mirror ref, and waits for the +# five fixed child definitions to complete. + +trigger: none + +pr: + branches: + include: + - main + paths: + include: + - src/** + - ado-aw-derive/** + - scripts/ado-script/** + - tests/safe-outputs/** + - tests/compiler-smoke-e2e/** + - Cargo.toml + - Cargo.lock + +schedules: + - cron: "0 1 * * *" + displayName: Nightly candidate compiler smoke + branches: + include: + - main + always: true + +pool: + name: AZS-1ES-L-Playground-ubuntu-22.04 + +variables: + EFFECTIVE_COMPILER_SMOKE_ARTIFACT_NAME: $[ coalesce(variables['COMPILER_SMOKE_ARTIFACT_NAME'], 'ado-aw-candidate') ] + EFFECTIVE_COMPILER_SMOKE_MIRROR_REPO: $[ coalesce(variables['COMPILER_SMOKE_MIRROR_REPO'], 'ado-aw-mirror') ] + EFFECTIVE_COMPILER_SMOKE_CANARY_DEFINITION_ID: $[ coalesce(variables['COMPILER_SMOKE_CANARY_DEFINITION_ID'], '') ] + EFFECTIVE_COMPILER_SMOKE_AZURE_CLI_DEFINITION_ID: $[ coalesce(variables['COMPILER_SMOKE_AZURE_CLI_DEFINITION_ID'], '') ] + EFFECTIVE_COMPILER_SMOKE_NOOP_TARGET_DEFINITION_ID: $[ coalesce(variables['COMPILER_SMOKE_NOOP_TARGET_DEFINITION_ID'], '') ] + EFFECTIVE_COMPILER_SMOKE_JANITOR_DEFINITION_ID: $[ coalesce(variables['COMPILER_SMOKE_JANITOR_DEFINITION_ID'], '') ] + EFFECTIVE_COMPILER_SMOKE_REPORTER_DEFINITION_ID: $[ coalesce(variables['COMPILER_SMOKE_REPORTER_DEFINITION_ID'], '') ] + EFFECTIVE_CRATES_IO_FEED: $[ coalesce(variables['CRATES_IO_FEED'], 'sparse+https://pkgs.dev.azure.com/msazuresphere/AgentPlayground/_packaging/cargo/Cargo/index/') ] + +jobs: + - job: CandidateSmoke + displayName: Build and run candidate compiler smoke + timeoutInMinutes: 180 + steps: + - checkout: self + fetchDepth: 0 + fetchTags: false + persistCredentials: false + displayName: Checkout ado-aw candidate + + - script: | + set -euo pipefail + if [ "${SYSTEM_PULLREQUEST_ISFORK:-false}" = "True" ] || \ + [ "${SYSTEM_PULLREQUEST_ISFORK:-false}" = "true" ]; then + echo "Fork PRs may not run this credentialed pipeline." >&2 + exit 1 + fi + displayName: Reject fork PR execution + env: + SYSTEM_PULLREQUEST_ISFORK: $(System.PullRequest.IsFork) + + - script: | + set -euo pipefail + mkdir -p .cargo + { + printf '\n[registries]\ncargo = { index = "%s" }\n\n' "$(EFFECTIVE_CRATES_IO_FEED)" + printf '[source.crates-io]\nreplace-with = "cargo"\n' + } >> .cargo/config.toml + echo "----- .cargo/config.toml -----" + cat .cargo/config.toml + displayName: Write cargo config for internal crates.io feed + + - task: CargoAuthenticate@0 + inputs: + configFile: ".cargo/config.toml" + displayName: Authenticate with cargo (internal feeds) + + - task: RustInstaller@1 + inputs: + rustVersion: ms-stable + toolchainFeed: "https://pkgs.dev.azure.com/msazuresphere/AgentPlayground/_packaging/AgentPlaygroundRustTools%40Local/nuget/v3/index.json" + cratesIoFeedOverride: "$(EFFECTIVE_CRATES_IO_FEED)" + displayName: Install Rust toolchain + + - script: | + set -euo pipefail + cargo build --release --bin ado-aw + target/release/ado-aw --version + displayName: Build ado-aw candidate + + - task: UseNode@1 + inputs: + version: "20.x" + displayName: Use Node.js 20 + + - script: | + set -euo pipefail + npm ci + npm run build + npm run build:compiler-smoke-e2e + workingDirectory: scripts/ado-script + displayName: Build ado-script and candidate-smoke harness + + - script: | + set -euo pipefail + STAGE="$(Build.ArtifactStagingDirectory)/$(EFFECTIVE_COMPILER_SMOKE_ARTIFACT_NAME)" + mkdir -p "$STAGE" + cp target/release/ado-aw "$STAGE/ado-aw-linux-x64" + + ( + cd scripts + shopt -s nullglob + bundles=(ado-script/*.js) + if [ "${#bundles[@]}" -eq 0 ]; then + echo "No ado-script bundles were produced." >&2 + exit 1 + fi + zip -q -r "$STAGE/ado-script.zip" "${bundles[@]}" + ) + + VERSION_JSON="$(target/release/ado-aw catalog --kind versions --json)" + AWF_VERSION="$(printf '%s' "$VERSION_JSON" | jq -er '.versions.awf')" + AWF_BASE="https://github.com/github/gh-aw-firewall/releases/download/v${AWF_VERSION}" + curl -fsSL --retry 3 --retry-delay 5 \ + "$AWF_BASE/awf-linux-x64" -o "$STAGE/awf-linux-x64" + curl -fsSL --retry 3 --retry-delay 5 \ + "$AWF_BASE/checksums.txt" -o "$STAGE/awf-upstream-checksums.txt" + + EXPECTED_AWF="$( + awk '$2 == "awf-linux-x64" || $2 == "*awf-linux-x64" { print $1; exit }' \ + "$STAGE/awf-upstream-checksums.txt" + )" + ACTUAL_AWF="$(sha256sum "$STAGE/awf-linux-x64" | awk '{ print $1 }')" + test -n "$EXPECTED_AWF" + test "$EXPECTED_AWF" = "$ACTUAL_AWF" + rm "$STAGE/awf-upstream-checksums.txt" + + ( + cd "$STAGE" + sha256sum ado-aw-linux-x64 awf-linux-x64 ado-script.zip > checksums.txt + ) + + COMPILER_VERSION="$(target/release/ado-aw --version | awk '{ print $2 }')" + jq -n \ + --arg schema "ado-aw/candidate-artifact/1" \ + --arg repository "$(Build.Repository.Name)" \ + --arg source_ref "$(Build.SourceBranch)" \ + --arg source_version "$(Build.SourceVersion)" \ + --arg reason "$(Build.Reason)" \ + --arg project "$(System.TeamProject)" \ + --arg build_url "$(System.CollectionUri)$(System.TeamProject)/_build/results?buildId=$(Build.BuildId)" \ + --arg compiler_version "$COMPILER_VERSION" \ + --arg awf_version "$AWF_VERSION" \ + --argjson producer_definition_id "$(System.DefinitionId)" \ + --argjson producer_build_id "$(Build.BuildId)" \ + --arg ado_aw_sha256 "$(awk '$2 == "ado-aw-linux-x64" { print $1 }' "$STAGE/checksums.txt")" \ + --arg awf_sha256 "$(awk '$2 == "awf-linux-x64" { print $1 }' "$STAGE/checksums.txt")" \ + --arg ado_script_sha256 "$(awk '$2 == "ado-script.zip" { print $1 }' "$STAGE/checksums.txt")" \ + '{ + schema: $schema, + repository: $repository, + source_ref: $source_ref, + source_version: $source_version, + reason: $reason, + project: $project, + producer_definition_id: $producer_definition_id, + producer_build_id: $producer_build_id, + build_url: $build_url, + compiler_version: $compiler_version, + awf_version: $awf_version, + assets: { + "ado-aw-linux-x64": { + origin: "built-from-checkout", + sha256: $ado_aw_sha256 + }, + "awf-linux-x64": { + origin: "verified-upstream-release", + sha256: $awf_sha256 + }, + "ado-script.zip": { + origin: "built-from-checkout", + sha256: $ado_script_sha256 + } + } + }' > "$STAGE/provenance.json" + + jq -e \ + --argjson definition "$(System.DefinitionId)" \ + --argjson build "$(Build.BuildId)" \ + '.schema == "ado-aw/candidate-artifact/1" + and .producer_definition_id == $definition + and .producer_build_id == $build' \ + "$STAGE/provenance.json" >/dev/null + ( + cd "$STAGE" + for asset in ado-aw-linux-x64 awf-linux-x64 ado-script.zip; do + awk -v name="$asset" '$2 == name { print; found=1 } END { exit(found ? 0 : 1) }' \ + checksums.txt | sha256sum -c - + done + ) + cat "$STAGE/provenance.json" + displayName: Package candidate compiler supply chain + + - task: PublishPipelineArtifact@1 + inputs: + targetPath: "$(Build.ArtifactStagingDirectory)/$(EFFECTIVE_COMPILER_SMOKE_ARTIFACT_NAME)" + artifact: "$(EFFECTIVE_COMPILER_SMOKE_ARTIFACT_NAME)" + publishLocation: pipeline + displayName: Publish candidate compiler artifact + + - task: AzureCLI@2 + displayName: Acquire ADO orchestration token + inputs: + azureSubscription: agent-playground-write + scriptType: bash + scriptLocation: inlineScript + addSpnToEnvironment: true + inlineScript: | + set -euo pipefail + ADO_TOKEN=$(az account get-access-token \ + --resource 499b84ac-1321-427f-aa17-267ca6975798 \ + --query accessToken -o tsv) + echo "##vso[task.setvariable variable=SC_WRITE_TOKEN;issecret=true]$ADO_TOKEN" + + - script: | + set -euo pipefail + BASE="$(System.CollectionUri)$(System.TeamProject)/_apis/build/definitions" + POLICY="tests/compiler-smoke-e2e/trigger-policy.json" + PR_IDS="$( + jq -er ' + select(.schema == "ado-aw/agentplayground-trigger-policy/1") + | .pr_definition_ids[] + ' "$POLICY" + ) $(System.DefinitionId)" + SCHEDULED_ONLY_IDS="$( + jq -er ' + select(.schema == "ado-aw/agentplayground-trigger-policy/1") + | .scheduled_only_definition_ids[] + ' "$POLICY" + )" + + for id in $PR_IDS; do + JSON="$(curl -fsS \ + -H "Authorization: Bearer $SYSTEM_ACCESSTOKEN" \ + "$BASE/$id?api-version=7.1")" + if ! printf '%s' "$JSON" | jq -e ' + ([.triggers[]? | select(.triggerType == "continuousIntegration")] | length == 0) + and any( + .triggers[]?; + .triggerType == "pullRequest" + and .forks.enabled == false + and .forks.allowSecrets == false + and .forks.allowFullAccessToken == false + and .pipelineTriggerSettings.buildsEnabledForForks == false + )' >/dev/null; then + NAME="$(printf '%s' "$JSON" | jq -r '.name // "unknown"')" + echo "PR trigger policy drift detected for definition $id ($NAME)." >&2 + exit 1 + fi + done + + for id in $SCHEDULED_ONLY_IDS; do + JSON="$(curl -fsS \ + -H "Authorization: Bearer $SYSTEM_ACCESSTOKEN" \ + "$BASE/$id?api-version=7.1")" + if ! printf '%s' "$JSON" | jq -e ' + [.triggers[]? + | select( + .triggerType == "continuousIntegration" + or .triggerType == "pullRequest" + )] + | length == 0' >/dev/null; then + NAME="$(printf '%s' "$JSON" | jq -r '.name // "unknown"')" + echo "Scheduled-only trigger policy drift detected for definition $id ($NAME)." >&2 + exit 1 + fi + done + displayName: Audit AgentPlayground trigger policy + env: + SYSTEM_ACCESSTOKEN: $(SC_WRITE_TOKEN) + + - script: | + set -euo pipefail + mkdir -p "$(Build.ArtifactStagingDirectory)/compiler-smoke-diagnostics" + node scripts/ado-script/test-bin/compiler-smoke-e2e.js \ + 2>&1 | tee "$(Build.ArtifactStagingDirectory)/compiler-smoke-diagnostics/run.log" + displayName: Run all candidate compiler smoke pipelines + env: + SYSTEM_COLLECTIONURI: $(System.CollectionUri) + SYSTEM_TEAMPROJECT: $(System.TeamProject) + SYSTEM_ACCESSTOKEN: $(SC_WRITE_TOKEN) + BUILD_BUILDID: $(Build.BuildId) + BUILD_SOURCEBRANCH: $(Build.SourceBranch) + BUILD_SOURCEVERSION: $(Build.SourceVersion) + BUILD_SOURCESDIRECTORY: $(Build.SourcesDirectory) + SYSTEM_DEFINITIONID: $(System.DefinitionId) + COMPILER_SMOKE_ADO_AW_BIN: $(Build.SourcesDirectory)/target/release/ado-aw + COMPILER_SMOKE_ARTIFACT_NAME: $(EFFECTIVE_COMPILER_SMOKE_ARTIFACT_NAME) + COMPILER_SMOKE_MIRROR_REPO: $(EFFECTIVE_COMPILER_SMOKE_MIRROR_REPO) + COMPILER_SMOKE_CANARY_DEFINITION_ID: $(EFFECTIVE_COMPILER_SMOKE_CANARY_DEFINITION_ID) + COMPILER_SMOKE_AZURE_CLI_DEFINITION_ID: $(EFFECTIVE_COMPILER_SMOKE_AZURE_CLI_DEFINITION_ID) + COMPILER_SMOKE_NOOP_TARGET_DEFINITION_ID: $(EFFECTIVE_COMPILER_SMOKE_NOOP_TARGET_DEFINITION_ID) + COMPILER_SMOKE_JANITOR_DEFINITION_ID: $(EFFECTIVE_COMPILER_SMOKE_JANITOR_DEFINITION_ID) + COMPILER_SMOKE_REPORTER_DEFINITION_ID: $(EFFECTIVE_COMPILER_SMOKE_REPORTER_DEFINITION_ID) + + - task: PublishPipelineArtifact@1 + condition: always() + inputs: + targetPath: "$(Build.ArtifactStagingDirectory)/compiler-smoke-diagnostics" + artifact: compiler-smoke-diagnostics + publishLocation: pipeline + displayName: Publish candidate smoke diagnostics diff --git a/tests/compiler-smoke-e2e/inert-child.yml b/tests/compiler-smoke-e2e/inert-child.yml new file mode 100644 index 00000000..00b1391a --- /dev/null +++ b/tests/compiler-smoke-e2e/inert-child.yml @@ -0,0 +1,15 @@ +# Seed content copied to each candidate child definition's YAML path on +# refs/heads/ado-aw-smoke-candidate-base. The orchestrator always queues an +# explicit per-run ref; a run that reaches this file is a setup error. + +trigger: none +pr: none + +pool: + name: AZS-1ES-L-Playground-ubuntu-22.04 + +steps: + - bash: | + echo "Candidate compiler smoke must be queued with an explicit generated ref." >&2 + exit 1 + displayName: Reject inert candidate-smoke base diff --git a/tests/compiler-smoke-e2e/trigger-policy.json b/tests/compiler-smoke-e2e/trigger-policy.json new file mode 100644 index 00000000..14d6263e --- /dev/null +++ b/tests/compiler-smoke-e2e/trigger-policy.json @@ -0,0 +1,15 @@ +{ + "schema": "ado-aw/agentplayground-trigger-policy/1", + "pr_definition_ids": [ + 2544, + 2550 + ], + "scheduled_only_definition_ids": [ + 2545, + 2546, + 2547, + 2548, + 2549, + 2551 + ] +} diff --git a/tests/compiler_tests.rs b/tests/compiler_tests.rs index f6a7359b..89872480 100644 --- a/tests/compiler_tests.rs +++ b/tests/compiler_tests.rs @@ -4,8 +4,7 @@ use std::path::PathBuf; fn compiled_has_enabled_tool(compiled: &str, tool: &str) -> bool { let lines: Vec<&str> = compiled.lines().map(str::trim).collect(); lines.windows(2).any(|pair| { - pair[0] == "\"--enabled-tools\"," - && pair[1].trim_end_matches(',').trim_matches('"') == tool + pair[0] == "\"--enabled-tools\"," && pair[1].trim_end_matches(',').trim_matches('"') == tool }) } @@ -255,7 +254,10 @@ fn test_compiled_output_no_unreplaced_markers() { && compiled.contains("-p 127.0.0.1:8080:8080"), "MCPG must use the stable bridge-network topology" ); - for line in compiled.lines().filter(|line| line.contains("--allow-domains")) { + for line in compiled + .lines() + .filter(|line| line.contains("--allow-domains")) + { assert!( !line.contains("host.docker.internal"), "the agent egress allowlist must not expose the host: {line}" @@ -280,8 +282,9 @@ fn legacy_default_collection_remote_matches_modern_remote_lock_output() { let legacy = compile_lock_with_origin( "https://exampleorg.visualstudio.com/DefaultCollection/Example%20Project/_git/example-repo", ); - let modern = - compile_lock_with_origin("https://dev.azure.com/exampleorg/Example%20Project/_git/example-repo"); + let modern = compile_lock_with_origin( + "https://dev.azure.com/exampleorg/Example%20Project/_git/example-repo", + ); assert_eq!( legacy, modern, @@ -1761,8 +1764,7 @@ Call the noop tool exactly once. ); let agent = extract_job_block(&compiled, "Agent").expect("Agent job should exist"); - let detection = - extract_job_block(&compiled, "Detection").expect("Detection job should exist"); + let detection = extract_job_block(&compiled, "Detection").expect("Detection job should exist"); assert!( agent.contains( @@ -4494,7 +4496,10 @@ fn test_standalone_pipeline_trigger_compiled_output_is_valid_yaml() { compiled.contains("trigger: none"), "pipeline-trigger output should disable CI trigger" ); - assert!(compiled.contains("pr: none"), "pipeline-trigger output should disable PR trigger"); + assert!( + compiled.contains("pr: none"), + "pipeline-trigger output should disable PR trigger" + ); } // ─── --skip-integrity flag tests ───────────────────────────────────────── @@ -5359,12 +5364,16 @@ fn test_byom_provider_env_compiles_and_merges() { // The compiler-owned in-job mint step runs before the Copilot invocation in // BOTH the Agent and Detection jobs, authenticated by the service connection. assert_eq!( - compiled.matches("displayName: Acquire provider bearer token").count(), + compiled + .matches("displayName: Acquire provider bearer token") + .count(), 2, "provider.token must emit the AzureCLI@2 mint step in both the Agent and Detection jobs: {compiled}" ); assert_eq!( - compiled.matches("azureSubscription: my-arm-connection").count(), + compiled + .matches("azureSubscription: my-arm-connection") + .count(), 2, "mint step must authenticate via the configured service connection in both jobs: {compiled}" ); @@ -5412,14 +5421,18 @@ fn test_byom_provider_env_compiles_and_merges() { // from --env-all in both jobs, so it can never ride the passthrough into the // agent container even if ADO ever exposed it as a process env var. assert_eq!( - compiled.matches("--exclude-env AW_PROVIDER_BEARER_TOKEN").count(), + compiled + .matches("--exclude-env AW_PROVIDER_BEARER_TOKEN") + .count(), 2, "the intermediate mint secret AW_PROVIDER_BEARER_TOKEN must be excluded in both jobs: {compiled}" ); // A credential key NOT configured must NOT be excluded (BEARER_TOKEN is never // used by ado-aw — the sidecar only reads API_KEY). assert_eq!( - compiled.matches("--exclude-env COPILOT_PROVIDER_BEARER_TOKEN").count(), + compiled + .matches("--exclude-env COPILOT_PROVIDER_BEARER_TOKEN") + .count(), 0, "COPILOT_PROVIDER_BEARER_TOKEN must never be referenced: {compiled}" ); @@ -5443,31 +5456,30 @@ fn test_byom_provider_env_compiles_and_merges() { #[test] fn test_byok_provider_api_key_compiles_without_mint_step() { // Reuse the token fixture but swap the `token:` block for a static `api-key`. - let compiled = compile_fixture_tree_with_flags( - "byom-foundry-agent.md", - &[], - &[], - |contents| { - // Line-based rewrite (robust to CRLF/LF): drop the `token:` block and - // insert a static `api-key:` in its place. - let newline = if contents.contains("\r\n") { "\r\n" } else { "\n" }; - contents - .lines() - .filter(|l| { - let t = l.trim(); - t != "token:" && t != "service-connection: my-arm-connection" - }) - .map(|l| { - if l.trim() == "type: azure" { - format!("{l}{newline} api-key: $(FOUNDRY_API_KEY)") - } else { - l.to_string() - } - }) - .collect::>() - .join(newline) - }, - ); + let compiled = compile_fixture_tree_with_flags("byom-foundry-agent.md", &[], &[], |contents| { + // Line-based rewrite (robust to CRLF/LF): drop the `token:` block and + // insert a static `api-key:` in its place. + let newline = if contents.contains("\r\n") { + "\r\n" + } else { + "\n" + }; + contents + .lines() + .filter(|l| { + let t = l.trim(); + t != "token:" && t != "service-connection: my-arm-connection" + }) + .map(|l| { + if l.trim() == "type: azure" { + format!("{l}{newline} api-key: $(FOUNDRY_API_KEY)") + } else { + l.to_string() + } + }) + .collect::>() + .join(newline) + }); assert_valid_yaml(&compiled, "byom-foundry-agent.md (api-key)"); // api-key maps to COPILOT_PROVIDER_API_KEY in both jobs (unquoted macro). @@ -5492,7 +5504,9 @@ fn test_byok_provider_api_key_compiles_without_mint_step() { "the api-key path must rely on AWF's always-on api-proxy: {compiled}" ); assert_eq!( - compiled.matches("--exclude-env COPILOT_PROVIDER_API_KEY").count(), + compiled + .matches("--exclude-env COPILOT_PROVIDER_API_KEY") + .count(), 2, "the api-key credential must be excluded from --env-all in both jobs: {compiled}" ); @@ -7025,6 +7039,210 @@ fn job_block<'a>(compiled: &'a str, job: &str) -> &'a str { } } +fn collect_pipeline_artifact_task_inputs( + value: &serde_yaml::Value, + inputs: &mut Vec, +) { + match value { + serde_yaml::Value::Mapping(map) => { + if map.get(yaml_key("task")).and_then(|v| v.as_str()) + == Some("DownloadPipelineArtifact@2") + && map + .get(yaml_key("inputs")) + .and_then(|v| v.as_mapping()) + .and_then(|task_inputs| task_inputs.get(yaml_key("artifact"))) + .and_then(|v| v.as_str()) + == Some("ado-aw-candidate") + { + inputs.push( + map.get(yaml_key("inputs")) + .and_then(|v| v.as_mapping()) + .unwrap() + .clone(), + ); + } + for child in map.values() { + collect_pipeline_artifact_task_inputs(child, inputs); + } + } + serde_yaml::Value::Sequence(sequence) => { + for child in sequence { + collect_pipeline_artifact_task_inputs(child, inputs); + } + } + _ => {} + } +} + +/// One pinned pipeline artifact replaces all three GitHub binary sources and +/// is acquired only through exact, typed DownloadPipelineArtifact@2 inputs. +#[test] +fn test_supply_chain_pipeline_artifact_reroutes_all_binary_consumers() { + let compiled = compile_fixture("pipeline-artifact-agent.md"); + assert_valid_yaml(&compiled, "pipeline-artifact-agent.md"); + + assert!( + !compiled.contains("github.com/githubnext/ado-aw/releases"), + "compiler and ado-script release URLs must be absent" + ); + assert!( + !compiled.contains("github.com/github/gh-aw-firewall/releases"), + "AWF release URLs must be absent" + ); + assert!( + !compiled.contains("DownloadPackage@1") + && !compiled.contains("NuGetAuthenticate@1") + && !compiled.contains("nuGetServiceConnections"), + "pipeline artifacts must not emit feed acquisition or authentication" + ); + + let document = parse_compiled_yaml(&compiled); + let mut task_inputs = Vec::new(); + collect_pipeline_artifact_task_inputs(&document, &mut task_inputs); + assert!( + task_inputs.len() >= 7, + "compiler, AWF, and ado-script consumers across isolated jobs must each acquire the pinned artifact" + ); + let allowed_targets = [ + "$(Pipeline.Workspace)/ado-aw-candidate/compiler", + "$(Pipeline.Workspace)/ado-aw-candidate/awf", + "/tmp/ado-aw-scripts/_artifact", + ]; + for inputs in task_inputs { + for (key, expected) in [ + ("source", "specific"), + ("project", "AgentPlayground"), + ("pipeline", "2560"), + ("runVersion", "specific"), + ("runId", "630001"), + ("artifact", "ado-aw-candidate"), + ] { + assert_eq!( + inputs.get(yaml_key(key)).and_then(|v| v.as_str()), + Some(expected), + "candidate task input {key} must be exact" + ); + } + assert!( + allowed_targets.contains( + &inputs + .get(yaml_key("targetPath")) + .and_then(|v| v.as_str()) + .expect("candidate task needs compiler-owned targetPath") + ), + "candidate task targetPath must be compiler-owned" + ); + assert_eq!( + inputs.len(), + 7, + "candidate acquisition must not add latest/tags/preferTriggering inputs" + ); + } + + for marker in [ + "PAYLOAD_NAME='ado-aw-linux-x64'", + "PAYLOAD_NAME='awf-linux-x64'", + "PAYLOAD_NAME='ado-script.zip'", + "locate_one checksums.txt", + "locate_one provenance.json", + "awk -v name=\"$PAYLOAD_NAME\"", + "candidate == name", + "sha256sum -c -", + "ado-aw/candidate-artifact/1", + "producer_definition_id", + "producer_build_id", + "\"repository\",", + "\"source_ref\",", + "\"source_version\",", + "\"reason\",", + "\"compiler_version\",", + "\"awf_version\",", + "type(definition) is not int", + "type(build) is not int", + "Validated candidate provenance:", + ] { + assert!( + compiled.contains(marker), + "candidate checksum/provenance staging marker missing: {marker}" + ); + } + + for (job, expected_payloads) in [ + ( + "Agent", + &["ado-aw-linux-x64", "awf-linux-x64", "ado-script.zip"][..], + ), + ("Detection", &["ado-aw-linux-x64", "awf-linux-x64"][..]), + ("SafeOutputs", &["ado-aw-linux-x64", "ado-script.zip"][..]), + ("Conclusion", &["ado-script.zip"][..]), + ] { + let block = job_block(&compiled, job); + for payload in expected_payloads { + assert!( + block.contains(payload), + "{job} must route its {payload} consumer through the candidate artifact" + ); + } + } +} + +/// Registry routing is independent and can accompany the pinned binary source. +#[test] +fn test_supply_chain_pipeline_artifact_with_registry() { + let source = r#"--- +name: "Candidate and Registry" +description: "pinned binaries and mirrored images" +supply-chain: + pipeline-artifact: + project: AgentPlayground + definition-id: 2560 + run-id: 630001 + artifact: ado-aw-candidate + registry: + name: myacr.azurecr.io/candidate + service-connection: acr-conn +--- + +## Body +"#; + let (ok, compiled, stderr) = compile_inline_source("candidate-registry", source); + assert!(ok, "pipeline-artifact + registry should compile: {stderr}"); + assert!(compiled.contains("source: specific")); + assert!(compiled.contains("runId: '630001'")); + assert!(compiled.contains("docker pull myacr.azurecr.io/candidate/squid:")); + assert!(compiled.contains("azureSubscription: acr-conn")); + assert!(!compiled.contains("docker pull ghcr.io")); + assert!(!compiled.contains("DownloadPackage@1")); + assert!(!compiled.contains("NuGetAuthenticate@1")); + assert!(!compiled.contains("github.com/githubnext/ado-aw/releases")); + assert!(!compiled.contains("github.com/github/gh-aw-firewall/releases")); +} + +/// The canonical acquisition path is shared by standalone and every wrapper +/// target. +#[test] +fn test_supply_chain_pipeline_artifact_all_compile_targets() { + for target in [None, Some("1es"), Some("job"), Some("stage")] { + let target_field = target + .map(|value| format!("target: {value}\n")) + .unwrap_or_default(); + let source = format!( + "---\nname: Candidate Target\ndescription: target coverage\n{target_field}supply-chain:\n pipeline-artifact:\n project: AgentPlayground\n definition-id: 2560\n run-id: 630001\n artifact: ado-aw-candidate\n---\n\n## Body\n" + ); + let name = target.unwrap_or("standalone"); + let (ok, compiled, stderr) = compile_inline_source(&format!("candidate-{name}"), &source); + assert!(ok, "{name} target should compile: {stderr}"); + assert!( + compiled.contains("- task: DownloadPipelineArtifact@2") + && compiled.contains("artifact: ado-aw-candidate") + && compiled.contains("runVersion: specific"), + "{name} target must use the canonical pinned artifact acquisition" + ); + assert!(!compiled.contains("github.com/githubnext/ado-aw/releases")); + assert!(!compiled.contains("github.com/github/gh-aw-firewall/releases")); + } +} + /// With `supply-chain.feed` + `supply-chain.registry` configured, every /// GitHub/GHCR fetch is rerouted to the internal feed + registry while /// checksum verification is preserved. @@ -7871,7 +8089,10 @@ repos: assert!(ok, "self full-history tuning should compile: {stderr}"); let self_checkouts = compiled.matches("- checkout: self").count(); - assert!(self_checkouts >= 2, "expected multiple self checkouts:\n{compiled}"); + assert!( + self_checkouts >= 2, + "expected multiple self checkouts:\n{compiled}" + ); assert_eq!( compiled.matches("fetchDepth: 0").count(), self_checkouts, @@ -8260,13 +8481,28 @@ fn variable_groups_standalone_emits_group_imports_in_order() { "vg-standalone", "---\nname: vg-standalone\ndescription: variable group import test\nvariable-groups:\n - Agentic Workflows\n - Shared Secrets\n---\n\n## Agent\n\nDo work.\n", ); - assert!(ok, "standalone compile with variable-groups should succeed.\nstderr: {stderr}"); - assert!(compiled.contains("variables:"), "top-level variables: block must be present:\n{compiled}"); - assert!(compiled.contains("- group: Agentic Workflows"), "first group import missing:\n{compiled}"); - assert!(compiled.contains("- group: Shared Secrets"), "second group import missing:\n{compiled}"); + assert!( + ok, + "standalone compile with variable-groups should succeed.\nstderr: {stderr}" + ); + assert!( + compiled.contains("variables:"), + "top-level variables: block must be present:\n{compiled}" + ); + assert!( + compiled.contains("- group: Agentic Workflows"), + "first group import missing:\n{compiled}" + ); + assert!( + compiled.contains("- group: Shared Secrets"), + "second group import missing:\n{compiled}" + ); let first = compiled.find("Agentic Workflows").unwrap(); let second = compiled.find("Shared Secrets").unwrap(); - assert!(first < second, "group imports must preserve declaration order"); + assert!( + first < second, + "group imports must preserve declaration order" + ); } /// A 1ES pipeline (`target: 1es`) also emits the `variables:` group import at @@ -8277,10 +8513,22 @@ fn variable_groups_onees_emits_group_imports() { "vg-onees", "---\nname: vg-onees\ndescription: variable group import test\ntarget: 1es\nvariable-groups:\n - Agentic Workflows\n---\n\n## Agent\n\nDo work.\n", ); - assert!(ok, "1es compile with variable-groups should succeed.\nstderr: {stderr}"); - assert!(compiled.contains("extends:"), "1ES pipeline must still emit extends:\n{compiled}"); - assert!(compiled.contains("variables:"), "top-level variables: block must be present:\n{compiled}"); - assert!(compiled.contains("- group: Agentic Workflows"), "group import missing:\n{compiled}"); + assert!( + ok, + "1es compile with variable-groups should succeed.\nstderr: {stderr}" + ); + assert!( + compiled.contains("extends:"), + "1ES pipeline must still emit extends:\n{compiled}" + ); + assert!( + compiled.contains("variables:"), + "top-level variables: block must be present:\n{compiled}" + ); + assert!( + compiled.contains("- group: Agentic Workflows"), + "group import missing:\n{compiled}" + ); } /// `target: job` cannot carry pipeline-level `variables:`, so a non-empty @@ -8292,8 +8540,14 @@ fn variable_groups_rejected_for_job_target() { "---\nname: vg-job\ndescription: variable group import test\ntarget: job\nvariable-groups:\n - Agentic Workflows\n---\n\n## Agent\n\nDo work.\n", ); assert!(!ok, "job target with variable-groups must fail to compile"); - assert!(stderr.contains("variable-groups"), "error must mention variable-groups:\n{stderr}"); - assert!(stderr.contains("target: job"), "error must name the job target:\n{stderr}"); + assert!( + stderr.contains("variable-groups"), + "error must mention variable-groups:\n{stderr}" + ); + assert!( + stderr.contains("target: job"), + "error must name the job target:\n{stderr}" + ); } /// `target: stage` is rejected for the same reason as `target: job`. @@ -8303,9 +8557,18 @@ fn variable_groups_rejected_for_stage_target() { "vg-stage", "---\nname: vg-stage\ndescription: variable group import test\ntarget: stage\nvariable-groups:\n - Agentic Workflows\n---\n\n## Agent\n\nDo work.\n", ); - assert!(!ok, "stage target with variable-groups must fail to compile"); - assert!(stderr.contains("variable-groups"), "error must mention variable-groups:\n{stderr}"); - assert!(stderr.contains("target: stage"), "error must name the stage target:\n{stderr}"); + assert!( + !ok, + "stage target with variable-groups must fail to compile" + ); + assert!( + stderr.contains("variable-groups"), + "error must mention variable-groups:\n{stderr}" + ); + assert!( + stderr.contains("target: stage"), + "error must name the stage target:\n{stderr}" + ); } /// A group name that carries an ADO macro expression is rejected as an unsafe @@ -8317,8 +8580,10 @@ fn variable_groups_rejects_injection_name() { "---\nname: vg-inject\ndescription: variable group import test\nvariable-groups:\n - \"$(evil)\"\n---\n\n## Agent\n\nDo work.\n", ); assert!(!ok, "an injection-bearing group name must fail to compile"); - assert!(stderr.contains("variable group name") || stderr.contains("variable-groups entry"), - "error must explain the invalid group name:\n{stderr}"); + assert!( + stderr.contains("variable group name") || stderr.contains("variable-groups entry"), + "error must explain the invalid group name:\n{stderr}" + ); } /// A workflow that references a GitHub App private key held in a project-level @@ -8331,10 +8596,18 @@ fn variable_groups_with_github_app_token_compiles_without_patch() { "vg-ghapp", "---\nname: vg-ghapp\ndescription: variable group + github app token\nvariable-groups:\n - Agentic Workflows\nengine:\n id: copilot\n github-app-token:\n app-id: 1234567\n owner: octo-org\n repositories: [octo-repo]\n private-key: AGENTIC_WORKFLOWS_GITHUB_APP_PRIVATE_KEY\n---\n\n## Agent\n\nDo work.\n", ); - assert!(ok, "compile with variable-groups + github-app-token should succeed.\nstderr: {stderr}"); - assert!(compiled.contains("- group: Agentic Workflows"), "group import missing:\n{compiled}"); - assert!(compiled.contains("$(AGENTIC_WORKFLOWS_GITHUB_APP_PRIVATE_KEY)"), - "private-key macro reference missing:\n{compiled}"); + assert!( + ok, + "compile with variable-groups + github-app-token should succeed.\nstderr: {stderr}" + ); + assert!( + compiled.contains("- group: Agentic Workflows"), + "group import missing:\n{compiled}" + ); + assert!( + compiled.contains("$(AGENTIC_WORKFLOWS_GITHUB_APP_PRIVATE_KEY)"), + "private-key macro reference missing:\n{compiled}" + ); } // ───────────────────────────────────────────────────────────────────── @@ -8519,8 +8792,7 @@ fn test_create_pull_request_safeoutputs_prepare_step_covers_all_checkout_repos() ); let safeoutputs = job_block(&compiled, "SafeOutputs"); assert!( - safeoutputs - .contains("--repo-dir \"$(Build.SourcesDirectory)\" --target-branch 'main'"), + safeoutputs.contains("--repo-dir \"$(Build.SourcesDirectory)\" --target-branch 'main'"), "self must target the literal default 'main' in the SafeOutputs job:\n{safeoutputs}" ); assert!( diff --git a/tests/executor-e2e/README.md b/tests/executor-e2e/README.md index 408a5de6..43a23f1f 100644 --- a/tests/executor-e2e/README.md +++ b/tests/executor-e2e/README.md @@ -123,6 +123,9 @@ In `https://dev.azure.com/msazuresphere/AgentPlayground`: `githubnext` service connection → existing YAML → `tests/executor-e2e/azure-pipelines.yml`. Place it in a `\executor-e2e` folder and skip the first run until variables are configured. + In the live pull-request trigger settings, disable builds from forks and + disable fork access to secrets/full tokens. Definition `2550` is audited by + `tests/compiler-smoke-e2e/trigger-policy.json`. 2. **Grant the principal behind `agent-playground-write` write access** on the `agent-definitions` repo (Contribute, Create branch, Contribute to PRs) and on Build (add tags). The YAML maps its AAD token to diff --git a/tests/fixtures/pipeline-artifact-agent.md b/tests/fixtures/pipeline-artifact-agent.md new file mode 100644 index 00000000..5e0a93b6 --- /dev/null +++ b/tests/fixtures/pipeline-artifact-agent.md @@ -0,0 +1,18 @@ +--- +name: "Pipeline Artifact Agent" +description: "Exercises the pinned candidate pipeline-artifact supply-chain source" +supply-chain: + pipeline-artifact: + project: AgentPlayground + definition-id: 2560 + run-id: 630001 + artifact: ado-aw-candidate +safe-outputs: + create-pull-request: + target-branch: main +--- + +## Pipeline Artifact Agent + +Use the compiler, AWF binary, and ado-script bundle from one pinned, +provenance-checked Azure DevOps pipeline artifact. diff --git a/tests/safe-outputs/README.md b/tests/safe-outputs/README.md index d683f22f..c7bea030 100644 --- a/tests/safe-outputs/README.md +++ b/tests/safe-outputs/README.md @@ -33,6 +33,25 @@ five pipelines: | `smoke-failure-reporter.md` / `smoke-failure-reporter.lock.yml` | Daily ~04:30: queries the canary and azure-cli pipelines for failures and files `[smoke-failure] …` issues on `jamesadevine/ado-aw-issues` while canonical-repo credentials are unavailable. | | `REGISTERED.md` | Contributor-maintained `fixture → ADO pipeline ID` mapping. | +## Release smoke and candidate smoke + +These five registered definitions remain the **release-backed** contract: their +checked-in YAML downloads the latest released compiler/runtime assets. A +separate [`tests/compiler-smoke-e2e/`](../compiler-smoke-e2e/) orchestrator +builds the exact same five workflows with a compiler from the current PR or +nightly `main`, stages the regenerated sources/YAML on a short-lived +`ado-aw-mirror` ref, and runs five fixed candidate definitions. Keeping both +lanes distinguishes release packaging/download failures from regressions in +unreleased compiler output. + +Do **not** regenerate the checked-in `tests/safe-outputs/*.lock.yml` files with +an unreleased development build. Their integrity step downloads the released +compiler, so a lock generated by newer `main` code will fail before the agent +runs even when both files display the same Cargo semver. Compiler PRs are +validated through the ephemeral candidate lane; the release workflow's +`recompile-safe-output-fixtures` automation updates the checked-in locks only +after matching release assets exist. + > **Deterministic complement.** For a flake-free regression check of > the Stage 3 executor with no LLM in the loop, see > [`tests/executor-e2e/`](../executor-e2e/). That suite covers all @@ -80,13 +99,19 @@ When you add `src/safe_outputs/.rs`: ## Running locally ```bash -# Recompile every fixture in this directory (idempotent): -cargo run -- compile tests/safe-outputs/ +# Verify a checked-in release fixture with the matching released binary: +ado-aw check tests/safe-outputs/canary.lock.yml -# Verify a single fixture compiled correctly: -cargo run -- check tests/safe-outputs/canary.lock.yml +# Exercise an unreleased checkout without changing release locks: +cd scripts/ado-script +npm run build:compiler-smoke-e2e ``` +Confirm `ado-aw --version` matches the version in the lock-file header before +using `ado-aw check`. Never use `cargo run -- compile` to overwrite these +release-owned locks; the candidate harness performs development recompilation +inside a temporary worktree. + ## Manual handoff (one-time ADO setup) In `https://dev.azure.com/msazuresphere/AgentPlayground`: @@ -104,6 +129,10 @@ In `https://dev.azure.com/msazuresphere/AgentPlayground`: tests/safe-outputs/ ``` + Keep CI and PR triggers disabled on every resulting release-smoke + definition. These definitions are scheduled/manual only; compiler PRs are + validated by `tests/compiler-smoke-e2e/`. + 3. Capture each Pipeline ID and update `REGISTERED.md`. 4. Provision the `ADO_AW_DEBUG_GITHUB_TOKEN` secret (fine-grained PAT, Issues: read/write on `jamesadevine/ado-aw-issues`) on the diff --git a/tests/safe-outputs/REGISTERED.md b/tests/safe-outputs/REGISTERED.md index 6dbb8d65..9fad2c47 100644 --- a/tests/safe-outputs/REGISTERED.md +++ b/tests/safe-outputs/REGISTERED.md @@ -28,6 +28,14 @@ mirror and every AgentPlayground test repository carry root `es-metadata.yml` inventory metadata so repository inventory automation keeps them enabled. +Definitions `2545`-`2549` must have no definition-level CI or PR trigger +overrides. The checked-in release locks emit `trigger: none` / `pr: none`; +their schedules and manual queues are the only intended activation paths. + +The PR/nightly compiler-candidate definitions are tracked separately in +[`tests/compiler-smoke-e2e/REGISTERED.md`](../compiler-smoke-e2e/REGISTERED.md); +they do not replace or repoint the release-backed definitions above. + ## Retired definitions The cutover deleted legacy definitions `2506`, `2513` through `2538`, and diff --git a/tests/safe-outputs/azure-cli.lock.yml b/tests/safe-outputs/azure-cli.lock.yml index 391390bd..9bc71a5d 100644 --- a/tests/safe-outputs/azure-cli.lock.yml +++ b/tests/safe-outputs/azure-cli.lock.yml @@ -118,60 +118,24 @@ jobs: # These duplicate the compile-time values baked into the YAML, but MCPG's # Docker container requires MCP_GATEWAY_PORT and MCP_GATEWAY_DOMAIN env vars # to start — the ADO variable indirection satisfies that contract. - echo "##vso[task.setvariable variable=MCP_GATEWAY_PORT]8080" - echo "##vso[task.setvariable variable=MCP_GATEWAY_DOMAIN]awmg-mcpg" + echo "##vso[task.setvariable variable=MCP_GATEWAY_PORT]80" + echo "##vso[task.setvariable variable=MCP_GATEWAY_DOMAIN]host.docker.internal" # Write MCPG (MCP Gateway) configuration to a file cat > "$(Agent.TempDirectory)/staging/mcpg-config.json" << 'MCPG_CONFIG_EOF' { "mcpServers": { "safeoutputs": { - "type": "stdio", - "container": "ghcr.io/github/gh-aw-firewall/agent:0.27.32", - "entrypoint": "/usr/local/bin/ado-aw", - "entrypointArgs": [ - "mcp", - "--enabled-tools", - "missing-data", - "--enabled-tools", - "missing-tool", - "--enabled-tools", - "noop", - "--enabled-tools", - "report-incomplete", - "/safeoutputs", - "$(Build.SourcesDirectory)" - ], - "mounts": [ - "/tmp/awf-tools/ado-aw:/usr/local/bin/ado-aw:ro", - "$(Build.SourcesDirectory):$(Build.SourcesDirectory):rw", - "/tmp/awf-tools/staging:/safeoutputs:rw" - ], - "args": [ - "--network", - "none", - "--user", - "${MCP_RUNNER_UID}:${MCP_RUNNER_GID}", - "--cap-drop", - "ALL", - "--security-opt", - "no-new-privileges", - "--read-only", - "--tmpfs", - "/tmp:rw,nosuid,nodev,noexec", - "--pids-limit", - "256", - "-w", - "$(Build.SourcesDirectory)" - ], - "env": { - "HOME": "/tmp" + "type": "http", + "url": "http://localhost:${SAFE_OUTPUTS_PORT}/mcp", + "headers": { + "Authorization": "Bearer ${SAFE_OUTPUTS_API_KEY}" } } }, "gateway": { - "port": 8080, - "domain": "awmg-mcpg", + "port": 80, + "domain": "host.docker.internal", "apiKey": "${MCP_GATEWAY_API_KEY}", "payloadDir": "/tmp/gh-aw/mcp-payloads" } @@ -221,7 +185,7 @@ jobs: - bash: | set -eo pipefail - AWF_VERSION="0.27.32" + AWF_VERSION="0.27.9" DOWNLOAD_DIR="$(Pipeline.Workspace)/awf" DOWNLOAD_URL="https://github.com/github/gh-aw-firewall/releases/download/v${AWF_VERSION}/awf-linux-x64" CHECKSUM_URL="https://github.com/github/gh-aw-firewall/releases/download/v${AWF_VERSION}/checksums.txt" @@ -238,15 +202,16 @@ jobs: chmod +x awf echo "##vso[task.prependpath]$(Pipeline.Workspace)/awf" ./awf --version - displayName: Download AWF (Agentic Workflow Firewall) v0.27.32 + displayName: Download AWF (Agentic Workflow Firewall) v0.27.9 - bash: | set -eo pipefail - docker pull ghcr.io/github/gh-aw-firewall/squid:0.27.32 - docker pull ghcr.io/github/gh-aw-firewall/agent:0.27.32 - docker pull ghcr.io/github/gh-aw-firewall/api-proxy:0.27.32 + docker pull ghcr.io/github/gh-aw-firewall/squid:0.27.9 + docker pull ghcr.io/github/gh-aw-firewall/agent:0.27.9 + docker tag ghcr.io/github/gh-aw-firewall/squid:0.27.9 ghcr.io/github/gh-aw-firewall/squid:latest + docker tag ghcr.io/github/gh-aw-firewall/agent:0.27.9 ghcr.io/github/gh-aw-firewall/agent:latest docker pull ghcr.io/github/gh-aw-mcpg:v0.4.1 - displayName: Pre-pull AWF and MCPG container images (v0.27.32) + displayName: Pre-pull AWF and MCPG container images (v0.27.9) - task: UseNode@1 inputs: version: 22.x @@ -323,13 +288,50 @@ jobs: echo "Azure CLI prompt appended" displayName: Append Azure CLI prompt condition: ne(variables['AW_AZ_MOUNTS'], '') + - bash: | + SAFE_OUTPUTS_PORT=8100 + SAFE_OUTPUTS_API_KEY=$(openssl rand -base64 45 | tr -d '/+=') + echo "##vso[task.setvariable variable=SAFE_OUTPUTS_PORT]$SAFE_OUTPUTS_PORT" + echo "##vso[task.setvariable variable=SAFE_OUTPUTS_API_KEY;issecret=true]$SAFE_OUTPUTS_API_KEY" + + mkdir -p "$(Agent.TempDirectory)/staging/logs" + + # Start SafeOutputs as HTTP server in the background + # NOTE: --enabled-tools missing-data --enabled-tools missing-tool --enabled-tools noop --enabled-tools report-incomplete expands to either "" or "--enabled-tools X ... " + # (with trailing space). The value MUST be newline-free; is_safe_tool_name enforces this. + # Positional args (output_directory, bounding_directory) MUST come after all named + # options — clap parses them positionally and reordering would break the command. + nohup /tmp/awf-tools/ado-aw mcp-http \ + --port "$SAFE_OUTPUTS_PORT" \ + --api-key "$SAFE_OUTPUTS_API_KEY" \ + --enabled-tools missing-data --enabled-tools missing-tool --enabled-tools noop --enabled-tools report-incomplete "/tmp/awf-tools/staging" \ + "$(Build.SourcesDirectory)" \ + > "$(Agent.TempDirectory)/staging/logs/safeoutputs.log" 2>&1 & + SAFE_OUTPUTS_PID=$! + echo "##vso[task.setvariable variable=SAFE_OUTPUTS_PID]$SAFE_OUTPUTS_PID" + echo "SafeOutputs HTTP server started on port $SAFE_OUTPUTS_PORT (PID: $SAFE_OUTPUTS_PID)" + + # Wait for server to be ready + READY=false + # shellcheck disable=SC2034 # i is intentionally unused; wait-N-times loop + for i in $(seq 1 30); do + if curl -sf "http://localhost:$SAFE_OUTPUTS_PORT/health" > /dev/null 2>&1; then + echo "SafeOutputs HTTP server is ready" + READY=true + break + fi + sleep 1 + done + if [ "$READY" != "true" ]; then + echo "##vso[task.complete result=Failed]SafeOutputs HTTP server did not become ready within 30s" + exit 1 + fi + displayName: Start SafeOutputs HTTP server - bash: | # Substitute runtime values into MCPG config - MCP_RUNNER_UID=$(id -u) - MCP_RUNNER_GID=$(id -g) MCPG_CONFIG=$(sed \ - -e "s|\${MCP_RUNNER_UID}|$MCP_RUNNER_UID|g" \ - -e "s|\${MCP_RUNNER_GID}|$MCP_RUNNER_GID|g" \ + -e "s|\${SAFE_OUTPUTS_PORT}|$(SAFE_OUTPUTS_PORT)|g" \ + -e "s|\${SAFE_OUTPUTS_API_KEY}|$(SAFE_OUTPUTS_API_KEY)|g" \ -e "s|\${MCP_GATEWAY_API_KEY}|$(MCP_GATEWAY_API_KEY)|g" \ /tmp/awf-tools/staging/mcpg-config.json) @@ -339,22 +341,26 @@ jobs: # Remove any leftover container or stale output from a previous interrupted run # (--rm only cleans up on clean exit; OOM/SIGKILL may leave it behind) - docker rm -f awmg-mcpg 2>/dev/null || true + docker rm -f mcpg 2>/dev/null || true GATEWAY_OUTPUT="/tmp/gh-aw/mcp-config/gateway-output.json" mkdir -p "$(dirname "$GATEWAY_OUTPUT")" /tmp/gh-aw/mcp-logs rm -f "$GATEWAY_OUTPUT" - # Start MCPG on Docker's bridge network. AWF attaches this named, - # trusted container to its internal network after creating awf-net. + # Start MCPG Docker container on host network. # The Docker socket mount is required because MCPG spawns stdio-based MCP # servers as sibling containers. This grants significant host access — acceptable # here because the pipeline agent is already trusted and network-isolated by AWF. # + # WORKAROUND: Override entrypoint to bypass run_containerized.sh which has a + # validate_port_mapping() bug — it calls `docker inspect .NetworkSettings.Ports` + # which is empty with --network host (by design), causing a spurious error: + # [ERROR] Port 80 is not exposed from the container + # Upstream fix: https://github.com/github/gh-aw-mcpg/issues/TBD + # # stdout → gateway-output.json (machine-readable config, read after health check) echo "$MCPG_CONFIG" | docker run -i --rm \ - --name awmg-mcpg \ - --network bridge \ - -p 127.0.0.1:8080:8080 \ + --name mcpg \ + --network host \ --entrypoint /app/awmg \ -v /var/run/docker.sock:/var/run/docker.sock \ -e MCP_GATEWAY_PORT="$(MCP_GATEWAY_PORT)" \ @@ -363,7 +369,7 @@ jobs: \ \ ghcr.io/github/gh-aw-mcpg:v0.4.1 \ - --routed --listen 0.0.0.0:8080 --config-stdin --log-dir /tmp/gh-aw/mcp-logs \ + --routed --listen 0.0.0.0:80 --config-stdin --log-dir /tmp/gh-aw/mcp-logs \ > "$GATEWAY_OUTPUT" 2> >(tee /tmp/gh-aw/mcp-logs/stderr.log >&2) & MCPG_PID=$! echo "MCPG started (PID: $MCPG_PID)" @@ -372,7 +378,7 @@ jobs: READY=false # shellcheck disable=SC2034 # i is intentionally unused; wait-N-times loop for i in $(seq 1 30); do - if curl -sf "http://localhost:8080/health" > /dev/null 2>&1; then + if curl -sf "http://localhost:80/health" > /dev/null 2>&1; then echo "MCPG is ready" READY=true break @@ -409,8 +415,8 @@ jobs: # Convert gateway output to Copilot CLI mcp-config.json. # Mirrors gh-aw's convert_gateway_config_copilot.cjs: - # - Rewrite gateway URLs to the stable MCPG container name that AWF - # attaches to its internal network + # - Rewrite URLs from 127.0.0.1 to host.docker.internal (AWF container needs + # host.docker.internal to reach MCPG on the host; 127.0.0.1 is container loopback) # - Ensure tools: ["*"] on each server entry (Copilot CLI requirement) # - Mark generated MCPG entries as default/trusted servers for Copilot CLI # - Preserve all other fields (headers, type, etc.) @@ -430,30 +436,28 @@ jobs: mkdir -p "$(Agent.TempDirectory)/staging/logs" echo "=== Running AI agent with AWF network isolation ===" - echo "Allowed domains: *.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" + echo "Allowed domains: *.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,host.docker.internal,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" - # AWF provides L7 domain whitelisting via a rootless Docker topology. - # The named MCPG container is attached to AWF's internal network as a - # trusted endpoint; the agent has no route to the host. + # AWF provides L7 domain whitelisting via Squid proxy + Docker containers. + # --enable-host-access allows the AWF container to reach host services + # (MCPG and SafeOutputs) via host.docker.internal. # AWF auto-mounts /tmp:/tmp:rw into the container, so copilot binary, # agent prompt, and MCP config are placed under /tmp/awf-tools/. # Stream agent output in real-time while filtering VSO commands. # sed -u = unbuffered (line-by-line) so output appears immediately. # tee writes to both stdout (ADO pipeline log) and the artifact file. # pipefail (set above) ensures AWF's exit code propagates through the pipe. - # shellcheck disable=SC2046,SC2016 # ADO macros are substituted before bash; the single-quoted engine command is intentionally expanded by AWF inside the sandbox - "$(Pipeline.Workspace)/awf/awf" \ - --allow-domains "*.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" \ - --network-isolation \ - --topology-attach "awmg-mcpg" \ - --image-tag "0.27.32" \ + # shellcheck disable=SC2046 # $(AW_AZ_MOUNTS) is an ADO macro substituted before bash sees it, not bash command substitution; word-splitting the expanded value into separate --mount tokens is intentional + sudo -E "$(Pipeline.Workspace)/awf/awf" \ + --allow-domains "*.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,host.docker.internal,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" \ --skip-pull \ --env-all \ - $(AW_AZ_MOUNTS) \ + --enable-host-access \ + $(AW_AZ_MOUNTS) \ --container-workdir "$(Build.SourcesDirectory)" \ --log-level info \ --proxy-logs-dir "$(Agent.TempDirectory)/staging/logs/firewall" \ - -- 'export NO_PROXY="${NO_PROXY:+$NO_PROXY,}awmg-mcpg"; export no_proxy="$NO_PROXY"; /tmp/awf-tools/copilot --prompt "$(cat /tmp/awf-tools/agent-prompt.md)" --additional-mcp-config @/tmp/awf-tools/mcp-config.json --model gpt-5-mini --disable-builtin-mcps --no-ask-user --allow-all-tools --allow-tool github --allow-tool safeoutputs --allow-all-paths' \ + -- '/tmp/awf-tools/copilot --prompt "$(cat /tmp/awf-tools/agent-prompt.md)" --additional-mcp-config @/tmp/awf-tools/mcp-config.json --model gpt-5-mini --disable-builtin-mcps --no-ask-user --allow-all-tools --allow-tool github --allow-tool safeoutputs --allow-all-paths' \ 2>&1 \ | sed -u 's/##vso\[/[VSO-FILTERED] vso[/g; s/##\[/[VSO-FILTERED] [/g' \ | tee "$AGENT_OUTPUT_FILE" \ @@ -493,9 +497,16 @@ jobs: - bash: | # Stop MCPG container echo "Stopping MCPG..." - docker stop awmg-mcpg 2>/dev/null || true - echo "MCPG and stdio child containers stopped" - displayName: Stop MCPG + docker stop mcpg 2>/dev/null || true + echo "MCPG stopped" + + # Stop SafeOutputs HTTP server + if [ -n "$(SAFE_OUTPUTS_PID)" ]; then + echo "Stopping SafeOutputs (PID: $(SAFE_OUTPUTS_PID))..." + kill "$(SAFE_OUTPUTS_PID)" 2>/dev/null || true + echo "SafeOutputs stopped" + fi + displayName: Stop MCPG and SafeOutputs condition: always() - bash: | # Copy all logs to output directory for artifact upload @@ -598,7 +609,7 @@ jobs: - bash: | set -eo pipefail - AWF_VERSION="0.27.32" + AWF_VERSION="0.27.9" DOWNLOAD_DIR="$(Pipeline.Workspace)/awf" DOWNLOAD_URL="https://github.com/github/gh-aw-firewall/releases/download/v${AWF_VERSION}/awf-linux-x64" CHECKSUM_URL="https://github.com/github/gh-aw-firewall/releases/download/v${AWF_VERSION}/checksums.txt" @@ -615,14 +626,15 @@ jobs: chmod +x awf echo "##vso[task.prependpath]$(Pipeline.Workspace)/awf" ./awf --version - displayName: Download AWF (Agentic Workflow Firewall) v0.27.32 + displayName: Download AWF (Agentic Workflow Firewall) v0.27.9 - bash: | set -eo pipefail - docker pull ghcr.io/github/gh-aw-firewall/squid:0.27.32 - docker pull ghcr.io/github/gh-aw-firewall/agent:0.27.32 - docker pull ghcr.io/github/gh-aw-firewall/api-proxy:0.27.32 - displayName: Pre-pull AWF container images (v0.27.32) + docker pull ghcr.io/github/gh-aw-firewall/squid:0.27.9 + docker pull ghcr.io/github/gh-aw-firewall/agent:0.27.9 + docker tag ghcr.io/github/gh-aw-firewall/squid:0.27.9 ghcr.io/github/gh-aw-firewall/squid:latest + docker tag ghcr.io/github/gh-aw-firewall/agent:0.27.9 ghcr.io/github/gh-aw-firewall/agent:latest + displayName: Pre-pull AWF container images (v0.27.9) - bash: | mkdir -p "$(Build.SourcesDirectory)/safe_outputs" cp -a "$(Pipeline.Workspace)/agent_outputs_$(Build.BuildId)/." "$(Build.SourcesDirectory)/safe_outputs" @@ -683,11 +695,8 @@ jobs: THREAT_OUTPUT_FILE="$(Agent.TempDirectory)/threat-analysis-output.txt" # Stream threat analysis output in real-time with VSO command filtering - # shellcheck disable=SC2016 # The single-quoted engine command is intentionally expanded by AWF inside the sandbox - "$(Pipeline.Workspace)/awf/awf" \ - --allow-domains "*.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" \ - --network-isolation \ - --image-tag "0.27.32" \ + sudo -E "$(Pipeline.Workspace)/awf/awf" \ + --allow-domains "*.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,host.docker.internal,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" \ --skip-pull \ --env-all \ --container-workdir "$(Build.SourcesDirectory)" \ diff --git a/tests/safe-outputs/canary.lock.yml b/tests/safe-outputs/canary.lock.yml index 42d50197..9e755379 100644 --- a/tests/safe-outputs/canary.lock.yml +++ b/tests/safe-outputs/canary.lock.yml @@ -118,64 +118,24 @@ jobs: # These duplicate the compile-time values baked into the YAML, but MCPG's # Docker container requires MCP_GATEWAY_PORT and MCP_GATEWAY_DOMAIN env vars # to start — the ADO variable indirection satisfies that contract. - echo "##vso[task.setvariable variable=MCP_GATEWAY_PORT]8080" - echo "##vso[task.setvariable variable=MCP_GATEWAY_DOMAIN]awmg-mcpg" + echo "##vso[task.setvariable variable=MCP_GATEWAY_PORT]80" + echo "##vso[task.setvariable variable=MCP_GATEWAY_DOMAIN]host.docker.internal" # Write MCPG (MCP Gateway) configuration to a file cat > "$(Agent.TempDirectory)/staging/mcpg-config.json" << 'MCPG_CONFIG_EOF' { "mcpServers": { "safeoutputs": { - "type": "stdio", - "container": "ghcr.io/github/gh-aw-firewall/agent:0.27.32", - "entrypoint": "/usr/local/bin/ado-aw", - "entrypointArgs": [ - "mcp", - "--enabled-tools", - "add-build-tag", - "--enabled-tools", - "create-work-item", - "--enabled-tools", - "missing-data", - "--enabled-tools", - "missing-tool", - "--enabled-tools", - "noop", - "--enabled-tools", - "report-incomplete", - "/safeoutputs", - "$(Build.SourcesDirectory)" - ], - "mounts": [ - "/tmp/awf-tools/ado-aw:/usr/local/bin/ado-aw:ro", - "$(Build.SourcesDirectory):$(Build.SourcesDirectory):rw", - "/tmp/awf-tools/staging:/safeoutputs:rw" - ], - "args": [ - "--network", - "none", - "--user", - "${MCP_RUNNER_UID}:${MCP_RUNNER_GID}", - "--cap-drop", - "ALL", - "--security-opt", - "no-new-privileges", - "--read-only", - "--tmpfs", - "/tmp:rw,nosuid,nodev,noexec", - "--pids-limit", - "256", - "-w", - "$(Build.SourcesDirectory)" - ], - "env": { - "HOME": "/tmp" + "type": "http", + "url": "http://localhost:${SAFE_OUTPUTS_PORT}/mcp", + "headers": { + "Authorization": "Bearer ${SAFE_OUTPUTS_API_KEY}" } } }, "gateway": { - "port": 8080, - "domain": "awmg-mcpg", + "port": 80, + "domain": "host.docker.internal", "apiKey": "${MCP_GATEWAY_API_KEY}", "payloadDir": "/tmp/gh-aw/mcp-payloads" } @@ -225,7 +185,7 @@ jobs: - bash: | set -eo pipefail - AWF_VERSION="0.27.32" + AWF_VERSION="0.27.9" DOWNLOAD_DIR="$(Pipeline.Workspace)/awf" DOWNLOAD_URL="https://github.com/github/gh-aw-firewall/releases/download/v${AWF_VERSION}/awf-linux-x64" CHECKSUM_URL="https://github.com/github/gh-aw-firewall/releases/download/v${AWF_VERSION}/checksums.txt" @@ -242,15 +202,16 @@ jobs: chmod +x awf echo "##vso[task.prependpath]$(Pipeline.Workspace)/awf" ./awf --version - displayName: Download AWF (Agentic Workflow Firewall) v0.27.32 + displayName: Download AWF (Agentic Workflow Firewall) v0.27.9 - bash: | set -eo pipefail - docker pull ghcr.io/github/gh-aw-firewall/squid:0.27.32 - docker pull ghcr.io/github/gh-aw-firewall/agent:0.27.32 - docker pull ghcr.io/github/gh-aw-firewall/api-proxy:0.27.32 + docker pull ghcr.io/github/gh-aw-firewall/squid:0.27.9 + docker pull ghcr.io/github/gh-aw-firewall/agent:0.27.9 + docker tag ghcr.io/github/gh-aw-firewall/squid:0.27.9 ghcr.io/github/gh-aw-firewall/squid:latest + docker tag ghcr.io/github/gh-aw-firewall/agent:0.27.9 ghcr.io/github/gh-aw-firewall/agent:latest docker pull ghcr.io/github/gh-aw-mcpg:v0.4.1 - displayName: Pre-pull AWF and MCPG container images (v0.27.32) + displayName: Pre-pull AWF and MCPG container images (v0.27.9) - task: UseNode@1 inputs: version: 22.x @@ -327,13 +288,50 @@ jobs: echo "Azure CLI prompt appended" displayName: Append Azure CLI prompt condition: ne(variables['AW_AZ_MOUNTS'], '') + - bash: | + SAFE_OUTPUTS_PORT=8100 + SAFE_OUTPUTS_API_KEY=$(openssl rand -base64 45 | tr -d '/+=') + echo "##vso[task.setvariable variable=SAFE_OUTPUTS_PORT]$SAFE_OUTPUTS_PORT" + echo "##vso[task.setvariable variable=SAFE_OUTPUTS_API_KEY;issecret=true]$SAFE_OUTPUTS_API_KEY" + + mkdir -p "$(Agent.TempDirectory)/staging/logs" + + # Start SafeOutputs as HTTP server in the background + # NOTE: --enabled-tools add-build-tag --enabled-tools create-work-item --enabled-tools missing-data --enabled-tools missing-tool --enabled-tools noop --enabled-tools report-incomplete expands to either "" or "--enabled-tools X ... " + # (with trailing space). The value MUST be newline-free; is_safe_tool_name enforces this. + # Positional args (output_directory, bounding_directory) MUST come after all named + # options — clap parses them positionally and reordering would break the command. + nohup /tmp/awf-tools/ado-aw mcp-http \ + --port "$SAFE_OUTPUTS_PORT" \ + --api-key "$SAFE_OUTPUTS_API_KEY" \ + --enabled-tools add-build-tag --enabled-tools create-work-item --enabled-tools missing-data --enabled-tools missing-tool --enabled-tools noop --enabled-tools report-incomplete "/tmp/awf-tools/staging" \ + "$(Build.SourcesDirectory)" \ + > "$(Agent.TempDirectory)/staging/logs/safeoutputs.log" 2>&1 & + SAFE_OUTPUTS_PID=$! + echo "##vso[task.setvariable variable=SAFE_OUTPUTS_PID]$SAFE_OUTPUTS_PID" + echo "SafeOutputs HTTP server started on port $SAFE_OUTPUTS_PORT (PID: $SAFE_OUTPUTS_PID)" + + # Wait for server to be ready + READY=false + # shellcheck disable=SC2034 # i is intentionally unused; wait-N-times loop + for i in $(seq 1 30); do + if curl -sf "http://localhost:$SAFE_OUTPUTS_PORT/health" > /dev/null 2>&1; then + echo "SafeOutputs HTTP server is ready" + READY=true + break + fi + sleep 1 + done + if [ "$READY" != "true" ]; then + echo "##vso[task.complete result=Failed]SafeOutputs HTTP server did not become ready within 30s" + exit 1 + fi + displayName: Start SafeOutputs HTTP server - bash: | # Substitute runtime values into MCPG config - MCP_RUNNER_UID=$(id -u) - MCP_RUNNER_GID=$(id -g) MCPG_CONFIG=$(sed \ - -e "s|\${MCP_RUNNER_UID}|$MCP_RUNNER_UID|g" \ - -e "s|\${MCP_RUNNER_GID}|$MCP_RUNNER_GID|g" \ + -e "s|\${SAFE_OUTPUTS_PORT}|$(SAFE_OUTPUTS_PORT)|g" \ + -e "s|\${SAFE_OUTPUTS_API_KEY}|$(SAFE_OUTPUTS_API_KEY)|g" \ -e "s|\${MCP_GATEWAY_API_KEY}|$(MCP_GATEWAY_API_KEY)|g" \ /tmp/awf-tools/staging/mcpg-config.json) @@ -343,22 +341,26 @@ jobs: # Remove any leftover container or stale output from a previous interrupted run # (--rm only cleans up on clean exit; OOM/SIGKILL may leave it behind) - docker rm -f awmg-mcpg 2>/dev/null || true + docker rm -f mcpg 2>/dev/null || true GATEWAY_OUTPUT="/tmp/gh-aw/mcp-config/gateway-output.json" mkdir -p "$(dirname "$GATEWAY_OUTPUT")" /tmp/gh-aw/mcp-logs rm -f "$GATEWAY_OUTPUT" - # Start MCPG on Docker's bridge network. AWF attaches this named, - # trusted container to its internal network after creating awf-net. + # Start MCPG Docker container on host network. # The Docker socket mount is required because MCPG spawns stdio-based MCP # servers as sibling containers. This grants significant host access — acceptable # here because the pipeline agent is already trusted and network-isolated by AWF. # + # WORKAROUND: Override entrypoint to bypass run_containerized.sh which has a + # validate_port_mapping() bug — it calls `docker inspect .NetworkSettings.Ports` + # which is empty with --network host (by design), causing a spurious error: + # [ERROR] Port 80 is not exposed from the container + # Upstream fix: https://github.com/github/gh-aw-mcpg/issues/TBD + # # stdout → gateway-output.json (machine-readable config, read after health check) echo "$MCPG_CONFIG" | docker run -i --rm \ - --name awmg-mcpg \ - --network bridge \ - -p 127.0.0.1:8080:8080 \ + --name mcpg \ + --network host \ --entrypoint /app/awmg \ -v /var/run/docker.sock:/var/run/docker.sock \ -e MCP_GATEWAY_PORT="$(MCP_GATEWAY_PORT)" \ @@ -367,7 +369,7 @@ jobs: \ \ ghcr.io/github/gh-aw-mcpg:v0.4.1 \ - --routed --listen 0.0.0.0:8080 --config-stdin --log-dir /tmp/gh-aw/mcp-logs \ + --routed --listen 0.0.0.0:80 --config-stdin --log-dir /tmp/gh-aw/mcp-logs \ > "$GATEWAY_OUTPUT" 2> >(tee /tmp/gh-aw/mcp-logs/stderr.log >&2) & MCPG_PID=$! echo "MCPG started (PID: $MCPG_PID)" @@ -376,7 +378,7 @@ jobs: READY=false # shellcheck disable=SC2034 # i is intentionally unused; wait-N-times loop for i in $(seq 1 30); do - if curl -sf "http://localhost:8080/health" > /dev/null 2>&1; then + if curl -sf "http://localhost:80/health" > /dev/null 2>&1; then echo "MCPG is ready" READY=true break @@ -413,8 +415,8 @@ jobs: # Convert gateway output to Copilot CLI mcp-config.json. # Mirrors gh-aw's convert_gateway_config_copilot.cjs: - # - Rewrite gateway URLs to the stable MCPG container name that AWF - # attaches to its internal network + # - Rewrite URLs from 127.0.0.1 to host.docker.internal (AWF container needs + # host.docker.internal to reach MCPG on the host; 127.0.0.1 is container loopback) # - Ensure tools: ["*"] on each server entry (Copilot CLI requirement) # - Mark generated MCPG entries as default/trusted servers for Copilot CLI # - Preserve all other fields (headers, type, etc.) @@ -434,30 +436,28 @@ jobs: mkdir -p "$(Agent.TempDirectory)/staging/logs" echo "=== Running AI agent with AWF network isolation ===" - echo "Allowed domains: *.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" + echo "Allowed domains: *.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,host.docker.internal,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" - # AWF provides L7 domain whitelisting via a rootless Docker topology. - # The named MCPG container is attached to AWF's internal network as a - # trusted endpoint; the agent has no route to the host. + # AWF provides L7 domain whitelisting via Squid proxy + Docker containers. + # --enable-host-access allows the AWF container to reach host services + # (MCPG and SafeOutputs) via host.docker.internal. # AWF auto-mounts /tmp:/tmp:rw into the container, so copilot binary, # agent prompt, and MCP config are placed under /tmp/awf-tools/. # Stream agent output in real-time while filtering VSO commands. # sed -u = unbuffered (line-by-line) so output appears immediately. # tee writes to both stdout (ADO pipeline log) and the artifact file. # pipefail (set above) ensures AWF's exit code propagates through the pipe. - # shellcheck disable=SC2046,SC2016 # ADO macros are substituted before bash; the single-quoted engine command is intentionally expanded by AWF inside the sandbox - "$(Pipeline.Workspace)/awf/awf" \ - --allow-domains "*.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" \ - --network-isolation \ - --topology-attach "awmg-mcpg" \ - --image-tag "0.27.32" \ + # shellcheck disable=SC2046 # $(AW_AZ_MOUNTS) is an ADO macro substituted before bash sees it, not bash command substitution; word-splitting the expanded value into separate --mount tokens is intentional + sudo -E "$(Pipeline.Workspace)/awf/awf" \ + --allow-domains "*.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,host.docker.internal,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" \ --skip-pull \ --env-all \ - $(AW_AZ_MOUNTS) \ + --enable-host-access \ + $(AW_AZ_MOUNTS) \ --container-workdir "$(Build.SourcesDirectory)" \ --log-level info \ --proxy-logs-dir "$(Agent.TempDirectory)/staging/logs/firewall" \ - -- 'export NO_PROXY="${NO_PROXY:+$NO_PROXY,}awmg-mcpg"; export no_proxy="$NO_PROXY"; /tmp/awf-tools/copilot --prompt "$(cat /tmp/awf-tools/agent-prompt.md)" --additional-mcp-config @/tmp/awf-tools/mcp-config.json --model gpt-5-mini --disable-builtin-mcps --no-ask-user --allow-all-tools --allow-tool github --allow-tool safeoutputs --allow-all-paths' \ + -- '/tmp/awf-tools/copilot --prompt "$(cat /tmp/awf-tools/agent-prompt.md)" --additional-mcp-config @/tmp/awf-tools/mcp-config.json --model gpt-5-mini --disable-builtin-mcps --no-ask-user --allow-all-tools --allow-tool github --allow-tool safeoutputs --allow-all-paths' \ 2>&1 \ | sed -u 's/##vso\[/[VSO-FILTERED] vso[/g; s/##\[/[VSO-FILTERED] [/g' \ | tee "$AGENT_OUTPUT_FILE" \ @@ -497,9 +497,16 @@ jobs: - bash: | # Stop MCPG container echo "Stopping MCPG..." - docker stop awmg-mcpg 2>/dev/null || true - echo "MCPG and stdio child containers stopped" - displayName: Stop MCPG + docker stop mcpg 2>/dev/null || true + echo "MCPG stopped" + + # Stop SafeOutputs HTTP server + if [ -n "$(SAFE_OUTPUTS_PID)" ]; then + echo "Stopping SafeOutputs (PID: $(SAFE_OUTPUTS_PID))..." + kill "$(SAFE_OUTPUTS_PID)" 2>/dev/null || true + echo "SafeOutputs stopped" + fi + displayName: Stop MCPG and SafeOutputs condition: always() - bash: | # Copy all logs to output directory for artifact upload @@ -602,7 +609,7 @@ jobs: - bash: | set -eo pipefail - AWF_VERSION="0.27.32" + AWF_VERSION="0.27.9" DOWNLOAD_DIR="$(Pipeline.Workspace)/awf" DOWNLOAD_URL="https://github.com/github/gh-aw-firewall/releases/download/v${AWF_VERSION}/awf-linux-x64" CHECKSUM_URL="https://github.com/github/gh-aw-firewall/releases/download/v${AWF_VERSION}/checksums.txt" @@ -619,14 +626,15 @@ jobs: chmod +x awf echo "##vso[task.prependpath]$(Pipeline.Workspace)/awf" ./awf --version - displayName: Download AWF (Agentic Workflow Firewall) v0.27.32 + displayName: Download AWF (Agentic Workflow Firewall) v0.27.9 - bash: | set -eo pipefail - docker pull ghcr.io/github/gh-aw-firewall/squid:0.27.32 - docker pull ghcr.io/github/gh-aw-firewall/agent:0.27.32 - docker pull ghcr.io/github/gh-aw-firewall/api-proxy:0.27.32 - displayName: Pre-pull AWF container images (v0.27.32) + docker pull ghcr.io/github/gh-aw-firewall/squid:0.27.9 + docker pull ghcr.io/github/gh-aw-firewall/agent:0.27.9 + docker tag ghcr.io/github/gh-aw-firewall/squid:0.27.9 ghcr.io/github/gh-aw-firewall/squid:latest + docker tag ghcr.io/github/gh-aw-firewall/agent:0.27.9 ghcr.io/github/gh-aw-firewall/agent:latest + displayName: Pre-pull AWF container images (v0.27.9) - bash: | mkdir -p "$(Build.SourcesDirectory)/safe_outputs" cp -a "$(Pipeline.Workspace)/agent_outputs_$(Build.BuildId)/." "$(Build.SourcesDirectory)/safe_outputs" @@ -687,11 +695,8 @@ jobs: THREAT_OUTPUT_FILE="$(Agent.TempDirectory)/threat-analysis-output.txt" # Stream threat analysis output in real-time with VSO command filtering - # shellcheck disable=SC2016 # The single-quoted engine command is intentionally expanded by AWF inside the sandbox - "$(Pipeline.Workspace)/awf/awf" \ - --allow-domains "*.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" \ - --network-isolation \ - --image-tag "0.27.32" \ + sudo -E "$(Pipeline.Workspace)/awf/awf" \ + --allow-domains "*.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,host.docker.internal,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" \ --skip-pull \ --env-all \ --container-workdir "$(Build.SourcesDirectory)" \ diff --git a/tests/safe-outputs/janitor.lock.yml b/tests/safe-outputs/janitor.lock.yml index 06388d72..e4b69847 100644 --- a/tests/safe-outputs/janitor.lock.yml +++ b/tests/safe-outputs/janitor.lock.yml @@ -141,60 +141,24 @@ jobs: # These duplicate the compile-time values baked into the YAML, but MCPG's # Docker container requires MCP_GATEWAY_PORT and MCP_GATEWAY_DOMAIN env vars # to start — the ADO variable indirection satisfies that contract. - echo "##vso[task.setvariable variable=MCP_GATEWAY_PORT]8080" - echo "##vso[task.setvariable variable=MCP_GATEWAY_DOMAIN]awmg-mcpg" + echo "##vso[task.setvariable variable=MCP_GATEWAY_PORT]80" + echo "##vso[task.setvariable variable=MCP_GATEWAY_DOMAIN]host.docker.internal" # Write MCPG (MCP Gateway) configuration to a file cat > "$(Agent.TempDirectory)/staging/mcpg-config.json" << 'MCPG_CONFIG_EOF' { "mcpServers": { "safeoutputs": { - "type": "stdio", - "container": "ghcr.io/github/gh-aw-firewall/agent:0.27.32", - "entrypoint": "/usr/local/bin/ado-aw", - "entrypointArgs": [ - "mcp", - "--enabled-tools", - "missing-data", - "--enabled-tools", - "missing-tool", - "--enabled-tools", - "noop", - "--enabled-tools", - "report-incomplete", - "/safeoutputs", - "$(Build.SourcesDirectory)" - ], - "mounts": [ - "/tmp/awf-tools/ado-aw:/usr/local/bin/ado-aw:ro", - "$(Build.SourcesDirectory):$(Build.SourcesDirectory):rw", - "/tmp/awf-tools/staging:/safeoutputs:rw" - ], - "args": [ - "--network", - "none", - "--user", - "${MCP_RUNNER_UID}:${MCP_RUNNER_GID}", - "--cap-drop", - "ALL", - "--security-opt", - "no-new-privileges", - "--read-only", - "--tmpfs", - "/tmp:rw,nosuid,nodev,noexec", - "--pids-limit", - "256", - "-w", - "$(Build.SourcesDirectory)" - ], - "env": { - "HOME": "/tmp" + "type": "http", + "url": "http://localhost:${SAFE_OUTPUTS_PORT}/mcp", + "headers": { + "Authorization": "Bearer ${SAFE_OUTPUTS_API_KEY}" } } }, "gateway": { - "port": 8080, - "domain": "awmg-mcpg", + "port": 80, + "domain": "host.docker.internal", "apiKey": "${MCP_GATEWAY_API_KEY}", "payloadDir": "/tmp/gh-aw/mcp-payloads" } @@ -244,7 +208,7 @@ jobs: - bash: | set -eo pipefail - AWF_VERSION="0.27.32" + AWF_VERSION="0.27.9" DOWNLOAD_DIR="$(Pipeline.Workspace)/awf" DOWNLOAD_URL="https://github.com/github/gh-aw-firewall/releases/download/v${AWF_VERSION}/awf-linux-x64" CHECKSUM_URL="https://github.com/github/gh-aw-firewall/releases/download/v${AWF_VERSION}/checksums.txt" @@ -261,15 +225,16 @@ jobs: chmod +x awf echo "##vso[task.prependpath]$(Pipeline.Workspace)/awf" ./awf --version - displayName: Download AWF (Agentic Workflow Firewall) v0.27.32 + displayName: Download AWF (Agentic Workflow Firewall) v0.27.9 - bash: | set -eo pipefail - docker pull ghcr.io/github/gh-aw-firewall/squid:0.27.32 - docker pull ghcr.io/github/gh-aw-firewall/agent:0.27.32 - docker pull ghcr.io/github/gh-aw-firewall/api-proxy:0.27.32 + docker pull ghcr.io/github/gh-aw-firewall/squid:0.27.9 + docker pull ghcr.io/github/gh-aw-firewall/agent:0.27.9 + docker tag ghcr.io/github/gh-aw-firewall/squid:0.27.9 ghcr.io/github/gh-aw-firewall/squid:latest + docker tag ghcr.io/github/gh-aw-firewall/agent:0.27.9 ghcr.io/github/gh-aw-firewall/agent:latest docker pull ghcr.io/github/gh-aw-mcpg:v0.4.1 - displayName: Pre-pull AWF and MCPG container images (v0.27.32) + displayName: Pre-pull AWF and MCPG container images (v0.27.9) - task: UseNode@1 inputs: version: 22.x @@ -346,13 +311,50 @@ jobs: echo "Azure CLI prompt appended" displayName: Append Azure CLI prompt condition: ne(variables['AW_AZ_MOUNTS'], '') + - bash: | + SAFE_OUTPUTS_PORT=8100 + SAFE_OUTPUTS_API_KEY=$(openssl rand -base64 45 | tr -d '/+=') + echo "##vso[task.setvariable variable=SAFE_OUTPUTS_PORT]$SAFE_OUTPUTS_PORT" + echo "##vso[task.setvariable variable=SAFE_OUTPUTS_API_KEY;issecret=true]$SAFE_OUTPUTS_API_KEY" + + mkdir -p "$(Agent.TempDirectory)/staging/logs" + + # Start SafeOutputs as HTTP server in the background + # NOTE: --enabled-tools missing-data --enabled-tools missing-tool --enabled-tools noop --enabled-tools report-incomplete expands to either "" or "--enabled-tools X ... " + # (with trailing space). The value MUST be newline-free; is_safe_tool_name enforces this. + # Positional args (output_directory, bounding_directory) MUST come after all named + # options — clap parses them positionally and reordering would break the command. + nohup /tmp/awf-tools/ado-aw mcp-http \ + --port "$SAFE_OUTPUTS_PORT" \ + --api-key "$SAFE_OUTPUTS_API_KEY" \ + --enabled-tools missing-data --enabled-tools missing-tool --enabled-tools noop --enabled-tools report-incomplete "/tmp/awf-tools/staging" \ + "$(Build.SourcesDirectory)" \ + > "$(Agent.TempDirectory)/staging/logs/safeoutputs.log" 2>&1 & + SAFE_OUTPUTS_PID=$! + echo "##vso[task.setvariable variable=SAFE_OUTPUTS_PID]$SAFE_OUTPUTS_PID" + echo "SafeOutputs HTTP server started on port $SAFE_OUTPUTS_PORT (PID: $SAFE_OUTPUTS_PID)" + + # Wait for server to be ready + READY=false + # shellcheck disable=SC2034 # i is intentionally unused; wait-N-times loop + for i in $(seq 1 30); do + if curl -sf "http://localhost:$SAFE_OUTPUTS_PORT/health" > /dev/null 2>&1; then + echo "SafeOutputs HTTP server is ready" + READY=true + break + fi + sleep 1 + done + if [ "$READY" != "true" ]; then + echo "##vso[task.complete result=Failed]SafeOutputs HTTP server did not become ready within 30s" + exit 1 + fi + displayName: Start SafeOutputs HTTP server - bash: | # Substitute runtime values into MCPG config - MCP_RUNNER_UID=$(id -u) - MCP_RUNNER_GID=$(id -g) MCPG_CONFIG=$(sed \ - -e "s|\${MCP_RUNNER_UID}|$MCP_RUNNER_UID|g" \ - -e "s|\${MCP_RUNNER_GID}|$MCP_RUNNER_GID|g" \ + -e "s|\${SAFE_OUTPUTS_PORT}|$(SAFE_OUTPUTS_PORT)|g" \ + -e "s|\${SAFE_OUTPUTS_API_KEY}|$(SAFE_OUTPUTS_API_KEY)|g" \ -e "s|\${MCP_GATEWAY_API_KEY}|$(MCP_GATEWAY_API_KEY)|g" \ /tmp/awf-tools/staging/mcpg-config.json) @@ -362,22 +364,26 @@ jobs: # Remove any leftover container or stale output from a previous interrupted run # (--rm only cleans up on clean exit; OOM/SIGKILL may leave it behind) - docker rm -f awmg-mcpg 2>/dev/null || true + docker rm -f mcpg 2>/dev/null || true GATEWAY_OUTPUT="/tmp/gh-aw/mcp-config/gateway-output.json" mkdir -p "$(dirname "$GATEWAY_OUTPUT")" /tmp/gh-aw/mcp-logs rm -f "$GATEWAY_OUTPUT" - # Start MCPG on Docker's bridge network. AWF attaches this named, - # trusted container to its internal network after creating awf-net. + # Start MCPG Docker container on host network. # The Docker socket mount is required because MCPG spawns stdio-based MCP # servers as sibling containers. This grants significant host access — acceptable # here because the pipeline agent is already trusted and network-isolated by AWF. # + # WORKAROUND: Override entrypoint to bypass run_containerized.sh which has a + # validate_port_mapping() bug — it calls `docker inspect .NetworkSettings.Ports` + # which is empty with --network host (by design), causing a spurious error: + # [ERROR] Port 80 is not exposed from the container + # Upstream fix: https://github.com/github/gh-aw-mcpg/issues/TBD + # # stdout → gateway-output.json (machine-readable config, read after health check) echo "$MCPG_CONFIG" | docker run -i --rm \ - --name awmg-mcpg \ - --network bridge \ - -p 127.0.0.1:8080:8080 \ + --name mcpg \ + --network host \ --entrypoint /app/awmg \ -v /var/run/docker.sock:/var/run/docker.sock \ -e MCP_GATEWAY_PORT="$(MCP_GATEWAY_PORT)" \ @@ -386,7 +392,7 @@ jobs: \ \ ghcr.io/github/gh-aw-mcpg:v0.4.1 \ - --routed --listen 0.0.0.0:8080 --config-stdin --log-dir /tmp/gh-aw/mcp-logs \ + --routed --listen 0.0.0.0:80 --config-stdin --log-dir /tmp/gh-aw/mcp-logs \ > "$GATEWAY_OUTPUT" 2> >(tee /tmp/gh-aw/mcp-logs/stderr.log >&2) & MCPG_PID=$! echo "MCPG started (PID: $MCPG_PID)" @@ -395,7 +401,7 @@ jobs: READY=false # shellcheck disable=SC2034 # i is intentionally unused; wait-N-times loop for i in $(seq 1 30); do - if curl -sf "http://localhost:8080/health" > /dev/null 2>&1; then + if curl -sf "http://localhost:80/health" > /dev/null 2>&1; then echo "MCPG is ready" READY=true break @@ -432,8 +438,8 @@ jobs: # Convert gateway output to Copilot CLI mcp-config.json. # Mirrors gh-aw's convert_gateway_config_copilot.cjs: - # - Rewrite gateway URLs to the stable MCPG container name that AWF - # attaches to its internal network + # - Rewrite URLs from 127.0.0.1 to host.docker.internal (AWF container needs + # host.docker.internal to reach MCPG on the host; 127.0.0.1 is container loopback) # - Ensure tools: ["*"] on each server entry (Copilot CLI requirement) # - Mark generated MCPG entries as default/trusted servers for Copilot CLI # - Preserve all other fields (headers, type, etc.) @@ -453,30 +459,28 @@ jobs: mkdir -p "$(Agent.TempDirectory)/staging/logs" echo "=== Running AI agent with AWF network isolation ===" - echo "Allowed domains: *.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" + echo "Allowed domains: *.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,host.docker.internal,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" - # AWF provides L7 domain whitelisting via a rootless Docker topology. - # The named MCPG container is attached to AWF's internal network as a - # trusted endpoint; the agent has no route to the host. + # AWF provides L7 domain whitelisting via Squid proxy + Docker containers. + # --enable-host-access allows the AWF container to reach host services + # (MCPG and SafeOutputs) via host.docker.internal. # AWF auto-mounts /tmp:/tmp:rw into the container, so copilot binary, # agent prompt, and MCP config are placed under /tmp/awf-tools/. # Stream agent output in real-time while filtering VSO commands. # sed -u = unbuffered (line-by-line) so output appears immediately. # tee writes to both stdout (ADO pipeline log) and the artifact file. # pipefail (set above) ensures AWF's exit code propagates through the pipe. - # shellcheck disable=SC2046,SC2016 # ADO macros are substituted before bash; the single-quoted engine command is intentionally expanded by AWF inside the sandbox - "$(Pipeline.Workspace)/awf/awf" \ - --allow-domains "*.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" \ - --network-isolation \ - --topology-attach "awmg-mcpg" \ - --image-tag "0.27.32" \ + # shellcheck disable=SC2046 # $(AW_AZ_MOUNTS) is an ADO macro substituted before bash sees it, not bash command substitution; word-splitting the expanded value into separate --mount tokens is intentional + sudo -E "$(Pipeline.Workspace)/awf/awf" \ + --allow-domains "*.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,host.docker.internal,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" \ --skip-pull \ --env-all \ - $(AW_AZ_MOUNTS) \ + --enable-host-access \ + $(AW_AZ_MOUNTS) \ --container-workdir "$(Build.SourcesDirectory)" \ --log-level info \ --proxy-logs-dir "$(Agent.TempDirectory)/staging/logs/firewall" \ - -- 'export NO_PROXY="${NO_PROXY:+$NO_PROXY,}awmg-mcpg"; export no_proxy="$NO_PROXY"; /tmp/awf-tools/copilot --prompt "$(cat /tmp/awf-tools/agent-prompt.md)" --additional-mcp-config @/tmp/awf-tools/mcp-config.json --model gpt-5-mini --disable-builtin-mcps --no-ask-user --allow-all-tools --allow-tool github --allow-tool safeoutputs --allow-all-paths' \ + -- '/tmp/awf-tools/copilot --prompt "$(cat /tmp/awf-tools/agent-prompt.md)" --additional-mcp-config @/tmp/awf-tools/mcp-config.json --model gpt-5-mini --disable-builtin-mcps --no-ask-user --allow-all-tools --allow-tool github --allow-tool safeoutputs --allow-all-paths' \ 2>&1 \ | sed -u 's/##vso\[/[VSO-FILTERED] vso[/g; s/##\[/[VSO-FILTERED] [/g' \ | tee "$AGENT_OUTPUT_FILE" \ @@ -516,9 +520,16 @@ jobs: - bash: | # Stop MCPG container echo "Stopping MCPG..." - docker stop awmg-mcpg 2>/dev/null || true - echo "MCPG and stdio child containers stopped" - displayName: Stop MCPG + docker stop mcpg 2>/dev/null || true + echo "MCPG stopped" + + # Stop SafeOutputs HTTP server + if [ -n "$(SAFE_OUTPUTS_PID)" ]; then + echo "Stopping SafeOutputs (PID: $(SAFE_OUTPUTS_PID))..." + kill "$(SAFE_OUTPUTS_PID)" 2>/dev/null || true + echo "SafeOutputs stopped" + fi + displayName: Stop MCPG and SafeOutputs condition: always() - bash: | # Copy all logs to output directory for artifact upload @@ -621,7 +632,7 @@ jobs: - bash: | set -eo pipefail - AWF_VERSION="0.27.32" + AWF_VERSION="0.27.9" DOWNLOAD_DIR="$(Pipeline.Workspace)/awf" DOWNLOAD_URL="https://github.com/github/gh-aw-firewall/releases/download/v${AWF_VERSION}/awf-linux-x64" CHECKSUM_URL="https://github.com/github/gh-aw-firewall/releases/download/v${AWF_VERSION}/checksums.txt" @@ -638,14 +649,15 @@ jobs: chmod +x awf echo "##vso[task.prependpath]$(Pipeline.Workspace)/awf" ./awf --version - displayName: Download AWF (Agentic Workflow Firewall) v0.27.32 + displayName: Download AWF (Agentic Workflow Firewall) v0.27.9 - bash: | set -eo pipefail - docker pull ghcr.io/github/gh-aw-firewall/squid:0.27.32 - docker pull ghcr.io/github/gh-aw-firewall/agent:0.27.32 - docker pull ghcr.io/github/gh-aw-firewall/api-proxy:0.27.32 - displayName: Pre-pull AWF container images (v0.27.32) + docker pull ghcr.io/github/gh-aw-firewall/squid:0.27.9 + docker pull ghcr.io/github/gh-aw-firewall/agent:0.27.9 + docker tag ghcr.io/github/gh-aw-firewall/squid:0.27.9 ghcr.io/github/gh-aw-firewall/squid:latest + docker tag ghcr.io/github/gh-aw-firewall/agent:0.27.9 ghcr.io/github/gh-aw-firewall/agent:latest + displayName: Pre-pull AWF container images (v0.27.9) - bash: | mkdir -p "$(Build.SourcesDirectory)/safe_outputs" cp -a "$(Pipeline.Workspace)/agent_outputs_$(Build.BuildId)/." "$(Build.SourcesDirectory)/safe_outputs" @@ -706,11 +718,8 @@ jobs: THREAT_OUTPUT_FILE="$(Agent.TempDirectory)/threat-analysis-output.txt" # Stream threat analysis output in real-time with VSO command filtering - # shellcheck disable=SC2016 # The single-quoted engine command is intentionally expanded by AWF inside the sandbox - "$(Pipeline.Workspace)/awf/awf" \ - --allow-domains "*.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" \ - --network-isolation \ - --image-tag "0.27.32" \ + sudo -E "$(Pipeline.Workspace)/awf/awf" \ + --allow-domains "*.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,host.docker.internal,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" \ --skip-pull \ --env-all \ --container-workdir "$(Build.SourcesDirectory)" \ diff --git a/tests/safe-outputs/noop-target.lock.yml b/tests/safe-outputs/noop-target.lock.yml index 1e8dfcb4..e8682c1c 100644 --- a/tests/safe-outputs/noop-target.lock.yml +++ b/tests/safe-outputs/noop-target.lock.yml @@ -109,60 +109,24 @@ jobs: # These duplicate the compile-time values baked into the YAML, but MCPG's # Docker container requires MCP_GATEWAY_PORT and MCP_GATEWAY_DOMAIN env vars # to start — the ADO variable indirection satisfies that contract. - echo "##vso[task.setvariable variable=MCP_GATEWAY_PORT]8080" - echo "##vso[task.setvariable variable=MCP_GATEWAY_DOMAIN]awmg-mcpg" + echo "##vso[task.setvariable variable=MCP_GATEWAY_PORT]80" + echo "##vso[task.setvariable variable=MCP_GATEWAY_DOMAIN]host.docker.internal" # Write MCPG (MCP Gateway) configuration to a file cat > "$(Agent.TempDirectory)/staging/mcpg-config.json" << 'MCPG_CONFIG_EOF' { "mcpServers": { "safeoutputs": { - "type": "stdio", - "container": "ghcr.io/github/gh-aw-firewall/agent:0.27.32", - "entrypoint": "/usr/local/bin/ado-aw", - "entrypointArgs": [ - "mcp", - "--enabled-tools", - "missing-data", - "--enabled-tools", - "missing-tool", - "--enabled-tools", - "noop", - "--enabled-tools", - "report-incomplete", - "/safeoutputs", - "$(Build.SourcesDirectory)" - ], - "mounts": [ - "/tmp/awf-tools/ado-aw:/usr/local/bin/ado-aw:ro", - "$(Build.SourcesDirectory):$(Build.SourcesDirectory):rw", - "/tmp/awf-tools/staging:/safeoutputs:rw" - ], - "args": [ - "--network", - "none", - "--user", - "${MCP_RUNNER_UID}:${MCP_RUNNER_GID}", - "--cap-drop", - "ALL", - "--security-opt", - "no-new-privileges", - "--read-only", - "--tmpfs", - "/tmp:rw,nosuid,nodev,noexec", - "--pids-limit", - "256", - "-w", - "$(Build.SourcesDirectory)" - ], - "env": { - "HOME": "/tmp" + "type": "http", + "url": "http://localhost:${SAFE_OUTPUTS_PORT}/mcp", + "headers": { + "Authorization": "Bearer ${SAFE_OUTPUTS_API_KEY}" } } }, "gateway": { - "port": 8080, - "domain": "awmg-mcpg", + "port": 80, + "domain": "host.docker.internal", "apiKey": "${MCP_GATEWAY_API_KEY}", "payloadDir": "/tmp/gh-aw/mcp-payloads" } @@ -212,7 +176,7 @@ jobs: - bash: | set -eo pipefail - AWF_VERSION="0.27.32" + AWF_VERSION="0.27.9" DOWNLOAD_DIR="$(Pipeline.Workspace)/awf" DOWNLOAD_URL="https://github.com/github/gh-aw-firewall/releases/download/v${AWF_VERSION}/awf-linux-x64" CHECKSUM_URL="https://github.com/github/gh-aw-firewall/releases/download/v${AWF_VERSION}/checksums.txt" @@ -229,15 +193,16 @@ jobs: chmod +x awf echo "##vso[task.prependpath]$(Pipeline.Workspace)/awf" ./awf --version - displayName: Download AWF (Agentic Workflow Firewall) v0.27.32 + displayName: Download AWF (Agentic Workflow Firewall) v0.27.9 - bash: | set -eo pipefail - docker pull ghcr.io/github/gh-aw-firewall/squid:0.27.32 - docker pull ghcr.io/github/gh-aw-firewall/agent:0.27.32 - docker pull ghcr.io/github/gh-aw-firewall/api-proxy:0.27.32 + docker pull ghcr.io/github/gh-aw-firewall/squid:0.27.9 + docker pull ghcr.io/github/gh-aw-firewall/agent:0.27.9 + docker tag ghcr.io/github/gh-aw-firewall/squid:0.27.9 ghcr.io/github/gh-aw-firewall/squid:latest + docker tag ghcr.io/github/gh-aw-firewall/agent:0.27.9 ghcr.io/github/gh-aw-firewall/agent:latest docker pull ghcr.io/github/gh-aw-mcpg:v0.4.1 - displayName: Pre-pull AWF and MCPG container images (v0.27.32) + displayName: Pre-pull AWF and MCPG container images (v0.27.9) - task: UseNode@1 inputs: version: 22.x @@ -314,13 +279,50 @@ jobs: echo "Azure CLI prompt appended" displayName: Append Azure CLI prompt condition: ne(variables['AW_AZ_MOUNTS'], '') + - bash: | + SAFE_OUTPUTS_PORT=8100 + SAFE_OUTPUTS_API_KEY=$(openssl rand -base64 45 | tr -d '/+=') + echo "##vso[task.setvariable variable=SAFE_OUTPUTS_PORT]$SAFE_OUTPUTS_PORT" + echo "##vso[task.setvariable variable=SAFE_OUTPUTS_API_KEY;issecret=true]$SAFE_OUTPUTS_API_KEY" + + mkdir -p "$(Agent.TempDirectory)/staging/logs" + + # Start SafeOutputs as HTTP server in the background + # NOTE: --enabled-tools missing-data --enabled-tools missing-tool --enabled-tools noop --enabled-tools report-incomplete expands to either "" or "--enabled-tools X ... " + # (with trailing space). The value MUST be newline-free; is_safe_tool_name enforces this. + # Positional args (output_directory, bounding_directory) MUST come after all named + # options — clap parses them positionally and reordering would break the command. + nohup /tmp/awf-tools/ado-aw mcp-http \ + --port "$SAFE_OUTPUTS_PORT" \ + --api-key "$SAFE_OUTPUTS_API_KEY" \ + --enabled-tools missing-data --enabled-tools missing-tool --enabled-tools noop --enabled-tools report-incomplete "/tmp/awf-tools/staging" \ + "$(Build.SourcesDirectory)" \ + > "$(Agent.TempDirectory)/staging/logs/safeoutputs.log" 2>&1 & + SAFE_OUTPUTS_PID=$! + echo "##vso[task.setvariable variable=SAFE_OUTPUTS_PID]$SAFE_OUTPUTS_PID" + echo "SafeOutputs HTTP server started on port $SAFE_OUTPUTS_PORT (PID: $SAFE_OUTPUTS_PID)" + + # Wait for server to be ready + READY=false + # shellcheck disable=SC2034 # i is intentionally unused; wait-N-times loop + for i in $(seq 1 30); do + if curl -sf "http://localhost:$SAFE_OUTPUTS_PORT/health" > /dev/null 2>&1; then + echo "SafeOutputs HTTP server is ready" + READY=true + break + fi + sleep 1 + done + if [ "$READY" != "true" ]; then + echo "##vso[task.complete result=Failed]SafeOutputs HTTP server did not become ready within 30s" + exit 1 + fi + displayName: Start SafeOutputs HTTP server - bash: | # Substitute runtime values into MCPG config - MCP_RUNNER_UID=$(id -u) - MCP_RUNNER_GID=$(id -g) MCPG_CONFIG=$(sed \ - -e "s|\${MCP_RUNNER_UID}|$MCP_RUNNER_UID|g" \ - -e "s|\${MCP_RUNNER_GID}|$MCP_RUNNER_GID|g" \ + -e "s|\${SAFE_OUTPUTS_PORT}|$(SAFE_OUTPUTS_PORT)|g" \ + -e "s|\${SAFE_OUTPUTS_API_KEY}|$(SAFE_OUTPUTS_API_KEY)|g" \ -e "s|\${MCP_GATEWAY_API_KEY}|$(MCP_GATEWAY_API_KEY)|g" \ /tmp/awf-tools/staging/mcpg-config.json) @@ -330,22 +332,26 @@ jobs: # Remove any leftover container or stale output from a previous interrupted run # (--rm only cleans up on clean exit; OOM/SIGKILL may leave it behind) - docker rm -f awmg-mcpg 2>/dev/null || true + docker rm -f mcpg 2>/dev/null || true GATEWAY_OUTPUT="/tmp/gh-aw/mcp-config/gateway-output.json" mkdir -p "$(dirname "$GATEWAY_OUTPUT")" /tmp/gh-aw/mcp-logs rm -f "$GATEWAY_OUTPUT" - # Start MCPG on Docker's bridge network. AWF attaches this named, - # trusted container to its internal network after creating awf-net. + # Start MCPG Docker container on host network. # The Docker socket mount is required because MCPG spawns stdio-based MCP # servers as sibling containers. This grants significant host access — acceptable # here because the pipeline agent is already trusted and network-isolated by AWF. # + # WORKAROUND: Override entrypoint to bypass run_containerized.sh which has a + # validate_port_mapping() bug — it calls `docker inspect .NetworkSettings.Ports` + # which is empty with --network host (by design), causing a spurious error: + # [ERROR] Port 80 is not exposed from the container + # Upstream fix: https://github.com/github/gh-aw-mcpg/issues/TBD + # # stdout → gateway-output.json (machine-readable config, read after health check) echo "$MCPG_CONFIG" | docker run -i --rm \ - --name awmg-mcpg \ - --network bridge \ - -p 127.0.0.1:8080:8080 \ + --name mcpg \ + --network host \ --entrypoint /app/awmg \ -v /var/run/docker.sock:/var/run/docker.sock \ -e MCP_GATEWAY_PORT="$(MCP_GATEWAY_PORT)" \ @@ -354,7 +360,7 @@ jobs: \ \ ghcr.io/github/gh-aw-mcpg:v0.4.1 \ - --routed --listen 0.0.0.0:8080 --config-stdin --log-dir /tmp/gh-aw/mcp-logs \ + --routed --listen 0.0.0.0:80 --config-stdin --log-dir /tmp/gh-aw/mcp-logs \ > "$GATEWAY_OUTPUT" 2> >(tee /tmp/gh-aw/mcp-logs/stderr.log >&2) & MCPG_PID=$! echo "MCPG started (PID: $MCPG_PID)" @@ -363,7 +369,7 @@ jobs: READY=false # shellcheck disable=SC2034 # i is intentionally unused; wait-N-times loop for i in $(seq 1 30); do - if curl -sf "http://localhost:8080/health" > /dev/null 2>&1; then + if curl -sf "http://localhost:80/health" > /dev/null 2>&1; then echo "MCPG is ready" READY=true break @@ -400,8 +406,8 @@ jobs: # Convert gateway output to Copilot CLI mcp-config.json. # Mirrors gh-aw's convert_gateway_config_copilot.cjs: - # - Rewrite gateway URLs to the stable MCPG container name that AWF - # attaches to its internal network + # - Rewrite URLs from 127.0.0.1 to host.docker.internal (AWF container needs + # host.docker.internal to reach MCPG on the host; 127.0.0.1 is container loopback) # - Ensure tools: ["*"] on each server entry (Copilot CLI requirement) # - Mark generated MCPG entries as default/trusted servers for Copilot CLI # - Preserve all other fields (headers, type, etc.) @@ -421,30 +427,28 @@ jobs: mkdir -p "$(Agent.TempDirectory)/staging/logs" echo "=== Running AI agent with AWF network isolation ===" - echo "Allowed domains: *.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" + echo "Allowed domains: *.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,host.docker.internal,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" - # AWF provides L7 domain whitelisting via a rootless Docker topology. - # The named MCPG container is attached to AWF's internal network as a - # trusted endpoint; the agent has no route to the host. + # AWF provides L7 domain whitelisting via Squid proxy + Docker containers. + # --enable-host-access allows the AWF container to reach host services + # (MCPG and SafeOutputs) via host.docker.internal. # AWF auto-mounts /tmp:/tmp:rw into the container, so copilot binary, # agent prompt, and MCP config are placed under /tmp/awf-tools/. # Stream agent output in real-time while filtering VSO commands. # sed -u = unbuffered (line-by-line) so output appears immediately. # tee writes to both stdout (ADO pipeline log) and the artifact file. # pipefail (set above) ensures AWF's exit code propagates through the pipe. - # shellcheck disable=SC2046,SC2016 # ADO macros are substituted before bash; the single-quoted engine command is intentionally expanded by AWF inside the sandbox - "$(Pipeline.Workspace)/awf/awf" \ - --allow-domains "*.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" \ - --network-isolation \ - --topology-attach "awmg-mcpg" \ - --image-tag "0.27.32" \ + # shellcheck disable=SC2046 # $(AW_AZ_MOUNTS) is an ADO macro substituted before bash sees it, not bash command substitution; word-splitting the expanded value into separate --mount tokens is intentional + sudo -E "$(Pipeline.Workspace)/awf/awf" \ + --allow-domains "*.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,host.docker.internal,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" \ --skip-pull \ --env-all \ - $(AW_AZ_MOUNTS) \ + --enable-host-access \ + $(AW_AZ_MOUNTS) \ --container-workdir "$(Build.SourcesDirectory)" \ --log-level info \ --proxy-logs-dir "$(Agent.TempDirectory)/staging/logs/firewall" \ - -- 'export NO_PROXY="${NO_PROXY:+$NO_PROXY,}awmg-mcpg"; export no_proxy="$NO_PROXY"; /tmp/awf-tools/copilot --prompt "$(cat /tmp/awf-tools/agent-prompt.md)" --additional-mcp-config @/tmp/awf-tools/mcp-config.json --model gpt-5-mini --disable-builtin-mcps --no-ask-user --allow-all-tools --allow-tool github --allow-tool safeoutputs --allow-all-paths' \ + -- '/tmp/awf-tools/copilot --prompt "$(cat /tmp/awf-tools/agent-prompt.md)" --additional-mcp-config @/tmp/awf-tools/mcp-config.json --model gpt-5-mini --disable-builtin-mcps --no-ask-user --allow-all-tools --allow-tool github --allow-tool safeoutputs --allow-all-paths' \ 2>&1 \ | sed -u 's/##vso\[/[VSO-FILTERED] vso[/g; s/##\[/[VSO-FILTERED] [/g' \ | tee "$AGENT_OUTPUT_FILE" \ @@ -484,9 +488,16 @@ jobs: - bash: | # Stop MCPG container echo "Stopping MCPG..." - docker stop awmg-mcpg 2>/dev/null || true - echo "MCPG and stdio child containers stopped" - displayName: Stop MCPG + docker stop mcpg 2>/dev/null || true + echo "MCPG stopped" + + # Stop SafeOutputs HTTP server + if [ -n "$(SAFE_OUTPUTS_PID)" ]; then + echo "Stopping SafeOutputs (PID: $(SAFE_OUTPUTS_PID))..." + kill "$(SAFE_OUTPUTS_PID)" 2>/dev/null || true + echo "SafeOutputs stopped" + fi + displayName: Stop MCPG and SafeOutputs condition: always() - bash: | # Copy all logs to output directory for artifact upload @@ -589,7 +600,7 @@ jobs: - bash: | set -eo pipefail - AWF_VERSION="0.27.32" + AWF_VERSION="0.27.9" DOWNLOAD_DIR="$(Pipeline.Workspace)/awf" DOWNLOAD_URL="https://github.com/github/gh-aw-firewall/releases/download/v${AWF_VERSION}/awf-linux-x64" CHECKSUM_URL="https://github.com/github/gh-aw-firewall/releases/download/v${AWF_VERSION}/checksums.txt" @@ -606,14 +617,15 @@ jobs: chmod +x awf echo "##vso[task.prependpath]$(Pipeline.Workspace)/awf" ./awf --version - displayName: Download AWF (Agentic Workflow Firewall) v0.27.32 + displayName: Download AWF (Agentic Workflow Firewall) v0.27.9 - bash: | set -eo pipefail - docker pull ghcr.io/github/gh-aw-firewall/squid:0.27.32 - docker pull ghcr.io/github/gh-aw-firewall/agent:0.27.32 - docker pull ghcr.io/github/gh-aw-firewall/api-proxy:0.27.32 - displayName: Pre-pull AWF container images (v0.27.32) + docker pull ghcr.io/github/gh-aw-firewall/squid:0.27.9 + docker pull ghcr.io/github/gh-aw-firewall/agent:0.27.9 + docker tag ghcr.io/github/gh-aw-firewall/squid:0.27.9 ghcr.io/github/gh-aw-firewall/squid:latest + docker tag ghcr.io/github/gh-aw-firewall/agent:0.27.9 ghcr.io/github/gh-aw-firewall/agent:latest + displayName: Pre-pull AWF container images (v0.27.9) - bash: | mkdir -p "$(Build.SourcesDirectory)/safe_outputs" cp -a "$(Pipeline.Workspace)/agent_outputs_$(Build.BuildId)/." "$(Build.SourcesDirectory)/safe_outputs" @@ -674,11 +686,8 @@ jobs: THREAT_OUTPUT_FILE="$(Agent.TempDirectory)/threat-analysis-output.txt" # Stream threat analysis output in real-time with VSO command filtering - # shellcheck disable=SC2016 # The single-quoted engine command is intentionally expanded by AWF inside the sandbox - "$(Pipeline.Workspace)/awf/awf" \ - --allow-domains "*.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" \ - --network-isolation \ - --image-tag "0.27.32" \ + sudo -E "$(Pipeline.Workspace)/awf/awf" \ + --allow-domains "*.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,host.docker.internal,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" \ --skip-pull \ --env-all \ --container-workdir "$(Build.SourcesDirectory)" \ diff --git a/tests/safe-outputs/smoke-failure-reporter.lock.yml b/tests/safe-outputs/smoke-failure-reporter.lock.yml index ccb09ca5..05659148 100644 --- a/tests/safe-outputs/smoke-failure-reporter.lock.yml +++ b/tests/safe-outputs/smoke-failure-reporter.lock.yml @@ -118,62 +118,24 @@ jobs: # These duplicate the compile-time values baked into the YAML, but MCPG's # Docker container requires MCP_GATEWAY_PORT and MCP_GATEWAY_DOMAIN env vars # to start — the ADO variable indirection satisfies that contract. - echo "##vso[task.setvariable variable=MCP_GATEWAY_PORT]8080" - echo "##vso[task.setvariable variable=MCP_GATEWAY_DOMAIN]awmg-mcpg" + echo "##vso[task.setvariable variable=MCP_GATEWAY_PORT]80" + echo "##vso[task.setvariable variable=MCP_GATEWAY_DOMAIN]host.docker.internal" # Write MCPG (MCP Gateway) configuration to a file cat > "$(Agent.TempDirectory)/staging/mcpg-config.json" << 'MCPG_CONFIG_EOF' { "mcpServers": { "safeoutputs": { - "type": "stdio", - "container": "ghcr.io/github/gh-aw-firewall/agent:0.27.32", - "entrypoint": "/usr/local/bin/ado-aw", - "entrypointArgs": [ - "mcp", - "--enabled-tools", - "create-issue", - "--enabled-tools", - "missing-data", - "--enabled-tools", - "missing-tool", - "--enabled-tools", - "noop", - "--enabled-tools", - "report-incomplete", - "/safeoutputs", - "$(Build.SourcesDirectory)" - ], - "mounts": [ - "/tmp/awf-tools/ado-aw:/usr/local/bin/ado-aw:ro", - "$(Build.SourcesDirectory):$(Build.SourcesDirectory):rw", - "/tmp/awf-tools/staging:/safeoutputs:rw" - ], - "args": [ - "--network", - "none", - "--user", - "${MCP_RUNNER_UID}:${MCP_RUNNER_GID}", - "--cap-drop", - "ALL", - "--security-opt", - "no-new-privileges", - "--read-only", - "--tmpfs", - "/tmp:rw,nosuid,nodev,noexec", - "--pids-limit", - "256", - "-w", - "$(Build.SourcesDirectory)" - ], - "env": { - "HOME": "/tmp" + "type": "http", + "url": "http://localhost:${SAFE_OUTPUTS_PORT}/mcp", + "headers": { + "Authorization": "Bearer ${SAFE_OUTPUTS_API_KEY}" } } }, "gateway": { - "port": 8080, - "domain": "awmg-mcpg", + "port": 80, + "domain": "host.docker.internal", "apiKey": "${MCP_GATEWAY_API_KEY}", "payloadDir": "/tmp/gh-aw/mcp-payloads" } @@ -223,7 +185,7 @@ jobs: - bash: | set -eo pipefail - AWF_VERSION="0.27.32" + AWF_VERSION="0.27.9" DOWNLOAD_DIR="$(Pipeline.Workspace)/awf" DOWNLOAD_URL="https://github.com/github/gh-aw-firewall/releases/download/v${AWF_VERSION}/awf-linux-x64" CHECKSUM_URL="https://github.com/github/gh-aw-firewall/releases/download/v${AWF_VERSION}/checksums.txt" @@ -240,15 +202,16 @@ jobs: chmod +x awf echo "##vso[task.prependpath]$(Pipeline.Workspace)/awf" ./awf --version - displayName: Download AWF (Agentic Workflow Firewall) v0.27.32 + displayName: Download AWF (Agentic Workflow Firewall) v0.27.9 - bash: | set -eo pipefail - docker pull ghcr.io/github/gh-aw-firewall/squid:0.27.32 - docker pull ghcr.io/github/gh-aw-firewall/agent:0.27.32 - docker pull ghcr.io/github/gh-aw-firewall/api-proxy:0.27.32 + docker pull ghcr.io/github/gh-aw-firewall/squid:0.27.9 + docker pull ghcr.io/github/gh-aw-firewall/agent:0.27.9 + docker tag ghcr.io/github/gh-aw-firewall/squid:0.27.9 ghcr.io/github/gh-aw-firewall/squid:latest + docker tag ghcr.io/github/gh-aw-firewall/agent:0.27.9 ghcr.io/github/gh-aw-firewall/agent:latest docker pull ghcr.io/github/gh-aw-mcpg:v0.4.1 - displayName: Pre-pull AWF and MCPG container images (v0.27.32) + displayName: Pre-pull AWF and MCPG container images (v0.27.9) - task: UseNode@1 inputs: version: 22.x @@ -325,13 +288,50 @@ jobs: echo "Azure CLI prompt appended" displayName: Append Azure CLI prompt condition: ne(variables['AW_AZ_MOUNTS'], '') + - bash: | + SAFE_OUTPUTS_PORT=8100 + SAFE_OUTPUTS_API_KEY=$(openssl rand -base64 45 | tr -d '/+=') + echo "##vso[task.setvariable variable=SAFE_OUTPUTS_PORT]$SAFE_OUTPUTS_PORT" + echo "##vso[task.setvariable variable=SAFE_OUTPUTS_API_KEY;issecret=true]$SAFE_OUTPUTS_API_KEY" + + mkdir -p "$(Agent.TempDirectory)/staging/logs" + + # Start SafeOutputs as HTTP server in the background + # NOTE: --enabled-tools create-issue --enabled-tools missing-data --enabled-tools missing-tool --enabled-tools noop --enabled-tools report-incomplete expands to either "" or "--enabled-tools X ... " + # (with trailing space). The value MUST be newline-free; is_safe_tool_name enforces this. + # Positional args (output_directory, bounding_directory) MUST come after all named + # options — clap parses them positionally and reordering would break the command. + nohup /tmp/awf-tools/ado-aw mcp-http \ + --port "$SAFE_OUTPUTS_PORT" \ + --api-key "$SAFE_OUTPUTS_API_KEY" \ + --enabled-tools create-issue --enabled-tools missing-data --enabled-tools missing-tool --enabled-tools noop --enabled-tools report-incomplete "/tmp/awf-tools/staging" \ + "$(Build.SourcesDirectory)" \ + > "$(Agent.TempDirectory)/staging/logs/safeoutputs.log" 2>&1 & + SAFE_OUTPUTS_PID=$! + echo "##vso[task.setvariable variable=SAFE_OUTPUTS_PID]$SAFE_OUTPUTS_PID" + echo "SafeOutputs HTTP server started on port $SAFE_OUTPUTS_PORT (PID: $SAFE_OUTPUTS_PID)" + + # Wait for server to be ready + READY=false + # shellcheck disable=SC2034 # i is intentionally unused; wait-N-times loop + for i in $(seq 1 30); do + if curl -sf "http://localhost:$SAFE_OUTPUTS_PORT/health" > /dev/null 2>&1; then + echo "SafeOutputs HTTP server is ready" + READY=true + break + fi + sleep 1 + done + if [ "$READY" != "true" ]; then + echo "##vso[task.complete result=Failed]SafeOutputs HTTP server did not become ready within 30s" + exit 1 + fi + displayName: Start SafeOutputs HTTP server - bash: | # Substitute runtime values into MCPG config - MCP_RUNNER_UID=$(id -u) - MCP_RUNNER_GID=$(id -g) MCPG_CONFIG=$(sed \ - -e "s|\${MCP_RUNNER_UID}|$MCP_RUNNER_UID|g" \ - -e "s|\${MCP_RUNNER_GID}|$MCP_RUNNER_GID|g" \ + -e "s|\${SAFE_OUTPUTS_PORT}|$(SAFE_OUTPUTS_PORT)|g" \ + -e "s|\${SAFE_OUTPUTS_API_KEY}|$(SAFE_OUTPUTS_API_KEY)|g" \ -e "s|\${MCP_GATEWAY_API_KEY}|$(MCP_GATEWAY_API_KEY)|g" \ /tmp/awf-tools/staging/mcpg-config.json) @@ -341,22 +341,26 @@ jobs: # Remove any leftover container or stale output from a previous interrupted run # (--rm only cleans up on clean exit; OOM/SIGKILL may leave it behind) - docker rm -f awmg-mcpg 2>/dev/null || true + docker rm -f mcpg 2>/dev/null || true GATEWAY_OUTPUT="/tmp/gh-aw/mcp-config/gateway-output.json" mkdir -p "$(dirname "$GATEWAY_OUTPUT")" /tmp/gh-aw/mcp-logs rm -f "$GATEWAY_OUTPUT" - # Start MCPG on Docker's bridge network. AWF attaches this named, - # trusted container to its internal network after creating awf-net. + # Start MCPG Docker container on host network. # The Docker socket mount is required because MCPG spawns stdio-based MCP # servers as sibling containers. This grants significant host access — acceptable # here because the pipeline agent is already trusted and network-isolated by AWF. # + # WORKAROUND: Override entrypoint to bypass run_containerized.sh which has a + # validate_port_mapping() bug — it calls `docker inspect .NetworkSettings.Ports` + # which is empty with --network host (by design), causing a spurious error: + # [ERROR] Port 80 is not exposed from the container + # Upstream fix: https://github.com/github/gh-aw-mcpg/issues/TBD + # # stdout → gateway-output.json (machine-readable config, read after health check) echo "$MCPG_CONFIG" | docker run -i --rm \ - --name awmg-mcpg \ - --network bridge \ - -p 127.0.0.1:8080:8080 \ + --name mcpg \ + --network host \ --entrypoint /app/awmg \ -v /var/run/docker.sock:/var/run/docker.sock \ -e MCP_GATEWAY_PORT="$(MCP_GATEWAY_PORT)" \ @@ -365,7 +369,7 @@ jobs: \ \ ghcr.io/github/gh-aw-mcpg:v0.4.1 \ - --routed --listen 0.0.0.0:8080 --config-stdin --log-dir /tmp/gh-aw/mcp-logs \ + --routed --listen 0.0.0.0:80 --config-stdin --log-dir /tmp/gh-aw/mcp-logs \ > "$GATEWAY_OUTPUT" 2> >(tee /tmp/gh-aw/mcp-logs/stderr.log >&2) & MCPG_PID=$! echo "MCPG started (PID: $MCPG_PID)" @@ -374,7 +378,7 @@ jobs: READY=false # shellcheck disable=SC2034 # i is intentionally unused; wait-N-times loop for i in $(seq 1 30); do - if curl -sf "http://localhost:8080/health" > /dev/null 2>&1; then + if curl -sf "http://localhost:80/health" > /dev/null 2>&1; then echo "MCPG is ready" READY=true break @@ -411,8 +415,8 @@ jobs: # Convert gateway output to Copilot CLI mcp-config.json. # Mirrors gh-aw's convert_gateway_config_copilot.cjs: - # - Rewrite gateway URLs to the stable MCPG container name that AWF - # attaches to its internal network + # - Rewrite URLs from 127.0.0.1 to host.docker.internal (AWF container needs + # host.docker.internal to reach MCPG on the host; 127.0.0.1 is container loopback) # - Ensure tools: ["*"] on each server entry (Copilot CLI requirement) # - Mark generated MCPG entries as default/trusted servers for Copilot CLI # - Preserve all other fields (headers, type, etc.) @@ -432,30 +436,28 @@ jobs: mkdir -p "$(Agent.TempDirectory)/staging/logs" echo "=== Running AI agent with AWF network isolation ===" - echo "Allowed domains: *.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" + echo "Allowed domains: *.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,host.docker.internal,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" - # AWF provides L7 domain whitelisting via a rootless Docker topology. - # The named MCPG container is attached to AWF's internal network as a - # trusted endpoint; the agent has no route to the host. + # AWF provides L7 domain whitelisting via Squid proxy + Docker containers. + # --enable-host-access allows the AWF container to reach host services + # (MCPG and SafeOutputs) via host.docker.internal. # AWF auto-mounts /tmp:/tmp:rw into the container, so copilot binary, # agent prompt, and MCP config are placed under /tmp/awf-tools/. # Stream agent output in real-time while filtering VSO commands. # sed -u = unbuffered (line-by-line) so output appears immediately. # tee writes to both stdout (ADO pipeline log) and the artifact file. # pipefail (set above) ensures AWF's exit code propagates through the pipe. - # shellcheck disable=SC2046,SC2016 # ADO macros are substituted before bash; the single-quoted engine command is intentionally expanded by AWF inside the sandbox - "$(Pipeline.Workspace)/awf/awf" \ - --allow-domains "*.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" \ - --network-isolation \ - --topology-attach "awmg-mcpg" \ - --image-tag "0.27.32" \ + # shellcheck disable=SC2046 # $(AW_AZ_MOUNTS) is an ADO macro substituted before bash sees it, not bash command substitution; word-splitting the expanded value into separate --mount tokens is intentional + sudo -E "$(Pipeline.Workspace)/awf/awf" \ + --allow-domains "*.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,host.docker.internal,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" \ --skip-pull \ --env-all \ - $(AW_AZ_MOUNTS) \ + --enable-host-access \ + $(AW_AZ_MOUNTS) \ --container-workdir "$(Build.SourcesDirectory)" \ --log-level info \ --proxy-logs-dir "$(Agent.TempDirectory)/staging/logs/firewall" \ - -- 'export NO_PROXY="${NO_PROXY:+$NO_PROXY,}awmg-mcpg"; export no_proxy="$NO_PROXY"; /tmp/awf-tools/copilot --prompt "$(cat /tmp/awf-tools/agent-prompt.md)" --additional-mcp-config @/tmp/awf-tools/mcp-config.json --model gpt-5-mini --disable-builtin-mcps --no-ask-user --allow-all-tools --allow-tool github --allow-tool safeoutputs --allow-all-paths' \ + -- '/tmp/awf-tools/copilot --prompt "$(cat /tmp/awf-tools/agent-prompt.md)" --additional-mcp-config @/tmp/awf-tools/mcp-config.json --model gpt-5-mini --disable-builtin-mcps --no-ask-user --allow-all-tools --allow-tool github --allow-tool safeoutputs --allow-all-paths' \ 2>&1 \ | sed -u 's/##vso\[/[VSO-FILTERED] vso[/g; s/##\[/[VSO-FILTERED] [/g' \ | tee "$AGENT_OUTPUT_FILE" \ @@ -487,9 +489,16 @@ jobs: - bash: | # Stop MCPG container echo "Stopping MCPG..." - docker stop awmg-mcpg 2>/dev/null || true - echo "MCPG and stdio child containers stopped" - displayName: Stop MCPG + docker stop mcpg 2>/dev/null || true + echo "MCPG stopped" + + # Stop SafeOutputs HTTP server + if [ -n "$(SAFE_OUTPUTS_PID)" ]; then + echo "Stopping SafeOutputs (PID: $(SAFE_OUTPUTS_PID))..." + kill "$(SAFE_OUTPUTS_PID)" 2>/dev/null || true + echo "SafeOutputs stopped" + fi + displayName: Stop MCPG and SafeOutputs condition: always() - bash: | # Copy all logs to output directory for artifact upload @@ -592,7 +601,7 @@ jobs: - bash: | set -eo pipefail - AWF_VERSION="0.27.32" + AWF_VERSION="0.27.9" DOWNLOAD_DIR="$(Pipeline.Workspace)/awf" DOWNLOAD_URL="https://github.com/github/gh-aw-firewall/releases/download/v${AWF_VERSION}/awf-linux-x64" CHECKSUM_URL="https://github.com/github/gh-aw-firewall/releases/download/v${AWF_VERSION}/checksums.txt" @@ -609,14 +618,15 @@ jobs: chmod +x awf echo "##vso[task.prependpath]$(Pipeline.Workspace)/awf" ./awf --version - displayName: Download AWF (Agentic Workflow Firewall) v0.27.32 + displayName: Download AWF (Agentic Workflow Firewall) v0.27.9 - bash: | set -eo pipefail - docker pull ghcr.io/github/gh-aw-firewall/squid:0.27.32 - docker pull ghcr.io/github/gh-aw-firewall/agent:0.27.32 - docker pull ghcr.io/github/gh-aw-firewall/api-proxy:0.27.32 - displayName: Pre-pull AWF container images (v0.27.32) + docker pull ghcr.io/github/gh-aw-firewall/squid:0.27.9 + docker pull ghcr.io/github/gh-aw-firewall/agent:0.27.9 + docker tag ghcr.io/github/gh-aw-firewall/squid:0.27.9 ghcr.io/github/gh-aw-firewall/squid:latest + docker tag ghcr.io/github/gh-aw-firewall/agent:0.27.9 ghcr.io/github/gh-aw-firewall/agent:latest + displayName: Pre-pull AWF container images (v0.27.9) - bash: | mkdir -p "$(Build.SourcesDirectory)/safe_outputs" cp -a "$(Pipeline.Workspace)/agent_outputs_$(Build.BuildId)/." "$(Build.SourcesDirectory)/safe_outputs" @@ -677,11 +687,8 @@ jobs: THREAT_OUTPUT_FILE="$(Agent.TempDirectory)/threat-analysis-output.txt" # Stream threat analysis output in real-time with VSO command filtering - # shellcheck disable=SC2016 # The single-quoted engine command is intentionally expanded by AWF inside the sandbox - "$(Pipeline.Workspace)/awf/awf" \ - --allow-domains "*.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" \ - --network-isolation \ - --image-tag "0.27.32" \ + sudo -E "$(Pipeline.Workspace)/awf/awf" \ + --allow-domains "*.applicationinsights.azure.com,*.blob.core.windows.net,*.copilot.github.com,*.dev.azure.com,*.github.com,*.githubcopilot.com,*.githubusercontent.com,*.in.applicationinsights.azure.com,*.msauth.net,*.msauthimages.net,*.msftauth.net,*.pkgs.dev.azure.com,*.queue.core.windows.net,*.table.core.windows.net,*.visualstudio.com,*.vsassets.io,*.vsblob.visualstudio.com,*.vsrm.dev.azure.com,*.vssps.visualstudio.com,aex.dev.azure.com,aexus.dev.azure.com,aka.ms,api.github.com,config.edge.skype.com,copilot-proxy.githubusercontent.com,dc.services.visualstudio.com,dev.azure.com,github.com,graph.microsoft.com,host.docker.internal,login.live.com,login.microsoftonline.com,login.windows.net,management.azure.com,pkgs.dev.azure.com,rt.services.visualstudio.com,vsrm.dev.azure.com,vssps.dev.azure.com,vstoken.dev.azure.com" \ --skip-pull \ --env-all \ --container-workdir "$(Build.SourcesDirectory)" \ diff --git a/tests/safe-outputs/smoke-failure-reporter.md b/tests/safe-outputs/smoke-failure-reporter.md index 362de409..b3a94f9c 100644 --- a/tests/safe-outputs/smoke-failure-reporter.md +++ b/tests/safe-outputs/smoke-failure-reporter.md @@ -31,18 +31,30 @@ suite running in the AgentPlayground ADO project. ### Monitored pipelines -Query only these two pipelines (matched by exact `definition.name`): +Query only these three pipelines (matched by exact `definition.name`): - `Daily safe-output smoke canary` - `Daily smoke az CLI access` +- `ado-aw candidate compiler smoke` + +The first two are the registered ADO **definition names** from +`tests/safe-outputs/REGISTERED.md`; do not substitute the colon-bearing +front-matter `name:` values from their source Markdown. ### Tasks -1. Query the ADO REST `builds?api-version=7.1` endpoint of the - AgentPlayground project to fetch the most recent **completed** run - of each monitored pipeline. Use the read service connection's - `SYSTEM_ACCESSTOKEN`-equivalent bearer token already available to - you in the agent environment. +1. Resolve each monitored pipeline by exact name, then query the ADO REST + `builds?api-version=7.1` endpoint of the AgentPlayground project to fetch + its most recent **completed** run. Use the read service connection's + `SYSTEM_ACCESSTOKEN`-equivalent bearer token already available to you in + the agent environment. + - For `Daily safe-output smoke canary` and `Daily smoke az CLI access`, + use the latest completed run with no reason/branch restriction. + - For `ado-aw candidate compiler smoke`, include both + `branchName=refs/heads/main` and `reasonFilter=schedule`, plus + `statusFilter=completed`, `queryOrder=finishTimeDescending`, and + `$top=1`. Never report its PR or manual runs; those failures are surfaced + directly on their ADO validation. 2. For every run with `result != "succeeded"`: 1. Search open issues on `jamesadevine/ado-aw-issues` for one whose title starts with `[smoke-failure] `. If one already