Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ fail-closed and only pauses when the agent actually proposed a reviewed output.
│ │ │ ├── 0003_flatten_work_item_config.rs # Legacy work-item config flatten codemod
│ │ │ ├── 0004_legacy_path_markers.rs # Migrate {{ workspace }}/{{ working_directory }}/{{ trigger_repo_directory }} markers → explicit ADO path exprs (resolved from workspace:/repos:)
│ │ │ ├── 0005_drop_build_attachment_allowed_build_ids.rs # Remove inert safe-outputs.upload-build-attachment.allowed-build-ids key (build attachments are current-run only)
│ │ │ ├── 0006_promote_debug_create_issue.rs # Move legacy ado-aw-debug.create-issue into public safe-outputs.create-issue with auth bridge
│ │ │ └── helpers.rs # take_key, insert_no_overwrite, rename_key, ConflictPolicy
│ │ ├── codemod_integration_test.rs # White-box rewrite-path tests (stub registry injection)
│ │ ├── types.rs # Front matter grammar and types
Expand Down Expand Up @@ -337,9 +338,9 @@ index to jump to the right page.
`$(System.AccessToken)` semantics, the "Limit job authorization
scope to current project" toggle, permission-bitmask decoder,
REST recipe for inspecting ACEs, and the three fix paths.
- [`docs/ado-aw-debug.md`](docs/ado-aw-debug.md) — debug-only `ado-aw-debug:`
front-matter section (`skip-integrity`, `create-issue` for filing GitHub
issues from dogfood pipelines). NOT a regular safe-output.
- [`docs/ado-aw-debug.md`](docs/ado-aw-debug.md) — debug-only
`ado-aw-debug.skip-integrity` front-matter control. GitHub issue filing is a
regular safe output.
- [`docs/supply-chain.md`](docs/supply-chain.md) — optional `supply-chain:`
front-matter section that mirrors the compiler, AWF binary, ado-script
bundle, and AWF/MCPG images from an internal Azure DevOps Artifacts feed,
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -731,8 +731,8 @@ index to jump to the right page.
- [`docs/supply-chain.md`](docs/supply-chain.md) — optional `supply-chain:`
front-matter section for internal feed/registry mirrors and exact
provenance-checked pipeline artifacts.
- [`docs/ado-aw-debug.md`](docs/ado-aw-debug.md) debug-only `ado-aw-debug:`
front-matter section (`skip-integrity`, `create-issue`).
- [`docs/ado-aw-debug.md`](docs/ado-aw-debug.md) - debug-only
`ado-aw-debug.skip-integrity` front-matter control.

**Compiler internals & operations**

Expand Down
202 changes: 28 additions & 174 deletions docs/ado-aw-debug.md
Original file line number Diff line number Diff line change
@@ -1,195 +1,49 @@
# `ado-aw-debug:` Debug-only front-matter section
# `ado-aw-debug:` - Debug-only front-matter section

_Part of the [ado-aw documentation](../AGENTS.md)._

> ⚠️ **This section is for dogfood pipelines only.**
> Anything declared under `ado-aw-debug:` is **not** part of the regular
> agent surface and is not recommended for general use. Knobs here exist so
> the team can validate `githubnext/ado-aw` changes against real Azure
> DevOps pipelines and file failures back to GitHub for triage. Each knob
> bypasses or weakens a normal safety control.
> This section is for compiler dogfood and local diagnostics. It is not a
> regular safe-output surface.

The compiler accepts a top-level `ado-aw-debug:` block in agent front
matter. Currently exposed knobs:

| Knob | Purpose | Default |
| --- | --- | --- |
| `skip-integrity` | Omit the "Verify pipeline integrity" step from the generated YAML. OR-ed with the `--skip-integrity` CLI flag. | `false` |
| `create-issue` | Enable the [debug-only `create-issue`](#create-issue) safe output. | absent (disabled) |

Unrecognised keys under `ado-aw-debug:` cause a compile-time error
(`#[serde(deny_unknown_fields)]`).

## `create-issue`

Files a GitHub issue against an operator-configured target repository.
Used to surface failures from ADO-hosted dogfood pipelines back to
`githubnext/ado-aw` for triage.

### Why it's gated

`create-issue` is **default-deny** at three layers:

1. **MCP layer.** The SafeOutputs MCP server lists `create-issue` in
[`DEBUG_ONLY_TOOLS`][debug-only-tools], so the route is removed from
the tool router unless the compiler explicitly opts in via
`--enabled-tools`.
2. **Compiler layer.** `--enabled-tools create-issue` is only emitted
when `ado-aw-debug.create-issue:` is present in front matter. The
compiler also rejects `safe-outputs.create-issue:` outright, so the
tool can't be smuggled in via the regular safe-outputs surface.
3. **Executor layer.** Stage 3 maintains a separate
`ExecutionContext.debug_enabled_tools` set populated only from
`ado-aw-debug:`. The executor refuses any NDJSON `create-issue`
entry that isn't in that set, so a forged or smuggled NDJSON entry
fails closed before any token is read.

[debug-only-tools]: ../src/safe_outputs/mod.rs

### Front-matter schema
The compiler accepts:

```yaml
ado-aw-debug:
create-issue:
target-repo: githubnext/ado-aw # REQUIRED. Operator-only; agent has no override.
title-prefix: "[pipeline-failure] " # Optional; prepended to every agent title.
labels: # Optional; static labels always applied.
- pipeline-failure
- automated
allowed-labels: # Optional; default-deny — see below.
- "agent-*"
- "pipeline-failure"
assignees: # Optional; static assignees always applied.
- "jamesdevine"
max: 3 # Optional; per-run budget. Default 1.
skip-integrity: true
```

* **`target-repo`** is required. Format `owner/repo`. The agent has no
parameter to override it; you cannot redirect issues to a different
repository at runtime.
* **`title-prefix`** is appended at execution time. The final title length
(prefix + agent title) must be ≤ 256 characters; longer titles fail at
Stage 3.
* **`labels`** are applied unconditionally to every issue, on top of any
agent-supplied labels that pass `allowed-labels`.
* **`allowed-labels`** is **default-deny**: an empty or absent list means
**no agent-supplied labels are accepted**. To accept any agent label,
set `allowed-labels: ["*"]` explicitly. Patterns may include `*`
wildcards (e.g. `"agent-*"`).
* **Allowed-label matching is case-insensitive.** It uses the same
`tag_matches_pattern` helper as ADO tag allow-lists. GitHub labels are
case-sensitive, so `allowed-labels: ["safe"]` will also admit
`SAFE` and `Safe` — keep that in mind when modelling policy.
* **`assignees`** are merged with agent-supplied assignees. There is
intentionally no `allowed-assignees` allowlist in v1; if you need
one, configure assignees only via the static `assignees:` list and
skip the agent parameter.
* **`max`** controls per-run budget the same way it does for other
safe-output tools.
Unrecognized keys fail compilation (`#[serde(deny_unknown_fields)]`).

### Agent-supplied parameters
## `skip-integrity`

The agent calls the `create-issue` MCP tool with:
Equivalent to passing `--skip-integrity` to `ado-aw compile`. Setting either
omits the generated **Verify pipeline integrity** step.

```jsonc
{
"title": "Pipeline failure on main",
"body": "<markdown body, ≥ 30 chars>",
"labels": ["pipeline-failure"], // optional
"assignees": ["copilot"] // optional
}
```
The integrity step downloads the same ado-aw version used at compile time and
runs `ado-aw check` against the committed pipeline. Without it, a modified
compiled YAML file is not detected at runtime. Use this option only for
short-lived dogfood pipelines.

The MCP-side `Validate` impl rejects ADO pipeline-command sequences in
labels and assignees (see [src/safe_outputs/create_issue.rs](../src/safe_outputs/create_issue.rs)).
Stage 3 also neutralises `##vso[…]` in any error messages it produces, so
agent-supplied content cannot escape the executor's stdout.
## Retired: `ado-aw-debug.create-issue`

### Pipeline variable: `ADO_AW_DEBUG_GITHUB_TOKEN`

Stage 3 authenticates against GitHub using the
**`ADO_AW_DEBUG_GITHUB_TOKEN`** ADO pipeline variable. The compiler emits
GitHub issue filing is now a regular safe output:

```yaml
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken) # default executor token (or $(SC_WRITE_TOKEN) if permissions.write is set)
ADO_AW_DEBUG_GITHUB_TOKEN: $(ADO_AW_DEBUG_GITHUB_TOKEN) # only when ado-aw-debug.create-issue is set
safe-outputs:
create-issue:
target-repo: githubnext/ado-aw
```

into the executor step's `env:` block. The token is **not** exposed to
the agent in Stage 1 — the read-only `GITHUB_TOKEN` the agent sees is a
separate variable wired through `engine.env` and used only for GitHub
MCP read access.

### Setting up the PAT
The front-matter codemod moves legacy `ado-aw-debug.create-issue` configuration
to `safe-outputs.create-issue`. When no public GitHub auth is already
configured, it also adds:

1. **Generate a fine-grained PAT** scoped to **only** `target-repo` (e.g.
`githubnext/ado-aw`). Required permissions:
* Repository access: only the target repo.
* Permissions: **Issues** = Read and write. Nothing else.
2. **Store as a secret pipeline variable** named exactly
`ADO_AW_DEBUG_GITHUB_TOKEN`. Mark it secret. Do **not** copy it into
`engine.env` or any non-secret variable.
3. **Confirm the operator-configured target-repo matches the PAT scope.**
The compiler validator only checks shape (`owner/repo`); it cannot
verify the PAT has access. If the PAT lacks Issues:write, the Stage 3
call fails with the GitHub API error and Stage 3 reports
`succeeded with issues`.
4. `ado-aw configure` does **not** automate this variable today — set it
manually in the ADO pipeline definition.

### Auto-footer

Every issue gets an auto-appended traceability footer that looks like:

```markdown
<!-- ado-aw -->
---
Pipeline: `dogfood-failure-reporter`
Run: <https://dev.azure.com/myorg/MyProject/_build/results?buildId=42>
Trigger: `Manual`
```yaml
safe-outputs:
github-token: "$(ADO_AW_DEBUG_GITHUB_TOKEN)"
```

The `<!-- ado-aw -->` marker is stable so that future tooling can locate
the generated content without parsing prose. The footer is built from
`BUILD_BUILDID`, `BUILD_DEFINITIONNAME`, `BUILD_REASON`,
`SYSTEM_TEAMFOUNDATIONCOLLECTIONURI` and `SYSTEM_TEAMPROJECT` — these are
present whenever Stage 3 runs inside an ADO pipeline.

If your pipeline / org / project names are sensitive, do not enable
`create-issue` against a public repo.

### Security checklist

- [ ] Target repo's GitHub PAT is scoped to that repo only and only has
Issues:write.
- [ ] `ADO_AW_DEBUG_GITHUB_TOKEN` is stored as a secret pipeline
variable, never hard-coded or printed.
- [ ] `allowed-labels` is set explicitly. Empty means default-deny;
`["*"]` accepts any agent label — pick deliberately.
- [ ] `target-repo` is private if the agent's prompts or pipeline
metadata are sensitive (the auto-footer publishes ADO run URLs and
pipeline names).
- [ ] `skip-integrity` is **not** enabled in pipelines triggered by
untrusted PRs.

## `skip-integrity`

Equivalent to passing `--skip-integrity` on the `ado-aw compile` CLI.
Setting either OR setting both omits the `Verify pipeline integrity`
step from the generated YAML.

The integrity step downloads the same `ado-aw` binary the pipeline was
compiled with and runs `ado-aw check` against the committed pipeline
file. Without it, a tampered `*.yml` won't be caught at run time.

Use this only for short-lived dogfood pipelines where you're iterating
on the compiler and re-compiling frequently.

## See also

- [`docs/safe-outputs.md`](safe-outputs.md) — regular safe-outputs
surface (`create-issue` is **not** in it).
- [`docs/cli.md`](cli.md) — `--skip-integrity` CLI flag.
- [`docs/ir.md`](ir.md) — typed pipeline IR and how debug-only choices such as
integrity-check omission are represented in generated steps.
That explicit token reference preserves the existing ADO secret during
migration; it is not a debug runtime path. New workflows should use the
`ADO_AW_GITHUB_TOKEN` default or GitHub App auth. See
[`docs/safe-outputs.md`](safe-outputs.md#github-issue-safe-outputs).
12 changes: 6 additions & 6 deletions docs/ado-script.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,11 +56,11 @@ pipeline** as runtime helpers. Today it produces thirteen bundles:
review is configured), and attaches it to the build's
`ado-aw-safe-outputs` summary tab via `##vso[task.uploadsummary]`. See
[`safe-outputs.md`](safe-outputs.md).
- `github-app-token.js` — GitHub App token minter that runs immediately
before the Copilot invocation in the **Agent and Detection jobs** when
`engine.github-app-token` is configured. Builds an RS256 JWT from the App ID
(argv) + private key (masked env secret), resolves the installation for the
owner, exchanges it for an installation access token, and exposes it as a
- `github-app-token.js` — GitHub App token minter used by Agent/Detection and
GitHub issue SafeOutputs. Builds an RS256 JWT from the App ID (argv) +
private key (masked env secret), resolves the installation for the owner,
exchanges it for a repository/permission-scoped installation access token,
and exposes it as a
masked same-job `GITHUB_APP_TOKEN`. Invoked again with a `revoke` argument
after the Copilot run (best-effort) to delete the token via
`DELETE /installation/token`. Compiler-owned, non-secret inputs (`--app-id`,
Expand Down Expand Up @@ -555,7 +555,7 @@ scripts/ado-script/
│ │ ├── index.ts # main(): inspect upstream results + safe-outputs manifest → file/append work items
│ │ └── __tests__/ # unit tests for signal detection and work-item filing behaviour
│ ├── github-app-token/ # github-app-token.js entry point + GitHub App token minter
│ │ ├── index.ts # main(): RS256 JWT → resolve installation → mint installation token → masked GITHUB_APP_TOKEN
│ │ ├── index.ts # main(): RS256 JWT → resolve installation → mint scoped installation token → masked same-job variable
│ │ └── __tests__/ # unit tests for JWT signing / installation resolution / token minting
│ ├── prepare-pr-base/ # prepare-pr-base.js entry point + create-pull-request base-ref fetch/deepen
│ │ ├── index.ts # main(): fetch/deepen target branch + set origin/HEAD so mcp.rs finds a diff base
Expand Down
16 changes: 15 additions & 1 deletion docs/codemods.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,8 @@ src/compile/codemods/
├── 0002_pool_object_form.rs # Legacy scalar pool → explicit object form codemod
├── 0003_flatten_work_item_config.rs # Legacy work-item config flatten codemod
├── 0004_legacy_path_markers.rs # {{ workspace }} / {{ working_directory }} / {{ trigger_repo_directory }} → explicit ADO path exprs
└── 0005_drop_build_attachment_allowed_build_ids.rs # remove no-op upload-build-attachment.allowed-build-ids
├── 0005_drop_build_attachment_allowed_build_ids.rs # remove no-op upload-build-attachment.allowed-build-ids
└── 0006_promote_debug_create_issue.rs # move debug GitHub issue filing to regular safe-outputs
```

(New codemods are appended as `<NNNN>_<id>.rs` files.)
Expand Down Expand Up @@ -408,6 +409,19 @@ standard codemod compile warning tells the author it was auto-removed. It does
stays meaningful (pipeline artifacts use the Build Artifacts `Create` API, which
can target an arbitrary build).

## GitHub `create-issue` promotion (`0006_promote_debug_create_issue`)

The dogfood-only `ado-aw-debug.create-issue` surface was retired when GitHub
issue filing became a regular safe output. The codemod moves the configuration
to `safe-outputs.create-issue`, preserves other debug fields such as
`skip-integrity`, and removes an empty `ado-aw-debug` mapping.

If no public GitHub auth is already present, it adds
`safe-outputs.github-token: "$(ADO_AW_DEBUG_GITHUB_TOKEN)"` so an existing ADO
secret continues to work through the normal Stage 3 auth path. When both old
and new `create-issue` keys exist, the codemod fails with a manual-migration
error rather than overwriting either value.

## Tests

The codemod framework is covered by three layers of tests:
Expand Down
Loading
Loading