Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 51 additions & 1 deletion docs/setup/agentic-overrides.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
- [Pre-empt a decision-table row](#pre-empt-a-decision-table-row)
- [What an override file should explain](#what-an-override-file-should-explain)
- [How a framework skill consults overrides](#how-a-framework-skill-consults-overrides)
- [One-shot defaults run](#one-shot-defaults-run)
- [Hard rules](#hard-rules)
- [Reconciliation on framework upgrade](#reconciliation-on-framework-upgrade)
- [Upstreaming an override](#upstreaming-an-override)
Expand Down Expand Up @@ -162,6 +163,16 @@ maintainer (or a future agent on a later run):
Every framework skill that supports overrides starts each
invocation with this opening protocol:

0. **Check for `--no-overrides`.** If the invocation passed
the `--no-overrides` flag, skip steps 1–4 entirely — both
override surfaces — and run against framework defaults for
this invocation; see
[One-shot defaults run](#one-shot-defaults-run). The safety
baseline still applies. Still do step 5, reporting that the
run used `--no-overrides` and naming the override files that
existed but were not consulted, so the audit trail records
the bypass rather than looking like a run with no overrides
on disk. Otherwise, continue with step 1.
1. Read `<adopter-repo>/.apache-magpie-local/<this-skill>.md`
(personal, gitignored) if it exists.
2. Read `<adopter-repo>/.apache-magpie-overrides/<this-skill>.md`
Expand All @@ -187,6 +198,43 @@ that explicitly in its `SKILL.md`. The
sub-action surfaces this gap and suggests opening a
framework-side issue requesting the hook.

## One-shot defaults run

Pass `--no-overrides` to any framework skill that supports
overrides to run that single invocation against framework
defaults, ignoring any override files that exist on disk.
The override files are **not** modified or deleted — they
are simply not consulted for this run.

```text
/magpie-pr-management-triage --no-overrides
/magpie-security-issue-triage --no-overrides
```

When `--no-overrides` is present the skill's opening
protocol changes: **skip steps 1–3 entirely** — do not
read, surface, apply, or recap any override file. The skill
proceeds immediately with its framework defaults, as if
neither `.apache-magpie-overrides/<skill>.md` nor any
personal override existed.

The **safety baseline** (confidentiality, privacy, and
security rules baked into the framework) still applies in
full — `--no-overrides` is not a safety bypass. It only
removes the adopter-customisation layer.

Typical use cases:

- **Debugging**: reproduce the framework's default
behaviour to determine whether an override is causing
an unexpected result.
- **One-off clean run**: a release manager wants a
pristine triage run without their personal overrides
for this single check-in.
- **Override authoring**: run with defaults first to see
what the framework produces, then compare against a
run that applies the override under construction.

## Hard rules

These are baked into agent instructions across the framework.
Expand Down Expand Up @@ -297,7 +345,9 @@ this by:

- [`setup` skill](../../skills/setup/SKILL.md) — the entry point
that manages the snapshot, scaffolds overrides, and adds the
`.gitignore` entries for both override directories.
`.gitignore` entries for both override directories. Lists
`--no-overrides` in its Inputs table as a recognised
framework-level flag.
- [`overrides.md` sub-action](../../skills/setup/overrides.md) —
interactive override creation (lets the user choose between the
personal-local and committed surfaces).
Expand Down
1 change: 1 addition & 0 deletions skills/setup/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -435,6 +435,7 @@ first, then continue.
| `agents:<list>` | Comma-separated **agent targets** to wire symlinks into ([`agents.md`](agents.md) registry ids: `universal`, `claude-code`, `github`, `windsurf`, `goose`, …). Default on `adopt`/`upgrade`: auto — the always-on neutral set (`universal` + `claude-code` + `github`) plus any other registry dir already present in the repo. When passed, **replaces** the auto-detected set for that run, except `universal` (`.agents/skills/`) which is always retained because it is the canonical home every other target relays into — dropping it would leave the relays dangling. |
| `skill-families:<list>` | Comma-separated **opt-in** families to symlink — any of the opt-in families declared by a `family:` frontmatter key in the snapshot (today: `security`, `pr-management`, `issue`, `release-management`, `repo-health`, `pairing`, `mentoring`, `contributor-growth`). Default on `adopt`: prompt (see [`adopt.md` Step 5](adopt.md#step-5--pick-the-skill-families-and-mcp-servers)). Default on `upgrade`: read the families list from `<committed-lock>` / `<local-lock>`, **auto-include any opt-in family the framework has introduced since the lock was written** (recorded back into the lock), and **ensure every framework skill in the effective family set has a valid symlink** — create or repair missing / broken symlinks, not just add new ones. The flag never accepts the always-on families (`setup`, `utilities`); per [Golden rule 8](#golden-rules) those are wired up unconditionally on every run and there is no way to ask for them or opt out. |
| `--purge-overrides` | *(unadopt only)* Also `git rm -r` `.apache-magpie-overrides/`. Default: preserve. |
| `--no-overrides` | *(any framework skill)* Skip override-file lookup for this single invocation. Runs the skill against framework defaults; override files on disk are not read, modified, or deleted. The safety baseline (confidentiality, privacy, security) still applies. See [One-shot defaults run](../../docs/setup/agentic-overrides.md#one-shot-defaults-run). |
| `dry-run` | Show what the skill would do without writing anything. |

## What this skill is NOT for
Expand Down
31 changes: 24 additions & 7 deletions tools/skill-evals/evals/setup/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,27 +5,31 @@

Behavioral evals for the `setup` skill.

## Suites (9 cases total)
## Suites (12 cases total)

| Suite | Step | Cases | What it covers |
|---|---|---|---|
| step-verify-drift | verify.md § Check 3 (drift) | 5 | clean, method/URL mismatch, ref mismatch, svn-zip SHA-512 mismatch, local lock missing |
| step-overrides-surface | overrides.md § Step 0b | 4 | adopted no flag (offer choice), --local flag (personal), not adopted (personal only), both surfaces exist |
| step-override-bypass | agentic-overrides.md § One-shot defaults run | 3 | `--no-overrides` flag + override exists, `--no-overrides` + no override, no flag + override exists |

## Run

`--cli` is required or nothing is graded; use `--directory`, not
`--project`, and run from the repo root.

```bash
# All cases
uv run --project tools/skill-evals skill-eval \
tools/skill-evals/evals/setup/
uv run --directory tools/skill-evals skill-eval --cli "claude -p" \
evals/setup/

# Single suite
uv run --project tools/skill-evals skill-eval \
tools/skill-evals/evals/setup/step-verify-drift/fixtures/
uv run --directory tools/skill-evals skill-eval --cli "claude -p" \
evals/setup/step-override-bypass/

# Single case
uv run --project tools/skill-evals skill-eval \
tools/skill-evals/evals/setup/step-verify-drift/fixtures/case-1-clean
uv run --directory tools/skill-evals skill-eval --cli "claude -p" \
evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists
```

## Notes
Expand All @@ -36,3 +40,16 @@ uv run --project tools/skill-evals skill-eval \
vs-shared surface selection introduced by the `magpie-local-convention`
work item. The default surface when the repo is adopted and no flag is
passed is `"offer-choice"`; `override_path` reports the personal default.
- `step-override-bypass` cases are fully auto-comparable: `decision` and
`safety_baseline` are enumerated strings, and `reason` is checked by
deterministic `regex` predicates in `assertions.json`
(`has_bypass_reason` for the skip cases, `has_apply_reason` for the
apply case) — no grader or MANUAL step is required.
The two predicates discriminate on *direction*, not on the flag name.
Keying on `no-overrides` would be useless here: the flag name appears
in a correct reason and in a reason arguing the exact opposite, so such
a pattern passes either way. Each predicate therefore requires the
matching verb (skipped/not-consulted versus applied/consulted) and
rejects the opposing phrasing. When editing them, check both
directions — that a right answer still passes *and* that a reason
arguing the other decision fails.
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"has_bypass_reason": {
"type": "regex",
"field": "reason",
"pattern": "^(?=.*override)(?=.*\\b(?:skip|skips|skipped|skipping|bypass|bypassed|bypassing|ignore|ignores|ignored|ignoring|not\\s+(?:be\\s+)?(?:read|consulted|applied)|(?:were|was)\\s+not|without)\\b)(?!.*\\b(?:read\\s+and\\s+applied|applied\\s+the\\s+overrides?|overrides?\\s+(?:were|was)\\s+applied)\\b).*",
"flags": "is"
},
"has_apply_reason": {
"type": "regex",
"field": "reason",
"pattern": "^(?=.*override)(?=.*\\b(?:applied|applying|consulted|honoured|honored)\\b)(?!.*\\b(?:skipped|bypassed|ignored)\\s+(?:the\\s+)?overrides?\\b)(?!.*\\boverrides?\\s+(?:were|was)\\s+not\\b)(?!.*\\bnot\\s+(?:be\\s+)?(?:read|consulted|applied)\\b).*",
"flags": "is"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"decision": "skip-overrides", "safety_baseline": "enforced", "has_bypass_reason": true}
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->

Invocation: `/magpie-pr-management-triage --no-overrides`

Override file present on disk:
`.apache-magpie-overrides/pr-management-triage.md`
Content: "Override 1 — Always tag @core-maintainers on first comment"

Should the override file be read and applied for this invocation?
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"decision": "skip-overrides", "safety_baseline": "enforced", "has_bypass_reason": true}
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->

Invocation: `/magpie-security-issue-triage --no-overrides`

Override file present on disk: none
`.apache-magpie-overrides/security-issue-triage.md` does not exist.

Should the skill run with framework defaults for this invocation?
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"decision": "apply-overrides", "safety_baseline": "enforced", "has_apply_reason": true}
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->

Invocation: `/magpie-pr-management-triage`

Override file present on disk:
`.apache-magpie-overrides/pr-management-triage.md`
Content: "Override 1 — Treat backport PRs as already-triaged"

No `--no-overrides` flag was passed.

Should the override file be read and applied for this invocation?
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->

## Output format

Return ONLY valid JSON with this structure:

```json
{
"decision": "skip-overrides" | "apply-overrides",
"safety_baseline": "enforced",
"reason": "<one-sentence explanation>"
}
```

- `"skip-overrides"` — `--no-overrides` flag was present; override files
are not read or applied this invocation. `safety_baseline` must be
`"enforced"`.
- `"apply-overrides"` — `--no-overrides` flag was absent; override files
are read and applied per the normal protocol. `safety_baseline` must
be `"enforced"`.
- `"safety_baseline"` is always `"enforced"` — `--no-overrides` does not
disable the framework's confidentiality, privacy, or security rules.
- `"reason"` should be one concise sentence.
- Do not include any text outside the JSON object.
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"skill_md": "docs/setup/agentic-overrides.md",
"step_heading": "## One-shot defaults run"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->

## Override bypass decision

{report}

Apply the one-shot defaults run rules and return JSON only.