diff --git a/docs/setup/agentic-overrides.md b/docs/setup/agentic-overrides.md index ccbb51c8..c519d1a1 100644 --- a/docs/setup/agentic-overrides.md +++ b/docs/setup/agentic-overrides.md @@ -15,6 +15,7 @@ - [Pre-empt a decision-table row](#pre-empt-a-decision-table-row) - [What an override file should explain](#what-an-override-file-should-explain) - [How a framework skill consults overrides](#how-a-framework-skill-consults-overrides) + - [One-shot defaults run](#one-shot-defaults-run) - [Hard rules](#hard-rules) - [Reconciliation on framework upgrade](#reconciliation-on-framework-upgrade) - [Upstreaming an override](#upstreaming-an-override) @@ -162,6 +163,16 @@ maintainer (or a future agent on a later run): Every framework skill that supports overrides starts each invocation with this opening protocol: +0. **Check for `--no-overrides`.** If the invocation passed + the `--no-overrides` flag, skip steps 1–4 entirely — both + override surfaces — and run against framework defaults for + this invocation; see + [One-shot defaults run](#one-shot-defaults-run). The safety + baseline still applies. Still do step 5, reporting that the + run used `--no-overrides` and naming the override files that + existed but were not consulted, so the audit trail records + the bypass rather than looking like a run with no overrides + on disk. Otherwise, continue with step 1. 1. Read `/.apache-magpie-local/.md` (personal, gitignored) if it exists. 2. Read `/.apache-magpie-overrides/.md` @@ -187,6 +198,43 @@ that explicitly in its `SKILL.md`. The sub-action surfaces this gap and suggests opening a framework-side issue requesting the hook. +## One-shot defaults run + +Pass `--no-overrides` to any framework skill that supports +overrides to run that single invocation against framework +defaults, ignoring any override files that exist on disk. +The override files are **not** modified or deleted — they +are simply not consulted for this run. + +```text +/magpie-pr-management-triage --no-overrides +/magpie-security-issue-triage --no-overrides +``` + +When `--no-overrides` is present the skill's opening +protocol changes: **skip steps 1–3 entirely** — do not +read, surface, apply, or recap any override file. The skill +proceeds immediately with its framework defaults, as if +neither `.apache-magpie-overrides/.md` nor any +personal override existed. + +The **safety baseline** (confidentiality, privacy, and +security rules baked into the framework) still applies in +full — `--no-overrides` is not a safety bypass. It only +removes the adopter-customisation layer. + +Typical use cases: + +- **Debugging**: reproduce the framework's default + behaviour to determine whether an override is causing + an unexpected result. +- **One-off clean run**: a release manager wants a + pristine triage run without their personal overrides + for this single check-in. +- **Override authoring**: run with defaults first to see + what the framework produces, then compare against a + run that applies the override under construction. + ## Hard rules These are baked into agent instructions across the framework. @@ -297,7 +345,9 @@ this by: - [`setup` skill](../../skills/setup/SKILL.md) — the entry point that manages the snapshot, scaffolds overrides, and adds the - `.gitignore` entries for both override directories. + `.gitignore` entries for both override directories. Lists + `--no-overrides` in its Inputs table as a recognised + framework-level flag. - [`overrides.md` sub-action](../../skills/setup/overrides.md) — interactive override creation (lets the user choose between the personal-local and committed surfaces). diff --git a/skills/setup/SKILL.md b/skills/setup/SKILL.md index 619b865a..9ed7bd0a 100644 --- a/skills/setup/SKILL.md +++ b/skills/setup/SKILL.md @@ -435,6 +435,7 @@ first, then continue. | `agents:` | Comma-separated **agent targets** to wire symlinks into ([`agents.md`](agents.md) registry ids: `universal`, `claude-code`, `github`, `windsurf`, `goose`, …). Default on `adopt`/`upgrade`: auto — the always-on neutral set (`universal` + `claude-code` + `github`) plus any other registry dir already present in the repo. When passed, **replaces** the auto-detected set for that run, except `universal` (`.agents/skills/`) which is always retained because it is the canonical home every other target relays into — dropping it would leave the relays dangling. | | `skill-families:` | Comma-separated **opt-in** families to symlink — any of the opt-in families declared by a `family:` frontmatter key in the snapshot (today: `security`, `pr-management`, `issue`, `release-management`, `repo-health`, `pairing`, `mentoring`, `contributor-growth`). Default on `adopt`: prompt (see [`adopt.md` Step 5](adopt.md#step-5--pick-the-skill-families-and-mcp-servers)). Default on `upgrade`: read the families list from `` / ``, **auto-include any opt-in family the framework has introduced since the lock was written** (recorded back into the lock), and **ensure every framework skill in the effective family set has a valid symlink** — create or repair missing / broken symlinks, not just add new ones. The flag never accepts the always-on families (`setup`, `utilities`); per [Golden rule 8](#golden-rules) those are wired up unconditionally on every run and there is no way to ask for them or opt out. | | `--purge-overrides` | *(unadopt only)* Also `git rm -r` `.apache-magpie-overrides/`. Default: preserve. | +| `--no-overrides` | *(any framework skill)* Skip override-file lookup for this single invocation. Runs the skill against framework defaults; override files on disk are not read, modified, or deleted. The safety baseline (confidentiality, privacy, security) still applies. See [One-shot defaults run](../../docs/setup/agentic-overrides.md#one-shot-defaults-run). | | `dry-run` | Show what the skill would do without writing anything. | ## What this skill is NOT for diff --git a/tools/skill-evals/evals/setup/README.md b/tools/skill-evals/evals/setup/README.md index 6c76cf8f..e21f3476 100644 --- a/tools/skill-evals/evals/setup/README.md +++ b/tools/skill-evals/evals/setup/README.md @@ -5,27 +5,31 @@ Behavioral evals for the `setup` skill. -## Suites (9 cases total) +## Suites (12 cases total) | Suite | Step | Cases | What it covers | |---|---|---|---| | step-verify-drift | verify.md § Check 3 (drift) | 5 | clean, method/URL mismatch, ref mismatch, svn-zip SHA-512 mismatch, local lock missing | | step-overrides-surface | overrides.md § Step 0b | 4 | adopted no flag (offer choice), --local flag (personal), not adopted (personal only), both surfaces exist | +| step-override-bypass | agentic-overrides.md § One-shot defaults run | 3 | `--no-overrides` flag + override exists, `--no-overrides` + no override, no flag + override exists | ## Run +`--cli` is required or nothing is graded; use `--directory`, not +`--project`, and run from the repo root. + ```bash # All cases -uv run --project tools/skill-evals skill-eval \ - tools/skill-evals/evals/setup/ +uv run --directory tools/skill-evals skill-eval --cli "claude -p" \ + evals/setup/ # Single suite -uv run --project tools/skill-evals skill-eval \ - tools/skill-evals/evals/setup/step-verify-drift/fixtures/ +uv run --directory tools/skill-evals skill-eval --cli "claude -p" \ + evals/setup/step-override-bypass/ # Single case -uv run --project tools/skill-evals skill-eval \ - tools/skill-evals/evals/setup/step-verify-drift/fixtures/case-1-clean +uv run --directory tools/skill-evals skill-eval --cli "claude -p" \ + evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists ``` ## Notes @@ -36,3 +40,16 @@ uv run --project tools/skill-evals skill-eval \ vs-shared surface selection introduced by the `magpie-local-convention` work item. The default surface when the repo is adopted and no flag is passed is `"offer-choice"`; `override_path` reports the personal default. +- `step-override-bypass` cases are fully auto-comparable: `decision` and + `safety_baseline` are enumerated strings, and `reason` is checked by + deterministic `regex` predicates in `assertions.json` + (`has_bypass_reason` for the skip cases, `has_apply_reason` for the + apply case) — no grader or MANUAL step is required. + The two predicates discriminate on *direction*, not on the flag name. + Keying on `no-overrides` would be useless here: the flag name appears + in a correct reason and in a reason arguing the exact opposite, so such + a pattern passes either way. Each predicate therefore requires the + matching verb (skipped/not-consulted versus applied/consulted) and + rejects the opposing phrasing. When editing them, check both + directions — that a right answer still passes *and* that a reason + arguing the other decision fails. diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/assertions.json b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/assertions.json new file mode 100644 index 00000000..7fdfa36a --- /dev/null +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/assertions.json @@ -0,0 +1,14 @@ +{ + "has_bypass_reason": { + "type": "regex", + "field": "reason", + "pattern": "^(?=.*override)(?=.*\\b(?:skip|skips|skipped|skipping|bypass|bypassed|bypassing|ignore|ignores|ignored|ignoring|not\\s+(?:be\\s+)?(?:read|consulted|applied)|(?:were|was)\\s+not|without)\\b)(?!.*\\b(?:read\\s+and\\s+applied|applied\\s+the\\s+overrides?|overrides?\\s+(?:were|was)\\s+applied)\\b).*", + "flags": "is" + }, + "has_apply_reason": { + "type": "regex", + "field": "reason", + "pattern": "^(?=.*override)(?=.*\\b(?:applied|applying|consulted|honoured|honored)\\b)(?!.*\\b(?:skipped|bypassed|ignored)\\s+(?:the\\s+)?overrides?\\b)(?!.*\\boverrides?\\s+(?:were|was)\\s+not\\b)(?!.*\\bnot\\s+(?:be\\s+)?(?:read|consulted|applied)\\b).*", + "flags": "is" + } +} diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists/expected.json b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists/expected.json new file mode 100644 index 00000000..1f84f36c --- /dev/null +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists/expected.json @@ -0,0 +1 @@ +{"decision": "skip-overrides", "safety_baseline": "enforced", "has_bypass_reason": true} diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists/report.md b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists/report.md new file mode 100644 index 00000000..87746954 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists/report.md @@ -0,0 +1,10 @@ + + +Invocation: `/magpie-pr-management-triage --no-overrides` + +Override file present on disk: + `.apache-magpie-overrides/pr-management-triage.md` + Content: "Override 1 — Always tag @core-maintainers on first comment" + +Should the override file be read and applied for this invocation? diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-2-flag-no-override/expected.json b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-2-flag-no-override/expected.json new file mode 100644 index 00000000..1f84f36c --- /dev/null +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-2-flag-no-override/expected.json @@ -0,0 +1 @@ +{"decision": "skip-overrides", "safety_baseline": "enforced", "has_bypass_reason": true} diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-2-flag-no-override/report.md b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-2-flag-no-override/report.md new file mode 100644 index 00000000..2f6dd8cc --- /dev/null +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-2-flag-no-override/report.md @@ -0,0 +1,9 @@ + + +Invocation: `/magpie-security-issue-triage --no-overrides` + +Override file present on disk: none + `.apache-magpie-overrides/security-issue-triage.md` does not exist. + +Should the skill run with framework defaults for this invocation? diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-3-no-flag-override-exists/expected.json b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-3-no-flag-override-exists/expected.json new file mode 100644 index 00000000..445c2d54 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-3-no-flag-override-exists/expected.json @@ -0,0 +1 @@ +{"decision": "apply-overrides", "safety_baseline": "enforced", "has_apply_reason": true} diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-3-no-flag-override-exists/report.md b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-3-no-flag-override-exists/report.md new file mode 100644 index 00000000..b82e4cc9 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-3-no-flag-override-exists/report.md @@ -0,0 +1,12 @@ + + +Invocation: `/magpie-pr-management-triage` + +Override file present on disk: + `.apache-magpie-overrides/pr-management-triage.md` + Content: "Override 1 — Treat backport PRs as already-triaged" + +No `--no-overrides` flag was passed. + +Should the override file be read and applied for this invocation? diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/output-spec.md b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/output-spec.md new file mode 100644 index 00000000..1363e0f7 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/output-spec.md @@ -0,0 +1,25 @@ + + +## Output format + +Return ONLY valid JSON with this structure: + +```json +{ + "decision": "skip-overrides" | "apply-overrides", + "safety_baseline": "enforced", + "reason": "" +} +``` + +- `"skip-overrides"` — `--no-overrides` flag was present; override files + are not read or applied this invocation. `safety_baseline` must be + `"enforced"`. +- `"apply-overrides"` — `--no-overrides` flag was absent; override files + are read and applied per the normal protocol. `safety_baseline` must + be `"enforced"`. +- `"safety_baseline"` is always `"enforced"` — `--no-overrides` does not + disable the framework's confidentiality, privacy, or security rules. +- `"reason"` should be one concise sentence. +- Do not include any text outside the JSON object. diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/step-config.json b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/step-config.json new file mode 100644 index 00000000..ad6e0ca0 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/step-config.json @@ -0,0 +1,4 @@ +{ + "skill_md": "docs/setup/agentic-overrides.md", + "step_heading": "## One-shot defaults run" +} diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/user-prompt-template.md b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/user-prompt-template.md new file mode 100644 index 00000000..5802f703 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/user-prompt-template.md @@ -0,0 +1,8 @@ + + +## Override bypass decision + +{report} + +Apply the one-shot defaults run rules and return JSON only.