From 0de5a97261630c8cd024073939217ec4bcdb2f44 Mon Sep 17 00:00:00 2001 From: Justin McLean Date: Sun, 12 Jul 2026 17:19:45 +1000 Subject: [PATCH 1/3] feat(setup): add --no-overrides one-shot defaults switch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements acceptance criterion 6 of specs/adoption-and-setup.md: a per-invocation --no-overrides flag that runs any framework skill against framework defaults for that single invocation, ignoring both override surfaces without modifying or deleting them. The safety baseline (confidentiality, privacy, security) still applies. Adds the protocol to docs/setup/agentic-overrides.md under a new "One-shot defaults run" section, lists the flag in the setup SKILL.md Inputs table, and ships a three-case eval suite exercising: - flag present + override exists → skip-overrides - flag present + no override → skip-overrides - flag absent + override exists → apply-overrides (baseline) Note: this branch is based on main and is independent of the in-flight magpie-local-convention branch. When that branch merges (adding .apache-magpie-local/ as a second override surface), the "One-shot defaults run" prose here will need a minor update to say "skip steps 1–5" instead of "1–3" and to mention .apache-magpie-local/ alongside .apache-magpie-overrides/. The merge conflict is shallow and localised to the one paragraph. Generated-by: Claude (claude-sonnet-4-6) --- docs/setup/agentic-overrides.md | 42 ++++++++++++++++++- skills/setup/SKILL.md | 1 + tools/skill-evals/evals/setup/README.md | 10 +++-- .../case-1-flag-override-exists/expected.json | 1 + .../case-1-flag-override-exists/report.md | 10 +++++ .../case-2-flag-no-override/expected.json | 1 + .../case-2-flag-no-override/report.md | 9 ++++ .../expected.json | 1 + .../case-3-no-flag-override-exists/report.md | 12 ++++++ .../fixtures/output-spec.md | 25 +++++++++++ .../fixtures/step-config.json | 4 ++ .../fixtures/user-prompt-template.md | 8 ++++ 12 files changed, 120 insertions(+), 4 deletions(-) create mode 100644 tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists/expected.json create mode 100644 tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists/report.md create mode 100644 tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-2-flag-no-override/expected.json create mode 100644 tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-2-flag-no-override/report.md create mode 100644 tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-3-no-flag-override-exists/expected.json create mode 100644 tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-3-no-flag-override-exists/report.md create mode 100644 tools/skill-evals/evals/setup/step-override-bypass/fixtures/output-spec.md create mode 100644 tools/skill-evals/evals/setup/step-override-bypass/fixtures/step-config.json create mode 100644 tools/skill-evals/evals/setup/step-override-bypass/fixtures/user-prompt-template.md diff --git a/docs/setup/agentic-overrides.md b/docs/setup/agentic-overrides.md index ccbb51c8..b5a356bd 100644 --- a/docs/setup/agentic-overrides.md +++ b/docs/setup/agentic-overrides.md @@ -15,6 +15,7 @@ - [Pre-empt a decision-table row](#pre-empt-a-decision-table-row) - [What an override file should explain](#what-an-override-file-should-explain) - [How a framework skill consults overrides](#how-a-framework-skill-consults-overrides) + - [One-shot defaults run](#one-shot-defaults-run) - [Hard rules](#hard-rules) - [Reconciliation on framework upgrade](#reconciliation-on-framework-upgrade) - [Upstreaming an override](#upstreaming-an-override) @@ -187,6 +188,43 @@ that explicitly in its `SKILL.md`. The sub-action surfaces this gap and suggests opening a framework-side issue requesting the hook. +## One-shot defaults run + +Pass `--no-overrides` to any framework skill that supports +overrides to run that single invocation against framework +defaults, ignoring any override files that exist on disk. +The override files are **not** modified or deleted — they +are simply not consulted for this run. + +```text +/magpie-pr-management-triage --no-overrides +/magpie-security-issue-triage --no-overrides +``` + +When `--no-overrides` is present the skill's opening +protocol changes: **skip steps 1–3 entirely** (do not +read, surface, or apply any override file). The skill +proceeds immediately with its framework defaults, as if +neither `.apache-magpie-overrides/.md` nor any +personal override existed. + +The **safety baseline** (confidentiality, privacy, and +security rules baked into the framework) still applies in +full — `--no-overrides` is not a safety bypass. It only +removes the adopter-customisation layer. + +Typical use cases: + +- **Debugging**: reproduce the framework's default + behaviour to determine whether an override is causing + an unexpected result. +- **One-off clean run**: a release manager wants a + pristine triage run without their personal overrides + for this single check-in. +- **Override authoring**: run with defaults first to see + what the framework produces, then compare against a + run that applies the override under construction. + ## Hard rules These are baked into agent instructions across the framework. @@ -297,7 +335,9 @@ this by: - [`setup` skill](../../skills/setup/SKILL.md) — the entry point that manages the snapshot, scaffolds overrides, and adds the - `.gitignore` entries for both override directories. + `.gitignore` entries for both override directories. Lists + `--no-overrides` in its Inputs table as a recognised + framework-level flag. - [`overrides.md` sub-action](../../skills/setup/overrides.md) — interactive override creation (lets the user choose between the personal-local and committed surfaces). diff --git a/skills/setup/SKILL.md b/skills/setup/SKILL.md index 619b865a..9ed7bd0a 100644 --- a/skills/setup/SKILL.md +++ b/skills/setup/SKILL.md @@ -435,6 +435,7 @@ first, then continue. | `agents:` | Comma-separated **agent targets** to wire symlinks into ([`agents.md`](agents.md) registry ids: `universal`, `claude-code`, `github`, `windsurf`, `goose`, …). Default on `adopt`/`upgrade`: auto — the always-on neutral set (`universal` + `claude-code` + `github`) plus any other registry dir already present in the repo. When passed, **replaces** the auto-detected set for that run, except `universal` (`.agents/skills/`) which is always retained because it is the canonical home every other target relays into — dropping it would leave the relays dangling. | | `skill-families:` | Comma-separated **opt-in** families to symlink — any of the opt-in families declared by a `family:` frontmatter key in the snapshot (today: `security`, `pr-management`, `issue`, `release-management`, `repo-health`, `pairing`, `mentoring`, `contributor-growth`). Default on `adopt`: prompt (see [`adopt.md` Step 5](adopt.md#step-5--pick-the-skill-families-and-mcp-servers)). Default on `upgrade`: read the families list from `` / ``, **auto-include any opt-in family the framework has introduced since the lock was written** (recorded back into the lock), and **ensure every framework skill in the effective family set has a valid symlink** — create or repair missing / broken symlinks, not just add new ones. The flag never accepts the always-on families (`setup`, `utilities`); per [Golden rule 8](#golden-rules) those are wired up unconditionally on every run and there is no way to ask for them or opt out. | | `--purge-overrides` | *(unadopt only)* Also `git rm -r` `.apache-magpie-overrides/`. Default: preserve. | +| `--no-overrides` | *(any framework skill)* Skip override-file lookup for this single invocation. Runs the skill against framework defaults; override files on disk are not read, modified, or deleted. The safety baseline (confidentiality, privacy, security) still applies. See [One-shot defaults run](../../docs/setup/agentic-overrides.md#one-shot-defaults-run). | | `dry-run` | Show what the skill would do without writing anything. | ## What this skill is NOT for diff --git a/tools/skill-evals/evals/setup/README.md b/tools/skill-evals/evals/setup/README.md index 6c76cf8f..9600228a 100644 --- a/tools/skill-evals/evals/setup/README.md +++ b/tools/skill-evals/evals/setup/README.md @@ -5,12 +5,13 @@ Behavioral evals for the `setup` skill. -## Suites (9 cases total) +## Suites (12 cases total) | Suite | Step | Cases | What it covers | |---|---|---|---| | step-verify-drift | verify.md § Check 3 (drift) | 5 | clean, method/URL mismatch, ref mismatch, svn-zip SHA-512 mismatch, local lock missing | | step-overrides-surface | overrides.md § Step 0b | 4 | adopted no flag (offer choice), --local flag (personal), not adopted (personal only), both surfaces exist | +| step-override-bypass | agentic-overrides.md § One-shot defaults run | 3 | `--no-overrides` flag + override exists, `--no-overrides` + no override, no flag + override exists | ## Run @@ -21,11 +22,11 @@ uv run --project tools/skill-evals skill-eval \ # Single suite uv run --project tools/skill-evals skill-eval \ - tools/skill-evals/evals/setup/step-verify-drift/fixtures/ + tools/skill-evals/evals/setup/step-override-bypass/fixtures/ # Single case uv run --project tools/skill-evals skill-eval \ - tools/skill-evals/evals/setup/step-verify-drift/fixtures/case-1-clean + tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists ``` ## Notes @@ -36,3 +37,6 @@ uv run --project tools/skill-evals skill-eval \ vs-shared surface selection introduced by the `magpie-local-convention` work item. The default surface when the repo is adopted and no flag is passed is `"offer-choice"`; `override_path` reports the personal default. +- `step-override-bypass` cases are fully auto-comparable: `decision` and + `safety_baseline` are enumerated strings; `reason` is free-form prose + and is graded as MANUAL unless a `--cli` grader is configured. diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists/expected.json b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists/expected.json new file mode 100644 index 00000000..638aa6ed --- /dev/null +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists/expected.json @@ -0,0 +1 @@ +{"decision": "skip-overrides", "safety_baseline": "enforced", "reason": "--no-overrides flag was present so the override file is not read or applied for this invocation."} diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists/report.md b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists/report.md new file mode 100644 index 00000000..87746954 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists/report.md @@ -0,0 +1,10 @@ + + +Invocation: `/magpie-pr-management-triage --no-overrides` + +Override file present on disk: + `.apache-magpie-overrides/pr-management-triage.md` + Content: "Override 1 — Always tag @core-maintainers on first comment" + +Should the override file be read and applied for this invocation? diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-2-flag-no-override/expected.json b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-2-flag-no-override/expected.json new file mode 100644 index 00000000..8ad48602 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-2-flag-no-override/expected.json @@ -0,0 +1 @@ +{"decision": "skip-overrides", "safety_baseline": "enforced", "reason": "--no-overrides flag was present so the skill runs with framework defaults regardless of whether any override file exists."} diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-2-flag-no-override/report.md b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-2-flag-no-override/report.md new file mode 100644 index 00000000..2f6dd8cc --- /dev/null +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-2-flag-no-override/report.md @@ -0,0 +1,9 @@ + + +Invocation: `/magpie-security-issue-triage --no-overrides` + +Override file present on disk: none + `.apache-magpie-overrides/security-issue-triage.md` does not exist. + +Should the skill run with framework defaults for this invocation? diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-3-no-flag-override-exists/expected.json b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-3-no-flag-override-exists/expected.json new file mode 100644 index 00000000..b051c2d6 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-3-no-flag-override-exists/expected.json @@ -0,0 +1 @@ +{"decision": "apply-overrides", "safety_baseline": "enforced", "reason": "No --no-overrides flag was passed so the override file is read and applied per the normal protocol."} diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-3-no-flag-override-exists/report.md b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-3-no-flag-override-exists/report.md new file mode 100644 index 00000000..b82e4cc9 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-3-no-flag-override-exists/report.md @@ -0,0 +1,12 @@ + + +Invocation: `/magpie-pr-management-triage` + +Override file present on disk: + `.apache-magpie-overrides/pr-management-triage.md` + Content: "Override 1 — Treat backport PRs as already-triaged" + +No `--no-overrides` flag was passed. + +Should the override file be read and applied for this invocation? diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/output-spec.md b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/output-spec.md new file mode 100644 index 00000000..1363e0f7 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/output-spec.md @@ -0,0 +1,25 @@ + + +## Output format + +Return ONLY valid JSON with this structure: + +```json +{ + "decision": "skip-overrides" | "apply-overrides", + "safety_baseline": "enforced", + "reason": "" +} +``` + +- `"skip-overrides"` — `--no-overrides` flag was present; override files + are not read or applied this invocation. `safety_baseline` must be + `"enforced"`. +- `"apply-overrides"` — `--no-overrides` flag was absent; override files + are read and applied per the normal protocol. `safety_baseline` must + be `"enforced"`. +- `"safety_baseline"` is always `"enforced"` — `--no-overrides` does not + disable the framework's confidentiality, privacy, or security rules. +- `"reason"` should be one concise sentence. +- Do not include any text outside the JSON object. diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/step-config.json b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/step-config.json new file mode 100644 index 00000000..ad6e0ca0 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/step-config.json @@ -0,0 +1,4 @@ +{ + "skill_md": "docs/setup/agentic-overrides.md", + "step_heading": "## One-shot defaults run" +} diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/user-prompt-template.md b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/user-prompt-template.md new file mode 100644 index 00000000..5802f703 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/user-prompt-template.md @@ -0,0 +1,8 @@ + + +## Override bypass decision + +{report} + +Apply the one-shot defaults run rules and return JSON only. From 68e27c570459736f57de9da2fbcdd89dd9e676d0 Mon Sep 17 00:00:00 2001 From: Justin McLean Date: Sun, 12 Jul 2026 22:41:41 +1000 Subject: [PATCH 2/3] wire --no-overrides into the override protocol and auto-grade its eval --- docs/setup/agentic-overrides.md | 14 +++++++++++-- tools/skill-evals/evals/setup/README.md | 21 ++++++++++++------- .../fixtures/assertions.json | 14 +++++++++++++ .../case-1-flag-override-exists/expected.json | 2 +- .../case-2-flag-no-override/expected.json | 2 +- .../expected.json | 2 +- 6 files changed, 42 insertions(+), 13 deletions(-) create mode 100644 tools/skill-evals/evals/setup/step-override-bypass/fixtures/assertions.json diff --git a/docs/setup/agentic-overrides.md b/docs/setup/agentic-overrides.md index b5a356bd..c519d1a1 100644 --- a/docs/setup/agentic-overrides.md +++ b/docs/setup/agentic-overrides.md @@ -163,6 +163,16 @@ maintainer (or a future agent on a later run): Every framework skill that supports overrides starts each invocation with this opening protocol: +0. **Check for `--no-overrides`.** If the invocation passed + the `--no-overrides` flag, skip steps 1–4 entirely — both + override surfaces — and run against framework defaults for + this invocation; see + [One-shot defaults run](#one-shot-defaults-run). The safety + baseline still applies. Still do step 5, reporting that the + run used `--no-overrides` and naming the override files that + existed but were not consulted, so the audit trail records + the bypass rather than looking like a run with no overrides + on disk. Otherwise, continue with step 1. 1. Read `/.apache-magpie-local/.md` (personal, gitignored) if it exists. 2. Read `/.apache-magpie-overrides/.md` @@ -202,8 +212,8 @@ are simply not consulted for this run. ``` When `--no-overrides` is present the skill's opening -protocol changes: **skip steps 1–3 entirely** (do not -read, surface, or apply any override file). The skill +protocol changes: **skip steps 1–3 entirely** — do not +read, surface, apply, or recap any override file. The skill proceeds immediately with its framework defaults, as if neither `.apache-magpie-overrides/.md` nor any personal override existed. diff --git a/tools/skill-evals/evals/setup/README.md b/tools/skill-evals/evals/setup/README.md index 9600228a..848c70e9 100644 --- a/tools/skill-evals/evals/setup/README.md +++ b/tools/skill-evals/evals/setup/README.md @@ -15,18 +15,21 @@ Behavioral evals for the `setup` skill. ## Run +`--cli` is required or nothing is graded; use `--directory`, not +`--project`, and run from the repo root. + ```bash # All cases -uv run --project tools/skill-evals skill-eval \ - tools/skill-evals/evals/setup/ +uv run --directory tools/skill-evals skill-eval --cli "claude -p" \ + evals/setup/ # Single suite -uv run --project tools/skill-evals skill-eval \ - tools/skill-evals/evals/setup/step-override-bypass/fixtures/ +uv run --directory tools/skill-evals skill-eval --cli "claude -p" \ + evals/setup/step-override-bypass/ # Single case -uv run --project tools/skill-evals skill-eval \ - tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists +uv run --directory tools/skill-evals skill-eval --cli "claude -p" \ + evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists ``` ## Notes @@ -38,5 +41,7 @@ uv run --project tools/skill-evals skill-eval \ work item. The default surface when the repo is adopted and no flag is passed is `"offer-choice"`; `override_path` reports the personal default. - `step-override-bypass` cases are fully auto-comparable: `decision` and - `safety_baseline` are enumerated strings; `reason` is free-form prose - and is graded as MANUAL unless a `--cli` grader is configured. + `safety_baseline` are enumerated strings, and `reason` is checked by + deterministic `regex` predicates in `assertions.json` + (`has_bypass_reason` for the skip cases, `has_apply_reason` for the + apply case) — no grader or MANUAL step is required. diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/assertions.json b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/assertions.json new file mode 100644 index 00000000..a4da3ad2 --- /dev/null +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/assertions.json @@ -0,0 +1,14 @@ +{ + "has_bypass_reason": { + "type": "regex", + "field": "reason", + "pattern": "no-?overrides|framework default|not (be )?read|not applied|ignor", + "flags": "i" + }, + "has_apply_reason": { + "type": "regex", + "field": "reason", + "pattern": "read and applied|normal protocol|no .*flag|without .*no-?overrides", + "flags": "i" + } +} diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists/expected.json b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists/expected.json index 638aa6ed..1f84f36c 100644 --- a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists/expected.json +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-1-flag-override-exists/expected.json @@ -1 +1 @@ -{"decision": "skip-overrides", "safety_baseline": "enforced", "reason": "--no-overrides flag was present so the override file is not read or applied for this invocation."} +{"decision": "skip-overrides", "safety_baseline": "enforced", "has_bypass_reason": true} diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-2-flag-no-override/expected.json b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-2-flag-no-override/expected.json index 8ad48602..1f84f36c 100644 --- a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-2-flag-no-override/expected.json +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-2-flag-no-override/expected.json @@ -1 +1 @@ -{"decision": "skip-overrides", "safety_baseline": "enforced", "reason": "--no-overrides flag was present so the skill runs with framework defaults regardless of whether any override file exists."} +{"decision": "skip-overrides", "safety_baseline": "enforced", "has_bypass_reason": true} diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-3-no-flag-override-exists/expected.json b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-3-no-flag-override-exists/expected.json index b051c2d6..445c2d54 100644 --- a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-3-no-flag-override-exists/expected.json +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/case-3-no-flag-override-exists/expected.json @@ -1 +1 @@ -{"decision": "apply-overrides", "safety_baseline": "enforced", "reason": "No --no-overrides flag was passed so the override file is read and applied per the normal protocol."} +{"decision": "apply-overrides", "safety_baseline": "enforced", "has_apply_reason": true} From 25d4f87c99e952f9479ef52519197a9f1e974d1b Mon Sep 17 00:00:00 2001 From: Jarek Potiuk Date: Mon, 3 Aug 2026 01:56:24 +0200 Subject: [PATCH 3/3] fixup: make the reason predicates discriminate on direction Rebased onto main and resolved two conflicts: main added the step-overrides-surface suite and the two-directory (personal-local + committed) override protocol while this branch was open. Both suites now coexist (12 cases), and the --no-overrides check becomes step 0 of the five-step protocol rather than the three-step one it was written against. Step 0 now also keeps the step-5 recap, so a bypassed run records which override files existed but were not consulted -- otherwise the audit trail cannot distinguish a bypass from a repo with no overrides at all. The eval predicates needed tightening. Both keyed on the flag name, which appears in a correct reason and in a reason arguing the exact opposite: has_bypass_reason: no-?overrides|framework default|not (be )?read|... Given the reason "The override file was read and applied because no --no-overrides flag was present" -- an apply-reason on a skip case -- that pattern matches, so the assertion passed on a wrong answer. The same held for has_apply_reason via its 'no .*flag' branch. decision is an enumerated field and still caught a wrong call, so the eval was not blind, but the reason predicates gave the appearance of prose checking without the substance. Both now require the matching verb (skipped/not-consulted versus applied/consulted) and reject the opposing phrasing, verified against eight cases in both directions -- including 'No bypass flag was supplied, so the override file was read and applied', where the skip word is itself negated and an earlier draft wrongly rejected it. Full-word alternations rather than truncated stems, because typos(1) reads skipp? and ignor as misspellings. Generated-by: Claude Code (Opus 5) --- tools/skill-evals/evals/setup/README.md | 8 ++++++++ .../setup/step-override-bypass/fixtures/assertions.json | 8 ++++---- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/tools/skill-evals/evals/setup/README.md b/tools/skill-evals/evals/setup/README.md index 848c70e9..e21f3476 100644 --- a/tools/skill-evals/evals/setup/README.md +++ b/tools/skill-evals/evals/setup/README.md @@ -45,3 +45,11 @@ uv run --directory tools/skill-evals skill-eval --cli "claude -p" \ deterministic `regex` predicates in `assertions.json` (`has_bypass_reason` for the skip cases, `has_apply_reason` for the apply case) — no grader or MANUAL step is required. + The two predicates discriminate on *direction*, not on the flag name. + Keying on `no-overrides` would be useless here: the flag name appears + in a correct reason and in a reason arguing the exact opposite, so such + a pattern passes either way. Each predicate therefore requires the + matching verb (skipped/not-consulted versus applied/consulted) and + rejects the opposing phrasing. When editing them, check both + directions — that a right answer still passes *and* that a reason + arguing the other decision fails. diff --git a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/assertions.json b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/assertions.json index a4da3ad2..7fdfa36a 100644 --- a/tools/skill-evals/evals/setup/step-override-bypass/fixtures/assertions.json +++ b/tools/skill-evals/evals/setup/step-override-bypass/fixtures/assertions.json @@ -2,13 +2,13 @@ "has_bypass_reason": { "type": "regex", "field": "reason", - "pattern": "no-?overrides|framework default|not (be )?read|not applied|ignor", - "flags": "i" + "pattern": "^(?=.*override)(?=.*\\b(?:skip|skips|skipped|skipping|bypass|bypassed|bypassing|ignore|ignores|ignored|ignoring|not\\s+(?:be\\s+)?(?:read|consulted|applied)|(?:were|was)\\s+not|without)\\b)(?!.*\\b(?:read\\s+and\\s+applied|applied\\s+the\\s+overrides?|overrides?\\s+(?:were|was)\\s+applied)\\b).*", + "flags": "is" }, "has_apply_reason": { "type": "regex", "field": "reason", - "pattern": "read and applied|normal protocol|no .*flag|without .*no-?overrides", - "flags": "i" + "pattern": "^(?=.*override)(?=.*\\b(?:applied|applying|consulted|honoured|honored)\\b)(?!.*\\b(?:skipped|bypassed|ignored)\\s+(?:the\\s+)?overrides?\\b)(?!.*\\boverrides?\\s+(?:were|was)\\s+not\\b)(?!.*\\bnot\\s+(?:be\\s+)?(?:read|consulted|applied)\\b).*", + "flags": "is" } }