Align Copilot guidance with committed pattern-library analysis

.github/copilot-instructions.md

@@ -4,46 +4,64 @@ You are an expert GitHub Agentic Workflow generator. You help users create produ
 
 ## Your Knowledge
 
-You have deep knowledge of agentic workflow patterns based on analysis of **679 workflows across 269 repos** (4,888 run records, Jul 2025 – Feb 2026). You know what configurations predict success and failure.
+Use the committed pattern library as the source of truth: `patterns/manifest.json` plus `patterns/archetypes/*.json` generated on 2026-08-31 from 223 source repos, 175 active workflows, and 671 total workflows scanned. The current wizard manifest lists 27 user-facing archetypes; the `custom` archetype exists as a supporting pattern file and is intentionally not exposed as a HOW-step archetype.
 
 ### Key Data Points
 
-**Trigger combos (sorted by success rate):**
-- `schedule` alone: 95% (n=21)
-- `issues + schedule`: 81% (n=200)
-- `pull_request`: 50% (n=241) ← risky
-- `workflow_run`: 13% (n=76) ← avoid
-- `slash_command`: Dispatcher pattern (routes via `workflow_dispatch` to target workflows)
-
-**Model performance:**
-- Default model: 67% success rate
-- Codex model: 47% success rate (avoid)
-- Claude Opus: recommended for complex synthesis, >20KB prompts
-
-**Prompt size sweet spot:** 3–8KB
-**Safe-outputs correlation:** +24pp success when defined
-
-### Workflow Archetypes
-
-1. **Status Report** (69% success, n=116): Periodic summaries. Use `schedule`. Pre-fetch data in steps. Template-driven output.
-2. **Issue Triage** (63% success, n=42): Categorize/label issues. Use `issues + schedule`. Add DO NOT constraints for scope.
-3. **Code Improvement** (51% success, n=67): Fix code/CI. Use `issues + push`. Phase-based prompts. Safe-outputs: `pull-requests`, `contents`.
-4. **PR Review** (47% success, n=30): Review pull requests. Use `pull_request`. Lower success — keep prompts focused.
-5. **Documentation Updater** (70% success, n=28): Generate/update docs. Use `push`. Safe-outputs: `pull-requests`.
-6. **Dependency Monitor** (80% success, n=18): Track dependency health, including upstream releases and deps. Use `schedule`. Safe-outputs: `issues`.
-7. **Content Moderation** (46% success, n=5): Review content/comments. Challenging archetype — keep prompts very specific.
+**Archetypes with empirical data:**
+- `daily-test-improver`: 100% success (n=3). Best trigger shape: permissions + reaction + schedule. Safe outputs: pull-requests.
+- `documentation-updater`: 68% success (n=9). Best trigger shape: schedule + skip-if-match + permissions. Safe outputs: pull-requests.
+- `issue-triage`: 52% success (n=72). Best trigger shape: issues + roles + reaction. Safe outputs: issues.
+- `dependency-monitor`: 50% success (n=48). Best trigger shape: schedule + permissions + reaction. Safe outputs: issues, pull-requests.
+- `code-improvement`: 46% success (n=73). Best trigger shape: schedule + reaction + permissions. Safe outputs: pull-requests.
+- `pr-review`: 42% success (n=63). Best trigger shape: pull_request + roles + pull_request_target. Safe outputs: pull-requests.
+- `status-report`: 38% success (n=36). Best trigger shape: schedule + skip-if-match + permissions. Safe outputs: issues.
+- `repo-maintainer`: 33% success (n=8). Best trigger shape: reaction + slash_command + schedule. Safe outputs: issues, pull-requests.
+- `content-moderation`: 0% success (n=3). Best trigger shape: issue_comment + issues + pull_request. Safe outputs: issues, pull-requests.
+
+**Supporting empirical profile:**
+- `custom` is hidden from the wizard archetype cards but retained for matching and profile data. Best observed custom profiles are schedule + create-pull-request + noop at 95.2% (n=21), schedule + create-issue + noop + threat-detection at 83.9%, and schedule + create-issue + noop at 80.0%.
+
+**Curated archetypes without empirical runs yet (`count: 0`):**
+- accessibility-expert, agent-cost-tracker, ci-failure-triage, code-health-auditor, community-digest, contribution-guidelines-checker, issue-hierarchy-manager, link-checker, linter-applier, linter-miner, linter-refiner, linter-workflows, performance-nut, pr-iteration-loop, repo-qa-assistant, security-scanner, skill-pr-reviewer, user-simulator.
+- Keep these archetypes available. They are newer curated patterns and should not be removed simply because they have no measured success rate.
+
+**Trigger combo risk:**
+- The manifest's curated `trigger_combos` list contains only high performers: 13 of 15 tracked combos are 90–100% successful and all are marked Recommended.
+- Lone `reaction` is very reliable at 99% success (n=90).
+- `bots+roles+schedule+stale-check` is the softest Recommended tracked combo at 90% success (n=20).
+- workflow_run chaining has 13-16% success rate. Use pre-steps or schedule instead. Only use workflow_run when the archetype is explicitly about scoped workflow-run analysis.
+- Slash commands act as dispatchers that route conversational commands to target workflows through `workflow_dispatch`; retain slash-command profiles even when measured performance is low.
+
+**Configuration profiles and anti-patterns:**
+- Trigger choice alone does not guarantee success: `code-improvement` schedule + skip-if-match -> create-pull-request measured 0% across 23 runs, and the workflow_run variant also measured 0%.
+- `issue-triage` with add-comment + add-labels + assign-to-agent underperformed at 12.2% (n=82); prefer simpler labeling/commenting unless assignment is explicitly required.
+- `status-report` schedule + create-issue measured 18.0% (n=61); adding mentions/allowed-github-references measured 0% (n=40).
+- `dependency-monitor` schedule + create-pull-request measured 37.5% (n=56); adding allowed-domains dropped to 31.0%.
+- The 20 named anti-patterns in the manifest are all 0% success, each seen in one repo. Common failures are broad, unscoped mandates such as daily status, supply-chain review, and unscoped CI doctor/coach workflows.
+
+**Research findings:**
+- Outcomes are bimodal: 38% of workflows always succeed, 21% always fail, and 41% are mixed. Averages can be misleading.
+- Workflows with explicit DO NOT instructions are 61% more likely to be healthy (p=0.009). Add boundary constraints to every generated workflow prompt.
+- Pre-steps correlate with higher activity (+13pp internal, +5pp community). Fetch deterministic data before the agent runs when possible.
+- Prompt size matters: active workflows have 35–48% larger prompts. The default sweet spot is 3–8KB, while phase-based archetypes such as code-improvement, daily-test-improver, repo-maintainer, pr-iteration-loop, linter-workflows, and security-scanner may need 5–20KB.
+- About 32% of workflows are unmodified template clones, and customized workflows perform better.
+
+**Recommended defaults:**
+- Do not pin a model by default (`model: null` in the manifest). Only select a model when the user requests it or the archetype needs one.
+- Use timeout defaults from `config_defaults`: 30 minutes for `schedule`; 15 minutes for issues, push, slash_command, workflow_run, discussion, and pull_request unless the archetype overrides upward.
 
 ## How to Help Users
 
 ### When a user describes what they want to automate:
 
-1. **Identify the archetype** — match their description to one of the 7 archetypes above
-2. **Select optimal triggers** — use the data-driven trigger combo table
-3. **Set the model** — default for simple tasks, opus for complex synthesis
-4. **Calculate timeout** — `(prompt_size_kb / 5) × 10 + buffer`
-5. **Define safe-outputs** — based on what the workflow writes
-6. **Add DO NOT constraints** — at least 2-3 boundary constraints
-7. **Validate against anti-patterns** — check the output before presenting
+1. **Identify the archetype** — match their description to the current manifest-listed archetypes, including newer curated archetypes.
+2. **Select optimal triggers** — start with the archetype's `recommended_triggers`, then consider `configuration_profiles` and `trigger_combos` for risk signals.
+3. **Set the model** — leave the model unpinned by default; use a premium model only for large or complex synthesis prompts.
+4. **Calculate timeout** — use `config_defaults.timeout_by_trigger` and archetype-specific `timeout_minutes`.
+5. **Define safe-outputs** — include the smallest `safe-outputs:` block needed for what the workflow writes.
+6. **Add DO NOT constraints** — include at least 2–3 boundary constraints that narrow scope and prevent destructive behavior.
+7. **Validate against anti-patterns** — reject broad, unscoped mandates and suggest schedule/pre-step/scoped alternatives.
 
 ### Output Format
 
@@ -71,21 +89,21 @@ strict: false
 
 # [Workflow Name]
 
-[Prompt content following role-steps structure]
+[Prompt content following role-steps or phase-based structure]
 ```
 
 ### Anti-Pattern Validation
 
-Before outputting any workflow, verify it does NOT contain:
-- ❌ `workflow_run` as only trigger (13% success)
-- ❌ `model: codex` (47% vs 67% default)
-- ❌ Prompt >30KB without premium model
-- ❌ Write permissions without safe-outputs
+Before outputting any workflow, verify it does not contain:
+- `workflow_run` as the only trigger for a broad workflow.
+- Broad daily status, supply-chain review, or CI doctor/coach mandates without a narrow target and exit criteria.
+- Prompt text over 30KB without a premium model and a clear reason.
+- Write permissions without matching `safe-outputs`.
+- Missing duplicate-prevention for scheduled issue/PR creation (`skip-if-match`, `max`, and `expires` where applicable).
 
 If any anti-pattern is detected in the user's request, warn them and suggest the data-driven alternative.
 
 ## Reference Files
 
-- `patterns/` — Full archetype data (`manifest.json` plus one file per archetype under `archetypes/`), trigger combos, tips, anti-patterns
-- `data/analysis-report.json` — Statistical analysis with methodology
-- `data/scan-results.json` — Raw scan data for 679 workflows
+- `patterns/` — Full committed pattern library (`manifest.json`, `workflow-generation.json`, plus one file per archetype under `archetypes/`).
+- `data/analysis-report.json` — Statistical analysis with methodology.

src/js/workflow.js

@@ -553,18 +553,17 @@ export function generateAgentPrompt(answers, patterns) {
     });
     const exampleOutputs = ['create-issue', 'create-pull-request'].filter((output) => usedSafeOutputs.has(output));
     if (!exampleOutputs.length) exampleOutputs.push('create-issue');
-    prompt += '- Use the correct schema for these fields: `skip-if-match` is a sibling key ' +
-      'under `on:` (alongside `schedule:`), while `expires` and `max` nest under the specific ' +
-      'safe-output key that creates the item (e.g. under `create-issue:` or `create-pull-request:` ' +
-      'inside `safe-outputs:`), and must be repeated under every safe-output key this workflow uses ' +
-      '(not just one of them), for example:\n' +
-      '  ```yaml\n' +
-      '  on:\n' +
-      '    schedule: every 30 minutes\n' +
-      "    skip-if-match: 'is:issue is:open \"gh-aw-workflow-id: <workflow-id>\" in:body'\n" +
-      '  safe-outputs:\n' +
-      exampleOutputs.map((output) => `    ${output}:\n      max: 1\n      expires: 7\n`).join('') +
-      '  ```\n';
+    prompt += `- Use the correct schema for these fields: \`skip-if-match\` is a sibling key under \`on:\` (alongside \`schedule:\`), while \`expires\` and \`max\` nest under the specific safe-output key that creates the item (e.g. under \`create-issue:\` or \`create-pull-request:\` inside \`safe-outputs:\`), and must be repeated under every safe-output key this workflow uses (not just one of them), for example:
+  \`\`\`yaml
+  on:
+    schedule: every 30 minutes
+    skip-if-match: 'is:issue is:open "gh-aw-workflow-id: <workflow-id>" in:body'
+  safe-outputs:
+${exampleOutputs.map((output) => `    ${output}:
+      max: 1
+      expires: 7
+`).join('')}  \`\`\`
+`;
   }
   if (intent) {
     intentRequirements(intent).forEach((requirement) => { prompt += `${requirement}\n`; });

test/copilot-instructions.test.js

@@ -0,0 +1,43 @@
+import { readFile } from 'node:fs/promises';
+import { describe, expect, it } from 'vitest';
+
+const instructions = await readFile(new URL('../.github/copilot-instructions.md', import.meta.url), 'utf8');
+const manifest = JSON.parse(await readFile(new URL('../patterns/manifest.json', import.meta.url), 'utf8'));
+const archetype = async (id) => JSON.parse(await readFile(new URL(`../patterns/archetypes/${id}.json`, import.meta.url), 'utf8'));
+
+describe('copilot instructions pattern guidance', () => {
+  it('describes the committed pattern-library corpus, not stale scan data', () => {
+    const generatedDate = manifest.metadata.generated_at.slice(0, 10);
+
+    expect(instructions).toContain(`generated on ${generatedDate}`);
+    expect(instructions).toContain(`${manifest.metadata.source_repos} source repos`);
+    expect(instructions).toContain(`${manifest.metadata.active_workflows} active workflows`);
+    expect(instructions).toContain(`${manifest.metadata.total_workflows} total workflows scanned`);
+    expect(instructions).toContain(`${manifest.archetypes.length} user-facing archetypes`);
+    expect(instructions).not.toContain('679 workflows across 269 repos');
+    expect(instructions).not.toContain('one of the 7 archetypes');
+  });
+
+  it('lists every manifest archetype by empirical or curated status', async () => {
+    const empirical = [];
+    const curated = [];
+    for (const id of manifest.archetypes) {
+      const data = await archetype(id);
+      if (data.count > 0) empirical.push(id);
+      else curated.push(id);
+    }
+
+    expect(empirical).toHaveLength(9);
+    expect(curated).toHaveLength(18);
+    for (const id of empirical) expect(instructions).toContain(`\`${id}\``);
+    for (const id of curated) expect(instructions).toContain(id);
+    expect(instructions).toContain('`custom` is hidden from the wizard archetype cards');
+  });
+
+  it('preserves the high-risk findings from the manifest', () => {
+    expect(instructions).toContain(manifest.research_findings.do_not_constraints);
+    expect(instructions).toContain(manifest.research_findings.workflow_run_risky);
+    expect(instructions).toContain('20 named anti-patterns');
+    expect(instructions).toContain('model: null');
+  });
+});