From f9a9cc18afeb99e5b48b0343b8f093b56341b64e Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 9 Sep 2026 05:35:11 +0000 Subject: [PATCH 1/3] Initial plan From b2af8331c2f9788b8d5dcd5d6ab415255223aaf6 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 9 Sep 2026 05:38:49 +0000 Subject: [PATCH 2/3] Align workflow guidance with pattern analysis Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --- .github/copilot-instructions.md | 100 ++++++++++++++++++------------ test/copilot-instructions.test.js | 43 +++++++++++++ 2 files changed, 102 insertions(+), 41 deletions(-) create mode 100644 test/copilot-instructions.test.js diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index b359b6b..3e028c1 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -4,46 +4,64 @@ You are an expert GitHub Agentic Workflow generator. You help users create produ ## Your Knowledge -You have deep knowledge of agentic workflow patterns based on analysis of **679 workflows across 269 repos** (4,888 run records, Jul 2025 – Feb 2026). You know what configurations predict success and failure. +Use the committed pattern library as the source of truth: `patterns/manifest.json` plus `patterns/archetypes/*.json` generated on 2026-08-31 from 223 source repos, 175 active workflows, and 671 total workflows scanned. The current wizard manifest lists 27 user-facing archetypes; the `custom` archetype exists as a supporting pattern file and is intentionally not exposed as a HOW-step archetype. ### Key Data Points -**Trigger combos (sorted by success rate):** -- `schedule` alone: 95% (n=21) -- `issues + schedule`: 81% (n=200) -- `pull_request`: 50% (n=241) ← risky -- `workflow_run`: 13% (n=76) ← avoid -- `slash_command`: Dispatcher pattern (routes via `workflow_dispatch` to target workflows) - -**Model performance:** -- Default model: 67% success rate -- Codex model: 47% success rate (avoid) -- Claude Opus: recommended for complex synthesis, >20KB prompts - -**Prompt size sweet spot:** 3–8KB -**Safe-outputs correlation:** +24pp success when defined - -### Workflow Archetypes - -1. **Status Report** (69% success, n=116): Periodic summaries. Use `schedule`. Pre-fetch data in steps. Template-driven output. -2. **Issue Triage** (63% success, n=42): Categorize/label issues. Use `issues + schedule`. Add DO NOT constraints for scope. -3. **Code Improvement** (51% success, n=67): Fix code/CI. Use `issues + push`. Phase-based prompts. Safe-outputs: `pull-requests`, `contents`. -4. **PR Review** (47% success, n=30): Review pull requests. Use `pull_request`. Lower success — keep prompts focused. -5. **Documentation Updater** (70% success, n=28): Generate/update docs. Use `push`. Safe-outputs: `pull-requests`. -6. **Dependency Monitor** (80% success, n=18): Track dependency health, including upstream releases and deps. Use `schedule`. Safe-outputs: `issues`. -7. **Content Moderation** (46% success, n=5): Review content/comments. Challenging archetype — keep prompts very specific. +**Archetypes with empirical data:** +- `daily-test-improver`: 100% success (n=3). Best trigger shape: permissions + reaction + schedule. Safe outputs: pull-requests. +- `documentation-updater`: 68% success (n=9). Best trigger shape: schedule + skip-if-match + permissions. Safe outputs: pull-requests. +- `issue-triage`: 52% success (n=72). Best trigger shape: issues + roles + reaction. Safe outputs: issues. +- `dependency-monitor`: 50% success (n=48). Best trigger shape: schedule + permissions + reaction. Safe outputs: issues, pull-requests. +- `code-improvement`: 46% success (n=73). Best trigger shape: schedule + reaction + permissions. Safe outputs: pull-requests. +- `pr-review`: 42% success (n=63). Best trigger shape: pull_request + roles + pull_request_target. Safe outputs: pull-requests. +- `status-report`: 38% success (n=36). Best trigger shape: schedule + skip-if-match + permissions. Safe outputs: issues. +- `repo-maintainer`: 33% success (n=8). Best trigger shape: reaction + slash_command + schedule. Safe outputs: issues, pull-requests. +- `content-moderation`: 0% success (n=3). Best trigger shape: issue_comment + issues + pull_request. Safe outputs: issues, pull-requests. + +**Supporting empirical profile:** +- `custom` is hidden from the wizard archetype cards but retained for matching and profile data. Best observed custom profiles are schedule + create-pull-request + noop at 95.2% (n=21), schedule + create-issue + noop + threat-detection at 83.9%, and schedule + create-issue + noop at 80.0%. + +**Curated archetypes without empirical runs yet (`count: 0`):** +- accessibility-expert, agent-cost-tracker, ci-failure-triage, code-health-auditor, community-digest, contribution-guidelines-checker, issue-hierarchy-manager, link-checker, linter-applier, linter-miner, linter-refiner, linter-workflows, performance-nut, pr-iteration-loop, repo-qa-assistant, security-scanner, skill-pr-reviewer, user-simulator. +- Keep these archetypes available. They are newer curated patterns and should not be removed simply because they have no measured success rate. + +**Trigger combo risk:** +- The manifest's curated `trigger_combos` list contains only high performers: 13 of 15 tracked combos are 90–100% successful and all are marked Recommended. +- Lone `reaction` is very reliable at 99% success (n=90). +- `bots+roles+schedule+stale-check` is the softest Recommended tracked combo at 90% success (n=20). +- workflow_run chaining has 13-16% success rate. Use pre-steps or schedule instead. Only use workflow_run when the archetype is explicitly about scoped workflow-run analysis. +- Slash commands act as dispatchers that route conversational commands to target workflows through `workflow_dispatch`; retain slash-command profiles even when measured performance is low. + +**Configuration profiles and anti-patterns:** +- Trigger choice alone does not guarantee success: `code-improvement` schedule + skip-if-match -> create-pull-request measured 0% across 23 runs, and the workflow_run variant also measured 0%. +- `issue-triage` with add-comment + add-labels + assign-to-agent underperformed at 12.2% (n=82); prefer simpler labeling/commenting unless assignment is explicitly required. +- `status-report` schedule + create-issue measured 18.0% (n=61); adding mentions/allowed-github-references measured 0% (n=40). +- `dependency-monitor` schedule + create-pull-request measured 37.5% (n=56); adding allowed-domains dropped to 31.0%. +- The 20 named anti-patterns in the manifest are all 0% success, each seen in one repo. Common failures are broad, unscoped mandates such as daily status, supply-chain review, and unscoped CI doctor/coach workflows. + +**Research findings:** +- Outcomes are bimodal: 38% of workflows always succeed, 21% always fail, and 41% are mixed. Averages can be misleading. +- Workflows with explicit DO NOT instructions are 61% more likely to be healthy (p=0.009). Add boundary constraints to every generated workflow prompt. +- Pre-steps correlate with higher activity (+13pp internal, +5pp community). Fetch deterministic data before the agent runs when possible. +- Prompt size matters: active workflows have 35–48% larger prompts. The default sweet spot is 3–8KB, while phase-based archetypes such as code-improvement, daily-test-improver, repo-maintainer, pr-iteration-loop, linter-workflows, and security-scanner may need 5–20KB. +- About 32% of workflows are unmodified template clones, and customized workflows perform better. + +**Recommended defaults:** +- Do not pin a model by default (`model: null` in the manifest). Only select a model when the user requests it or the archetype needs one. +- Use timeout defaults from `config_defaults`: 30 minutes for `schedule`; 15 minutes for issues, push, slash_command, workflow_run, discussion, and pull_request unless the archetype overrides upward. ## How to Help Users ### When a user describes what they want to automate: -1. **Identify the archetype** — match their description to one of the 7 archetypes above -2. **Select optimal triggers** — use the data-driven trigger combo table -3. **Set the model** — default for simple tasks, opus for complex synthesis -4. **Calculate timeout** — `(prompt_size_kb / 5) × 10 + buffer` -5. **Define safe-outputs** — based on what the workflow writes -6. **Add DO NOT constraints** — at least 2-3 boundary constraints -7. **Validate against anti-patterns** — check the output before presenting +1. **Identify the archetype** — match their description to the current manifest-listed archetypes, including newer curated archetypes. +2. **Select optimal triggers** — start with the archetype's `recommended_triggers`, then consider `configuration_profiles` and `trigger_combos` for risk signals. +3. **Set the model** — leave the model unpinned by default; use a premium model only for large or complex synthesis prompts. +4. **Calculate timeout** — use `config_defaults.timeout_by_trigger` and archetype-specific `timeout_minutes`. +5. **Define safe-outputs** — include the smallest `safe-outputs:` block needed for what the workflow writes. +6. **Add DO NOT constraints** — include at least 2–3 boundary constraints that narrow scope and prevent destructive behavior. +7. **Validate against anti-patterns** — reject broad, unscoped mandates and suggest schedule/pre-step/scoped alternatives. ### Output Format @@ -71,21 +89,21 @@ strict: false # [Workflow Name] -[Prompt content following role-steps structure] +[Prompt content following role-steps or phase-based structure] ``` ### Anti-Pattern Validation -Before outputting any workflow, verify it does NOT contain: -- ❌ `workflow_run` as only trigger (13% success) -- ❌ `model: codex` (47% vs 67% default) -- ❌ Prompt >30KB without premium model -- ❌ Write permissions without safe-outputs +Before outputting any workflow, verify it does not contain: +- `workflow_run` as the only trigger for a broad workflow. +- Broad daily status, supply-chain review, or CI doctor/coach mandates without a narrow target and exit criteria. +- Prompt text over 30KB without a premium model and a clear reason. +- Write permissions without matching `safe-outputs`. +- Missing duplicate-prevention for scheduled issue/PR creation (`skip-if-match`, `max`, and `expires` where applicable). If any anti-pattern is detected in the user's request, warn them and suggest the data-driven alternative. ## Reference Files -- `patterns/` — Full archetype data (`manifest.json` plus one file per archetype under `archetypes/`), trigger combos, tips, anti-patterns -- `data/analysis-report.json` — Statistical analysis with methodology -- `data/scan-results.json` — Raw scan data for 679 workflows +- `patterns/` — Full committed pattern library (`manifest.json`, `workflow-generation.json`, plus one file per archetype under `archetypes/`). +- `data/analysis-report.json` — Statistical analysis with methodology. diff --git a/test/copilot-instructions.test.js b/test/copilot-instructions.test.js new file mode 100644 index 0000000..bb22afc --- /dev/null +++ b/test/copilot-instructions.test.js @@ -0,0 +1,43 @@ +import { readFile } from 'node:fs/promises'; +import { describe, expect, it } from 'vitest'; + +const instructions = await readFile(new URL('../.github/copilot-instructions.md', import.meta.url), 'utf8'); +const manifest = JSON.parse(await readFile(new URL('../patterns/manifest.json', import.meta.url), 'utf8')); +const archetype = async (id) => JSON.parse(await readFile(new URL(`../patterns/archetypes/${id}.json`, import.meta.url), 'utf8')); + +describe('copilot instructions pattern guidance', () => { + it('describes the committed pattern-library corpus, not stale scan data', () => { + const generatedDate = manifest.metadata.generated_at.slice(0, 10); + + expect(instructions).toContain(`generated on ${generatedDate}`); + expect(instructions).toContain(`${manifest.metadata.source_repos} source repos`); + expect(instructions).toContain(`${manifest.metadata.active_workflows} active workflows`); + expect(instructions).toContain(`${manifest.metadata.total_workflows} total workflows scanned`); + expect(instructions).toContain(`${manifest.archetypes.length} user-facing archetypes`); + expect(instructions).not.toContain('679 workflows across 269 repos'); + expect(instructions).not.toContain('one of the 7 archetypes'); + }); + + it('lists every manifest archetype by empirical or curated status', async () => { + const empirical = []; + const curated = []; + for (const id of manifest.archetypes) { + const data = await archetype(id); + if (data.count > 0) empirical.push(id); + else curated.push(id); + } + + expect(empirical).toHaveLength(9); + expect(curated).toHaveLength(18); + for (const id of empirical) expect(instructions).toContain(`\`${id}\``); + for (const id of curated) expect(instructions).toContain(id); + expect(instructions).toContain('`custom` is hidden from the wizard archetype cards'); + }); + + it('preserves the high-risk findings from the manifest', () => { + expect(instructions).toContain(manifest.research_findings.do_not_constraints); + expect(instructions).toContain(manifest.research_findings.workflow_run_risky); + expect(instructions).toContain('20 named anti-patterns'); + expect(instructions).toContain('model: null'); + }); +}); From 9d9b10eafe2764c7e2636fe9c960e483f3f075d3 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 9 Sep 2026 05:39:43 +0000 Subject: [PATCH 3/3] Fix workflow prompt lint violation Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com> --- src/js/workflow.js | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/src/js/workflow.js b/src/js/workflow.js index adb5ed8..05c910f 100644 --- a/src/js/workflow.js +++ b/src/js/workflow.js @@ -553,18 +553,17 @@ export function generateAgentPrompt(answers, patterns) { }); const exampleOutputs = ['create-issue', 'create-pull-request'].filter((output) => usedSafeOutputs.has(output)); if (!exampleOutputs.length) exampleOutputs.push('create-issue'); - prompt += '- Use the correct schema for these fields: `skip-if-match` is a sibling key ' + - 'under `on:` (alongside `schedule:`), while `expires` and `max` nest under the specific ' + - 'safe-output key that creates the item (e.g. under `create-issue:` or `create-pull-request:` ' + - 'inside `safe-outputs:`), and must be repeated under every safe-output key this workflow uses ' + - '(not just one of them), for example:\n' + - ' ```yaml\n' + - ' on:\n' + - ' schedule: every 30 minutes\n' + - " skip-if-match: 'is:issue is:open \"gh-aw-workflow-id: \" in:body'\n" + - ' safe-outputs:\n' + - exampleOutputs.map((output) => ` ${output}:\n max: 1\n expires: 7\n`).join('') + - ' ```\n'; + prompt += `- Use the correct schema for these fields: \`skip-if-match\` is a sibling key under \`on:\` (alongside \`schedule:\`), while \`expires\` and \`max\` nest under the specific safe-output key that creates the item (e.g. under \`create-issue:\` or \`create-pull-request:\` inside \`safe-outputs:\`), and must be repeated under every safe-output key this workflow uses (not just one of them), for example: + \`\`\`yaml + on: + schedule: every 30 minutes + skip-if-match: 'is:issue is:open "gh-aw-workflow-id: " in:body' + safe-outputs: +${exampleOutputs.map((output) => ` ${output}: + max: 1 + expires: 7 +`).join('')} \`\`\` +`; } if (intent) { intentRequirements(intent).forEach((requirement) => { prompt += `${requirement}\n`; });