diff --git a/.github/.agents/skills/optimize-workflow/SKILL.md b/.github/.agents/skills/optimize-workflow/SKILL.md new file mode 100644 index 00000000000..1d26f090859 --- /dev/null +++ b/.github/.agents/skills/optimize-workflow/SKILL.md @@ -0,0 +1,86 @@ +--- +name: optimize-workflow +description: Optimize CI runners and workflow structure for speed, simplicity, stability, and cost. +disable-model-invocation: true +--- + + +Prompt user if missing: +1. Workflow to optimize. +2. Priorities (default: stability > speed > cost). +3. "Speed" definition (e.g., cache hits vs raw execution). +4. "Stability" definition (e.g., are flaky tests acceptable?). +5. Spot setting (infer via scenarios, do not ask directly). [runs-on spot reference](https://runs-on.com/docs/costs/spot-pricing/) +6. Acceptable structural changes (e.g., job splitting, caching, lint reorganization, trigger optimizations). + +Setup: +1. Analyze workflow: identify jobs, runners, step dependencies, critical path, caching, and bottlenecks. +2. Identify structural optimizations: + - Parallelize sequential steps/jobs (e.g., matrix builds). + - Add/fix caching (e.g., go modules, cargo, npm, docker layers). + - Move checks/lints before compilation/tests to fail early. + - Refactor triggers/scope. +3. Permitted scope: High freedom. OK to introduce breaking edits to internal `smartcontractkit` actions/workflows to improve speed/simplicity. +4. Ask user to target: specific job, whole workflow runner config, or workflow structure. +5. Setup test workflow (bypass gates, mock inputs, add `workflow_dispatch`). +6. Init/resume trial log: `.github/.agents/skills/optimize-workflow/trials/.md`. +7. Run baseline trial with current configuration to benchmark. + + + +- No OOM or Out of Disk Space failures allowed. +- Prefer default `ubuntu-latest`. Use `runs-on` only for larger runner requirements. + - Note: `runs-on` with `tmpfs` enabled can cause silent OOMs (not visible in memory metrics) under high disk load. Toggle `tmpfs` off if unexplained OOMs occur, and evaluate speed/cost impact. +- Verify runner configuration via available runners API before use. +- Validate workflow YAML syntax (e.g., `actionlint` or dry-run) before committing. +- Maintain semantic correctness. OK to break internal `smartcontractkit` action/workflow interfaces for speed/simplicity. +- Use `gh` CLI for runs and PRs. +- Change only one variable per trial. +- Ensure fair comparison: cache status must be identical across trials compared for speed. +- Document runner config and structure layout in each trial to ensure reproducibility. + + + +- [runs-on docs](https://runs-on.com/docs/) +- [available runners](https://go.runs-on.com/api) +- [GitHub Action docs](https://docs.github.com/en/actions) + + + +1. Define trials. Update `.md` log. +2. User approves trials and execution method (parallel vs sequential). +3. Push to new disposable branch. + - Commit message: `cpu=X/ram=Y` or `struct/[caching|parallel|ordering|simplify]` + - PR title: `[DO NOT MERGE] Trial: ` + - PR body: Details of configuration. Include before-after Mermaid diagram showing structural flow. +4. Trigger workflow. +5. Monitor run: + `python3 .github/.agents/skills/optimize-workflow/scripts/workflow_monitor.py [run_id] --format json --out-file .github/.agents/skills/optimize-workflow/trials/[trial-name].json` +6. Compare trials: + `python3 .github/.agents/skills/optimize-workflow/scripts/workflow_compare.py .github/.agents/skills/optimize-workflow/trials/[trial-1].json .github/.agents/skills/optimize-workflow/trials/[trial-2].json --out-file .github/.agents/skills/optimize-workflow/trials/[trial-1]-[trial-2]-comparison.md` +7. Update trial log. Include updated before-after diagram if structural changes occurred. +8. Show condensed results. Prompt for more trials or stop. + + +| Runner | Structure | Experiment | Expectation | Branch | Run ID | Commit | Stability | Runtime | Cost | Notes | +|---|---|---|---|---|---|---|---|---|---|---| +| `config` | `default / cached / parallel` | What's tested | Expected result | `branch` | `run-id` | `sha` | Pass/Fail/Flaky | mm:ss | $ | Findings | + + + + +1. Recommend final configuration. +2. Add PR summary table and before-after Mermaid diagrams: + +```md +### [Workflow/Job Name] Runner & Structure Changes + +| Approach | Runner | Structure | Stability | Runtime | Runtime Delta (Abs/%) | Cost | Cost Delta (Abs/%) | +|---|---|---|---|---|---|---|---| +| [Old](https://github.com/link/to/baseline/workflow_run) | [original runner] | [original structure] | Pass/Fail/Flaky | mm:ss | +0:00 (+0%) | $ | +$ (+0%) | +| [New](https://github.com/link/to/final/workflow_run) | [new runner] | [new structure] | Pass/Fail/Flaky | mm:ss | +0:00 (+0%) | $ | +$ (+0%) | +``` + +3. Clean debug lines. Make final approved edits on a new branch. Ask user to commit. +4. Delete trial branches, logs, and PRs. + \ No newline at end of file diff --git a/.github/.agents/skills/right-size-runners/scripts/test_workflow_compare.py b/.github/.agents/skills/optimize-workflow/scripts/test_workflow_compare.py similarity index 100% rename from .github/.agents/skills/right-size-runners/scripts/test_workflow_compare.py rename to .github/.agents/skills/optimize-workflow/scripts/test_workflow_compare.py diff --git a/.github/.agents/skills/right-size-runners/scripts/test_workflow_monitor.py b/.github/.agents/skills/optimize-workflow/scripts/test_workflow_monitor.py similarity index 100% rename from .github/.agents/skills/right-size-runners/scripts/test_workflow_monitor.py rename to .github/.agents/skills/optimize-workflow/scripts/test_workflow_monitor.py diff --git a/.github/.agents/skills/right-size-runners/scripts/workflow_compare.py b/.github/.agents/skills/optimize-workflow/scripts/workflow_compare.py similarity index 100% rename from .github/.agents/skills/right-size-runners/scripts/workflow_compare.py rename to .github/.agents/skills/optimize-workflow/scripts/workflow_compare.py diff --git a/.github/.agents/skills/right-size-runners/scripts/workflow_monitor.py b/.github/.agents/skills/optimize-workflow/scripts/workflow_monitor.py similarity index 100% rename from .github/.agents/skills/right-size-runners/scripts/workflow_monitor.py rename to .github/.agents/skills/optimize-workflow/scripts/workflow_monitor.py diff --git a/.github/.agents/skills/right-size-runners/SKILL.md b/.github/.agents/skills/right-size-runners/SKILL.md deleted file mode 100644 index b5365360a83..00000000000 --- a/.github/.agents/skills/right-size-runners/SKILL.md +++ /dev/null @@ -1,74 +0,0 @@ ---- -name: right-size-runners -description: Properly size CI runners for price, performance, and stability. -disable-model-invocation: true ---- - - -Missing info? Ask user: -1. Workflow to optimize. -2. Priorities (default: stability > speed > cost). -3. "Speed" definition (e.g., cache hits vs raw execution). -4. "Stability" definition (e.g., are flaky tests acceptable?). -5. Determine "spot" setting. Don't ask user directly about what spot setting to use, ask questions to lead you to correct setting. [runs-on spot reference](https://runs-on.com/docs/costs/spot-pricing/) - -Setup: -1. Read workflow. Identify jobs, runners, critical path, and bottlenecks. -2. Ask to optimize specific job or whole workflow. -3. Modify workflow for testing (bypass gates, use mock inputs, add `workflow_dispatch`). -4. Init or resume trial log at `.github/.agents/skills/right-size-runners/trials/.md`. -5. Run a baseline trial with the current runner configuration to establish a performance and stability benchmark. - - - -- OOM/Out of Disk Space failures are NEVER acceptable. -- Prefer default `ubuntu-latest` (4core, 16GB RAM). Use `runs-on` for anything more powerful. -- Validate runner config via available runners API before use. -- Use `gh` CLI for workflow execution and PRs. -- Only change one variable per trial to accurately assess its impact. -- Always compare Apples to Apples - - If looking to optimize speed when caching isn't a factor, ensure that cache hits on one trial do not unfairly advantage it over another trial. - - Always document the exact runner configuration used for each trial to maintain reproducibility. - - - -- [runs-on docs](https://runs-on.com/docs/) -- [available runners](https://go.runs-on.com/api) -- [GitHub Action docs](https://docs.github.com/en/actions) - - - -1. Define trials. Update `.md` log. -2. Ask user to approve trials. Check if they want to run them in parallel or sequentially. -3. New (disposable) branch + commit + push. Message: `cpu=X/ram=Y`. PR title: `[DO NOT MERGE] Trial: ` description: details of trial -4. Trigger workflow. -5. Collect the `workflow_run_id` and run `python3 .github/.agents/skills/right-size-runners/scripts/workflow_monitor.py [run_id] --format json --out-file .github/.agents/skills/right-size-runners/trials/[trial-name].json` to monitor the run and collect details. -6. Analyze results, run `python3 .github/.agents/skills/right-size-runners/scripts/workflow_compare.py .github/.agents/skills/right-size-runners/trials/[trial-1].json .github/.agents/skills/right-size-runners/trials/[trial-2].json --out-file .github/.agents/skills/right-size-runners/trials/[trial-1]-[trial-2]-comparison.md` to compare trial results. -7. Update the trial log with the results and findings. -8. Present user with condensed results and ask if they want to run more trials or stop. - - -| Runner | Experiment | Expectation | Branch | Run ID | Commit | Stability | Runtime | Cost | Notes | -|---|---|---|---|---|---|---|---|---|---| -| `config` | What's tested | Expected result | `branch` | `run-id` | `sha` | Pass/Fail/Flaky | mm:ss | $ | Findings | - - - - -When the user says stop, or all possible experiments have been exhausted: - -* Suggest final recommendations. -* Summarize all findings (cost + speed + stability changes) per workflow/job as table(s) for PR description as below format in raw markdown. - -```md -### [Workflow/Job Name] Runner Changes - -| Approach | Runner | Stability | Runtime | Runtime Delta (Abs/%) | Cost | Cost Delta (Abs/%) | -|---|---|---|---|---|---|---| -| [Old](https://github.com/link/to/baseline/workflow_run) | [original runner before trials] | Pass/Fail/Flaky | mm:ss | +0:00 (+0%) | $ | +$ (+0%) | -| [New](https://github.com/link/to/final/workflow_run) | [new runner] | Pass/Fail/Flaky | mm:ss | +0:00 (+0%) | $ | +$ (+0%) | -``` - -* Remove all debugs, and make final edits on a new branch after approval, and ask user to commit. -* Cleanup all trial branches, logs, and PRs. - \ No newline at end of file diff --git a/.gitignore b/.gitignore index bb9f433a047..813b34301a5 100644 --- a/.gitignore +++ b/.gitignore @@ -26,7 +26,7 @@ tools/clroot/db.sqlite3-wal .env* .dbenv !.github/actions/setup-postgres/.env -/.github/.agents/skills/right-size-runners/trials/ +/.github/.agents/skills/**/trials/ .direnv .idea .vscode/