[Mathis (echobt)]

Summary

Fix the ci run that always ends "cancelled" on main, fix the nightly heavy full-suite kill, and move CI to Blacksmith runners.

Why

• Cancelled on main. ci.yml ran on push: main and was also called by images.yml on the same push. Both runs used the group ci-refs/heads/main with cancel-in-progress: true, so one cancelled the other. Evidence: 8 of the last 10 main-push ci runs were cancelled while images succeeded on the same SHAs (gh run list --branch main --workflow ci).
• heavy killed after ~3 min. On the last 3 nights full-suite got "The runner has received a shutdown signal" after exactly 504 tests, each time inside tests/auditor/test_od_bisect.py. Run locally, test_fault_in_od_encoder_layer_bisected peaks at 18.4 GB RSS and test_fault_in_embed_and_tail at 18.9 GB. The hosted runner has 16 GB, so the kill is an OOM eviction.
• Slow. PR CI ran on 2 vCPU hosted runners and jobs waited in the queue: test-slow (e2e) started 28 min after the run was created.

What changed

1. ci.yml: removed the push: main trigger (main is covered by images → workflow_call). The group is now ci-${{ github.workflow }}-${{ github.ref }}; cancel-in-progress applies only to pull_request. In a called workflow, github.workflow is the caller's name (images), so the images-called run is in ci-images-refs/heads/main and a PR run is in ci-ci-refs/pull/N/merge. They never share a group, and the run that gates publication is never cancelled. Pushes to main still serialize through images-main (cancel-in-progress: false). Workflow and job names are unchanged.
2. heavy.yml: full-suite runs on blacksmith-16vcpu-ubuntu-2404 (62 GB), timeout-minutes 150 → 90 (measured 55 min 27 s + ~60% margin). native and canary pass already and are unchanged. The suite now runs to the end and fails on real tests (77 failed / 17 errors; mostly pre-existing od-fixture and native-only cases, plus 2 test_publish_exclusions failures that are also red locally on main). Tracked in heavy full-suite: 77 real failures once the runner no longer OOMs #19, not hidden.
3. ci.yml: lint, the test shards and test-slow run on blacksmith-8vcpu-ubuntu-2404. The 8 shards and their split are unchanged. The astral-sh/setup-uv cache is kept.
4. images.yml: the called ci runs on Blacksmith, but publish (build, push and attest) stays on ubuntu-24.04 (GitHub-hosted); see Risk. docker/setup-buildx-action and docker/build-push-action are kept. The attestation subject is that action's digest output, and provenance is signed by actions/attest-build-provenance. A branch must never publish, so a Blacksmith builder could not be proven to keep the exact-digest + gh attestation verify path before merge.

• miner-flow.yml was checked and left unchanged. It is not called by another workflow, and its main-push runs are all success. The single cancelled run was a superseded PR push, which is the intended behaviour.

Runner check (canary run 39420280084, since removed)

All of 4/8/16/32 vCPU ubuntu-2404 started within about 10 s, even though 8 and 16 vCPU are not listed in any org runner group. All Blacksmith sizes report AMD EPYC, avx512f=yes, systemd as PID 1, and docker 28.0.4. The GitHub-hosted runners report AMD EPYC 7763, avx512f=no. The CPU class differs. ATEN_CPU_CAPABILITY=avx2 is still forced. tests/trainer/test_golden_replay.py (shard 0) passed with no xfail on Blacksmith (173 passed), and GOLDEN is unchanged.

How it was tested

	before	after (this PR)
PR ci wall clock	45 min 32 s (39352445497)	15 min 21 s (39420785993, b9097b4)
slowest job (run time)	test (7) 20 min 53 s	test (7) 15 min 09 s
images (ci + publish)	28 min 48 s (39356735098)	measured after merge
heavy full-suite	killed at 3 min 18 s (39362756310)	ran to completion in 55 min 27 s (39420692371); 77 real failures → #19

python3 -c "import yaml,..." .github/workflows/*.yml: all 6 files valid. actionlint is not installed on the host.

Risk and rollback

Blacksmith billing now applies to these jobs. Rollback: revert the runs-on commits, which are independent of the cancellation fix.

Attested builds stay GitHub-hosted (commit 14068dd). A Blacksmith runner is recorded in the build provenance as runnerEnvironment=self-hosted. Cortex verifies its own image attestations with gh attestation verify --deny-self-hosted-runners, and the existing hypertrain attestations are github-hosted; the miners' and operators' deployment guidance checks provenance too. So the publish job (the only one that creates provenance) returns to ubuntu-24.04; everything else (lint, shards, test-slow, heavy, and the ci called by images) stays on Blacksmith. No measurement of publish alone on Blacksmith vs hosted was made, so nothing here argues against the hosted default.
CPU class. Blacksmith runners report AMD EPYC with avx512f=yes; GitHub-hosted report EPYC 7763 with avx512f=no. ATEN_CPU_CAPABILITY=avx2 is still forced and tests/trainer/test_golden_replay.py passed without xfail in this PR's runs.

Checklist

• Actions unchanged and still SHA-pinned; no new action, no secret, no continue-on-error
• Workflow and job names unchanged
• images after-merge timing and attestation verify (cannot run from a branch)

🤖 Generated with Claude Code

Summary by CodeRabbit

• Chores
  • Updated continuous integration workflows to use higher-capacity runners for standard and full test suites.
  • Reduced the maximum runtime for the full test suite.
  • Adjusted workflow execution and concurrency behavior to avoid unnecessary runs while preserving pull request and reusable workflow support.
  • Documented infrastructure choices for image publication and build attestations.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 0ae494eb-6c5d-4eae-bb46-ba34f7a39ae2

📥 Commits

Reviewing files that changed from the base of the PR and between b9097b4 and 14068dd.

📒 Files selected for processing: 2

• .github/workflows/heavy.yml
• .github/workflows/images.yml

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

The CI workflow no longer runs on direct pushes to main. Concurrency handling and runner assignments changed. The full-suite timeout decreased, and image publication documentation now explains its GitHub-hosted runner requirement.

Changes: CI workflow runner updates

Layer / File(s)	Summary
Standard CI scheduling and runners .github/workflows/ci.yml	The workflow no longer triggers on pushes to main. Its concurrency group now includes the workflow name, and cancellation applies only to pull request runs. The lint, test, and test-slow jobs use blacksmith-8vcpu-ubuntu-2404.
Full-suite and image publication execution .github/workflows/heavy.yml, .github/workflows/images.yml	The full-suite job uses blacksmith-16vcpu-ubuntu-2404 and a 90-minute timeout. Comments document its memory requirement. Image workflow comments document the required GitHub-hosted runner for image publication and distinguish it from Blacksmith-based CI.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Suggested reviewers: mathismassimino-hub

Merge Risk: ⚪ Minimal · up to 14068

CI routing and provenance execution remain intact, with no actionable merge-blocking risk identified.

Pre-merge checks: 5

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly summarizes the main CI changes: preventing cancellation, addressing the heavy-suite OOM issue, and moving CI jobs to Blacksmith runners.
Docstring Coverage		No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (2 skipped: 2 unsupported.)

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}