Skip to content

ci: verify real capture on Blacksmith and run the FULL tier once a PR is ready - #2652

Merged
richiemcilroy merged 3 commits into
mainfrom
ci/real-capture-on-blacksmith
Oct 16, 2026
Merged

richiemcilroy merged 3 commits into
mainfrom
ci/real-capture-on-blacksmith

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Real-capture verification (Verify recording) now records on Blacksmith macOS for pull requests, pushes and dispatches, so it can become a required check, and draft pull requests run the FAST tier.

Event Before After
PR, low risk FAST, GitHub-hosted macOS 15 FAST, Blacksmith macOS 26
PR, high risk or engine crate FULL, GitHub-hosted macOS 15 and 26 FULL, Blacksmith macOS 26
Nightly FULL, GitHub-hosted macOS 15 and 26 FULL, Blacksmith and GitHub-hosted macOS 15 and 26
Dispatch default GitHub-hosted Blacksmith (runners=hosted or both still available)
  • Blacksmith jobs start immediately with no concurrency cap; GitHub-hosted macOS shares a 5-job cap across the organisation.
  • GitHub-hosted runs added rare failures with four different signatures: screen frames missing at the Studio start gate, one camera frame lost, a VideoToolbox encode error at finish, and an oracle self-check that did not trigger.
  • Five repeats on main (run 38931552822): Blacksmith macOS 26 recorded all 65 recordings as expected. Blacksmith macOS 15 recorded every Studio case as expected, but its Instant recordings with system audio lost one to three microphone markers in every run, on main and with fix(recording): align mixed Instant audio sources to their capture times #2647. Pull requests run on macOS 26 until that finding is fixed; the nightly keeps macOS 15.
  • Blacksmith VMs have no VideoToolbox hardware H.264 encoder (-12903, probe run 38847791287), so their recordings use libx264. The nightly run keeps GitHub-hosted runners for the hardware encoder path.

Tiers

  • Draft pull requests that change engine or high-risk code run the FAST tier of both Engine verify and Verify recording on every push.

  • Marking a pull request ready for review, and any push after that, runs the FULL tier on the head that will merge. The verify-full label still forces FULL on a draft.

  • Blacksmith setup (Screen Recording and Microphone TCC grants, display wake, BlackHole audio) is unchanged in .github/workflows/verify-recording.yml.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge.

Summary

Moves real-capture checks to Blacksmith and keeps draft PRs on FAST unless FULL is explicitly requested.

  • Engine checks use FAST for drafts and FULL after review.
  • Real-capture checks use Blacksmith for pull requests and cover both runner types nightly.

Reviews (2) · Last reviewed commit: "ci: verify pull requests on Blacksmith m..." · Reviewed by Greptile

Pull requests, pushes and dispatches now record on Blacksmith macOS 15
and 26: jobs start at once with no concurrency cap and every run gets the
same machine. GitHub-hosted runs added rare failures with four different
signatures (start frames, a camera frame, a VideoToolbox encode error, a
self-check). Blacksmith has no hardware H.264 encoder, so the nightly run
also records on GitHub-hosted macOS 15 and 26 to keep the VideoToolbox
path covered.
@vercel

vercel Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
cap-status Ready Ready Preview Oct 16, 2026 5:04pm UTC

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

Open in Capy

@richiemcilroy
richiemcilroy marked this pull request as draft October 16, 2026 16:33
Draft pull requests that change engine or high-risk code now run the FAST
tier on every push; marking the pull request ready (and any push after
that) runs the FULL tier on the head that will merge. The verify-full
label still forces FULL on a draft.
@richiemcilroy richiemcilroy changed the title ci: run real-capture verification on Blacksmith macOS ci: verify real capture on Blacksmith and run the FULL tier once a PR is ready Oct 16, 2026
…stant finding is fixed

Five repeats on main: Blacksmith macOS 26 recorded every case as expected
(65 of 65); Blacksmith macOS 15 recorded every Studio case as expected but
its Instant recordings with system audio lost one to three microphone
markers in every run. Pull requests run on macOS 26; the nightly keeps
Blacksmith and GitHub-hosted macOS 15 and 26.

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

@richiemcilroy
richiemcilroy merged commit 68bb422 into main Oct 16, 2026
61 checks passed

This branch was successfully deployed

2 active deployments
Preview – cap-web — 462334b5 Deployed Oct 16, 2026 by vercel[bot]
Preview – cap-status — 462334b5 Deployed Oct 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant