ci: add the Engine verify oracle suite for export, playback, seek, audio, recording, captions and upload

[richiemcilroy]

Engine verify: oracle regression suite for the GPUI desktop engines

Synthetic video frames carry machine-readable identities, and timed audio markers identify each source. Independent reference models define the expected frames and audio positions. The oracles decode the real output and compare it with those references. Known product bugs are strict expected failures, so a fix that lands without updating them fails CI, and a new failure fails CI.

flowchart LR
  F["fixtures<br/>marker video + FSK audio markers<br/>(synthetic, and real Studio / Instant<br/>pipelines recorded in real time)"] --> P["product code<br/>export, playback, seek, decoders,<br/>audio output, Instant finalize,<br/>captions, upload queue"]
  P --> O["oracle<br/>decode output, read markers"]
  M["reference model<br/>timeline, audio and decoder models"] --> O
  O --> R["case record (JSON line)<br/>classes: stale, future, sync,<br/>timestamps, audio-missing, ..."]
  R --> J{"judge against<br/>expectations.toml"}
  J -->|class not listed| FAIL
  J -->|strict class gone| XPASS["FAIL: remove the entry"]
  J -->|listed and seen| XFAIL[ok]

Loading

What it checks

• Export: every frame of MP4 (30 and 60 fps), GIF, MOV and 4K exports shows the right source frame of the right segment, through cuts, speeds, reorders, transitions, VFR holds, late and early cameras, crop and camera offsets, on AVAssetReader and FFmpeg decoders. Two exports of the same project must be identical.
• Playback and seek: offline editor playback, seeks, scrubs and random seek storms answer each request with the frame the model expects.
• Audio: every mic, system-audio and music marker is heard exactly once, at the right time (5 ms) and level (25%), in exports and in offline playback, including speed modes, mute, volume, Studio Sound and the stale-startup repair.
• Instant and Studio recordings: fixtures recorded in real time through the real OutputPipelines and finalize_instant_output, checked for frame timestamps and A/V sync against the emitted truth.
• Real capture (macOS runners): a stimulus process draws screen markers and plays system-audio markers while Cap records through ScreenCaptureKit and VideoToolbox with real-time synthetic camera and microphone feeds. Every frame and marker is checked against the host clock.
• Decoder model: model-based fuzzing of the AVAssetReader and FFmpeg decoders' request decisions on scripted media, per seed, continuing after a wrong answer to record the full failure set.
• Crash recovery: compare in-process and out-of-process muxers, kill the recorder or muxer at known capture points, and compare recovery with the fragments physically on disk.
• Captions and editing: caption segmentation, timing and migration; random edit sequences exported frame by frame.
• Upload: queue retry, backoff and resume (in cargo test --bin cap-gpui).

Tiers and timing

Tier	Runs on	Wall time
FAST	every PR push, change-scoped by scripts/verify/plan.py	a subset of FULL
FULL	every PR touching an engine crate, verify-full label, merge queue, nightly, dispatch	20 to 24 min; longest test job: Linux 7, Windows 9, macOS 11, hosted macos-vt 20 min
Real capture FAST	PRs touching capture, encode, timestamp or verify code	7 min (macos-15)
Real capture FULL	high-risk PRs, nightly, dispatch	23 min, 3 repeats on macos-15 and macos-26

• plan.py maps changed files to suites through the workspace dependency graph. Docs and web-only changes run nothing; any apps/desktop-gpui/** change runs every suite.
• Shards split cases by a stable hash of the case id. Every test has a 15 minute limit (30 for the FULL seek storms, twice their slowest measured run), so a hang fails with a named timeout in the job summary.
• Linux, macOS and Windows engine jobs run on Blacksmith. Windows uses the WARP software adapter; Media Foundation cases report allowlisted unsupported. The long jitter-camera and seek-storm cases run on Linux and macOS.
• Blacksmith macOS has no hardware VideoToolbox encoder. Its exports use libx264; a separate hosted macos-vt job repeats export and audio suites on the hardware encoder for every FULL run and changes reaching encoding paths.
• Each suite declares an executed-case floor, checked across the collected shards. Zero-test binaries fail. Unsupported cases outside the explicit platform allowlist fail. Fixture cache keys include vendor code, build scripts, manifests and the Rust toolchain.

Expectations model

One format for every suite (crates/verify/expectations.toml, crates/rendering/src/decoder/model/expectations.toml, crates/rendering/tests/expectations.toml):

[[xfail]]
case = "export/jitter-camera-40s/full/mp460/avf"
class = "future"
platforms = ["macos"]
bound = { camera-frames = [500, 560], delta = 320, frame = [1580, 2399] }
reason = "Camera runaway after the pts-hole map evicts an earlier hold"
link = "https://github.com/CapSoftware/Cap/pull/2638"

• Every engine expectation has a measured signature: affected sources, frame count and range, source-frame error, timing offset or another case-specific metric. A failure outside the bound fails CI; a measured metric absent from the bound must be zero.
• Strict (default): the class must occur. If it stops occurring the run fails with "remove this xfail", so fixes are locked in.
• Flaky: allowed only with a measured mechanism in the reason. The summary counts every failure excused by such an entry. Host scope is reserved for environmental mechanisms; an encoder-specific product failure is not an environmental exception.
• Sources are judged separately: audio-mic and audio-system, sync-mic and sync-system, and the real-capture source pairs. A known failure in one source cannot hide a new failure in the other.
• A starvation sentinel records scheduling gaps. Retry requires at least 1 s of measured stalls overlapping the 60 s before a timeout or 15 s before a GPU panic, or an inconclusive real-time case. A test is retried once, at most three tests per job, with a warning and summary count. A second failure remains a failure. The narrow starvation-only export expectations also require their documented frame-order signature.
• Not measured: a class that cannot be checked because an earlier check failed is recorded as not measured. It never confirms or clears an expectation. An unmeasured class without an explanatory failure fails as a harness error.
• Real capture checks both start and end coverage against the stimulus clock. A missing picture is excused as a capture-service drop only when the stimulus's own observer covered that window and also missed it. The summary counts those drops and cases requiring a second attempt.
• The cross-fade oracle checks both sides across the transition; an alternative frame pair must improve the fit by at least 0.05 and leave less than one third of the expected residual. Real-time playback also has a 75% delivery floor and a 2 s warm-up limit on Linux and macOS.
• scripts/verify/bless.sh --run <id> can propose expectation updates from all platforms. Those proposals require manual review of the mechanism and bound. Every new or expanded proposal has a TODO reason that the Plan guard rejects until explained. JSONL reports carry their source host; proposals use that host and preserve unobserved host slices. Legacy reports without host provenance cannot remove or narrow existing entries, and entries covering every host remain intact.

Guard warnings and escalation

• scripts/verify/guards.toml lists high-risk regions (encoder timestamps, audio mixing and Studio Sound, decoder seek and frame selection, recording timestamps and sync, Instant finalize, recovery, upload retry) and guard sentinels: small tests pinned to one mechanism (E02 encoder pts rebase in crates/enc-ffmpeg/src/base.rs, A09 Studio Sound seek reset in crates/audio/src/voice.rs).
• A PR that touches a region runs the FULL tier, and real capture on both macOS versions, and gets the needs-deep-review label. A failing or missing sentinel is a warning in the job summary, not a silent pass.

Running it locally

scripts/verify/run.sh fast                          # what CI runs on this OS for a PR
scripts/verify/run.sh full --suites seek            # one suite, FULL tier
scripts/verify/run.sh fast --case 'export/basic/*'  # case globs
CAP_VERIFY_SEEK_SCRIPT='seek-native@60:0,8.8,0.016667' \
  cargo test -p cap-verify --test seek script -- --nocapture   # replay a seek sequence
cargo test -p cap-verify --test recording -- --nocapture       # real capture (macOS, screen recording permission)

Each failing case prints a one-line repro with its CAP_VERIFY_CASE.

Product bugs it found

The suite records the following product findings. Merged fixes have their obsolete expectations removed; open fixes remain subject to the final proof and baseline comparisons.

Area	Bug	Fix
Rendering (macOS)	The decoded pixel buffer was released before GPU sampling, allowing recycled IOSurfaces to corrupt transitions and exports	#2625, merged
Recording timestamps	After warm-up, video timestamps followed the muxer's processing clock; stalls shifted later screen or camera frames	#2632, merged
Camera offset	The calculated camera offset repeated the decoder's start alignment	#2633, merged
Recording audio	Delivery catch-up made the overlap detector delete real samples	#2635, merged
Camera capture	Runtime stalls silently dropped camera frames	#2637, merged
Export colour	BT.601 conversion coefficients were used for a stream tagged BT.709	#2639, merged
Export under load	A decode deadline emitted a fallback ahead of an outstanding GPU readback; PTS repair shifted later frames	#2636, merged
AVAssetReader decoder	Reader selection, evicted pts holes, cross-fade resets and rounded keyframe seeks returned frames from the wrong time	#2638, open; supersedes #2595 and #2603
Crash recovery	A physically complete fragment still marked pending was discarded	#2640, merged
Studio crash loss	On a static screen a segment spans up to 20 s and stays in the writer's memory, so a crash lost all of it	#2646, open (replaces #2641)
Recovery track offsets	Recovered tracks lost their original start-time relationship	#2642, merged
Out-of-process muxer	Buffered packets were not flushed to the child, increasing loss and hiding child failure	#2643, open
Recorder death	The muxer did not finish the open fragment after the recorder died	#2644, open
Muxer restart timeline	Recovery rebases fragments contiguously after a muxer crash, moving video 1.2 to 2.0 s early	Separate follow-up, exposed by #2643
FFmpeg decoder	First-frame shortcuts, same-clip cross-fades and backward-seek lookback can return stale frames	Follow-up
Instant mixer	Each mixer input's first sample became sample 0 in amix, so system audio landed up to 60 ms before the microphone	#2647, open
Instant audio	First-sample anchoring of the mixed track moves late audio early; gated audio and loopback also have bounded sync or missing-marker findings	Follow-up
Renderer lifecycle	Dropping the last queue handle with outstanding GPU work can hit wgpu's queue-drain timeout	Follow-up
Editing	Fullscreen text after the clips is omitted and the export ends early	Follow-up

Known limits

• Engine jobs run on Blacksmith. Windows exercises FFmpeg plus WARP rendering; Media Foundation is not tested because WARP has no D3D11 video device. Blacksmith macOS exports use libx264, with hosted macos-vt covering the hardware VideoToolbox encoder.
• Real capture runs on hosted macOS 15 and 26 and uses synthetic camera and microphone feeds. It does not validate physical devices. Blacksmith real capture is an optional dispatch covering the software encoder path.
• Real-capture system audio is not verified by the default jobs: hosted output devices yield silence to ScreenCaptureKit, and Blacksmith keeps its stimulus silent unless explicitly enabled. BlackHole latency is not included in the stimulus truth. The allowlisted system-only case reports unsupported in those default jobs.
• Strict AVFoundation expectations are calibrated on the runners. Some pass on a local Apple-silicon Mac and correctly report an unexpected pass there.
• Two rare real-capture failures are under investigation and keep Verify recording out of the required checks for now: Studio recordings occasionally miss screen frames 3 to 6 right after the start gate (1 in about 20 runs, not the capture-service drop signature), and one Studio camera frame was lost mid-recording once in 13 runs with no machine stall.
• Existing bounded flaky entries remain visible in summaries. Seeded decoder-model nondeterminism requires investigation; new flaky entries are not an automatic response to differing runs.

Baseline comparisons

Both baselines must run the final suite before comparison. Missing baseline APIs are N/A and do not count as passes. Required source tracks absent from a baseline recording are explicitly unobserved; newly measured classes appear in a separate coverage-gained column, while main is still judged against its own expectations. Other new failing classes and larger failure magnitudes or shares are reported as worse. A class observed on the baseline that becomes unobserved on main is reported as worse and counted in its own coverage-lost column, including when the baseline class passed. Simultaneous coverage gains do not hide losses.

The environment-dependent real-capture missing-after-hold class requires at least three independent runs per side on the same OS image. Main is worse only when its per-run counts are all above the baseline range; overlapping ranges and insufficient repetitions remain explicit comparison outcomes.

Three FULL Engine runs per side, and three real-capture repeats per macOS version per side. Cells count cases worse / better than the baseline.

Baseline	Linux	macOS	Hosted macOS VT	Windows	Real capture macOS 15	Real capture macOS 26
Nightly, 2c51caa	2 / 5	7 / 33	2 / 10	0 / 5	0 / 4	0 / 2
Production, cap-v0.6.0	3 / 130	7 / 36	2 / 13	0 / 58	0 / 3	0 / 3

Every worse case has a cause:

Case	Worse than	Cause	Disposition
export/camera-pts-late, export/jitter-camera-120s, AVF decoder-model camera-fast/sequential/export and syn/edlist/export, seek/vfr-holds/into-holds60/avf, seek/seek-native/storm30/avf	both	#2516 shrank the sequential decoder cache, so AVAssetReader answers from a later frame when nothing at or before the request is cached	Fixed by #2638
Linux editing/export/basic/seed-5, multi-segment/seed-7, same-clip-transition/forward/ffmpeg	both	#2639's BT.709 colour lets the oracle see stale transition frames it previously accepted	Detection improvement; the FFmpeg same-clip reuse bug is a listed follow-up
seek/seek-native/storm30/ffmpeg	both	Two modes between runs: main 78, 65, 78 stale seeks; baselines 65	Run-to-run variation, not a regression
playback/basic/transition/seq60-from0/avf	nightly	Present at the nightly itself (3 of 3 vs 2 of 3 runs)	Existing bounded flaky entry

Product code touched

Test seams only, behind cfg(test) or features that release builds never enable: the scripted decoder (crates/video-decode, scripted), an offline audio sink (crates/editor, test-hooks), synthetic camera and microphone feeds (crates/recording, test-utils), and the decoder model (cfg(test) in crates/rendering). Three behaviour-identical refactors make code reachable from tests: transcribe_project_with and caption_segments_from_tokens in apps/desktop-gpui/src/transcription.rs, sync_video_playhead in crates/editor/src/audio_output.rs, and the ReaderSource enum in crates/video-decode/src/avassetreader.rs (one variant in release builds). The upload manifest throttle uses tokio::time::Instant, which is std::time::Instant unless a test pauses tokio's clock.

Confidence Score: 5/5

The changes since the previous review appear safe to merge.

Summary

Adds engine regression tests that compare real media output with independent frame and audio references.

• Engine changes now trigger planned verification runs across CI platforms.
• Marker-based oracles compare engine output with independent reference models.
• Recording checks compare real pipeline output with timed screen and audio markers.
• Scripted media lets decoder tests check frame choices and seek behavior.
• Caption and upload tests check editing and retry behavior with controlled inputs.
• Test-only audio and recording seams expose existing product paths to verification.

_{Reviews (5) · Last reviewed commit: "test: build the pending-fragment bound i..." · Reviewed by Greptile}

[richiemcilroy]

@greptileai review

[richiemcilroy]

@greptileai review

[superagent-security]

Superagent found 1 security concern(s).

[superagent-security]

P2: Fork pull requests can repeatedly consume the Blacksmith runner pool with full engine verification

Fork PRs can trigger up to 9 Blacksmith FULL-tier test shards; cancellation is per PR, so separate PRs can multiply runner load.

Keep fork PR jobs on GitHub-hosted runners or enforce a global Blacksmith concurrency/quota limit.

AI prompt

Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name=".github/workflows/engine-verify.yml">
<violation number="1" location=".github/workflows/engine-verify.yml:16">
<priority>P2</priority>
<title>Fork pull requests can repeatedly consume the Blacksmith runner pool with full engine verification</title>
<evidence>The new `pull_request` trigger at `.github/workflows/engine-verify.yml:16-17` admits fork PRs; the workflow comment at lines 208-210 confirms fork PRs run with a read-only token. A PR that changes an engine crate is escalated to FULL at lines 164-178, then dispatched through the platform matrix at lines 228-245. `scripts/verify/plan.py:41-65` makes changes under `crates/rendering/**`, `crates/recording/**`, and other engine roots select that tier. The added runner matrix in `scripts/verify/suites.json:304-330` fans a FULL run out to 3 Linux, 2 macOS, and 4 Windows Blacksmith shards. These are configured as self-hosted runner labels in `.github/actionlint.yaml:1-6`. The jobs execute tests on each run (`engine-verify-os.yml:112-117`), with a 60-minute job timeout; `.config/nextest.toml:35-41` documents seek storms taking 240-570 seconds per decoder and up to about 900 seconds per shard. Restored build and fixture caches do not skip test execution. The per-PR concurrency group at `engine-verify.yml:59-65` cancels superseded runs for that PR, but distinct PR numbers use distinct groups and the checkout contains no global concurrency cap or per-actor quota. An external contributor can therefore open or update multiple fork PRs targeting engine paths to occupy additional shared Blacksmith CI capacity. This is incremental over the base: `repository-base/.github/workflows/ci.yml:251-284` ran its Linux Rust tests on `ubuntu-24.04`, and lines 286-301 used GitHub-hosted macOS, Windows, and Ubuntu runners; the base had no Engine verify workflow or Blacksmith runner matrix. The PR also moves existing CI jobs to Blacksmith, including the Linux Rust test runner (`.github/workflows/ci.yml:256`) and cross-platform build runners (`:296-301`).</evidence>
<recommendation>Keep fork-controlled PR workloads on GitHub-hosted runners and reserve Blacksmith runners for trusted branches, or apply an enforceable global concurrency/quota limit to PR jobs before dispatching the FULL matrix. Preserve the per-job timeouts and cache strategy, but do not rely on per-PR cancellation to protect the shared runner pool from multiple PRs.</recommendation>
</violation>
</file>