fix(rendering): stop the AVAssetReader decoder answering with frames from the wrong time

[richiemcilroy]

The AVAssetReader decoder (macOS) answers some frame requests with the wrong source frame: a jittery camera runs up to 74 s ahead in a 120 s export, a camera whose first sample is late plays ahead from the first frame, seeks land in the previous GOP, and reused readers answer from an older seek. The pts-hole map that causes the camera runaway is shared with the FFmpeg decoder, so the same 120 s export is also wrong on Linux and Windows builds that decode with FFmpeg.

This PR fixes six request-handling root causes in crates/rendering/src/decoder and the AVAssetReader start time in crates/video-decode. It supersedes #2595 (its decoder commits) and #2603. The IOSurface part of #2595 already shipped in #2625.

flowchart TD
    R[frame request] --> P{before the track's first frame?}
    P -- yes --> F[answer with the first frame from the cache]
    P -- no --> S[pick a pooled reader]
    S --> X{reader already vended past the request,<br/>or its newest cached frame comes from an older seek?}
    X -- yes --> Z[reset: seek to the exact keyframe tick]
    X -- no --> V[read forward]
    Z --> V
    V --> G{request falls in a pts gap}
    G -- previous vend still cached --> H[answer with the previous vend]
    G -- previous vend evicted --> T[retry once from a fresh seek]

Loading

1. Pts-hole map drops the hole it just recorded

record_pts_hole (crates/rendering/src/decoder/mod.rs:683) kept 64 holes and evicted the narrowest one. A camera that delivers about 12 fps into a 30 fps track creates a narrow hole per sample, so after 64 holes every new hole was evicted as soon as it was recorded. Requests inside the forgotten holes read one sample further ahead each time, and the camera drifted up to 2232 source frames (74 s) ahead on AVAssetReader. FFmpeg uses the same map and showed 1551 stale camera frames instead.

• The map now keeps up to 16384 holes (MAX_TRACKED_HOLES, mod.rs:677), enough for every hole of a realistic recording.
• When full, it drops the hole farthest from the one just recorded. Requests trail the reader, so the newest holes are the ones about to be asked for.

2. A pooled reader that already passed the request is reused

After a read, a reader's pool position is capped to the newest request, so a reader that decoded past a frame still looks usable for it. It can never vend that frame again, and its next sample is a later frame (seek 0, then 8.8 s, then 0.017 s shows source 1 for source 0).

• reader_passed_request (crates/rendering/src/decoder/avassetreader.rs:1384) makes select_best_decoder reset such a reader before reading.

3. The reader's previous vend is no longer held

When a request falls in a gap between two vended frames, the only correct answer is the frame vended just before it. Cross-fades read one decoder at two positions, and a reset for one position purged the other position's frames from the cache; a backward jump hit the same eviction. The nearest older cached frame then answered (1048 for 1049).

• Each pooled reader's last vended frame is pinned against cache eviction and against the reset window (cache_eviction_victim, avassetreader.rs:1402, and pinned_frames, avassetreader.rs:968).
• If the answer is still not the previous vend, the request is retried once from a fresh seek (answer_misses_previous_vend, avassetreader.rs:1390), which re-vends that frame.

4. Requests before a late track's first frame step the reader

A camera whose first sample is 2 s into the recording should hold that first frame until then. Every earlier request read one more sample instead, so the camera ran up to 53 frames ahead from output frame 8.

• Requests before the first keyframe are answered with the first frame from the cache (avassetreader.rs:879), with no reset and no passed-reader seek.

5. AVAssetReader starts on a keyframe rounded to 10 ms

get_reader_track_output started the reader at cm::Time::with_secs(t, 100). When the keyframe time rounds down below the keyframe pts, the reader decodes the whole previous GOP and vends its last frame, so the frame after a seek showed keyframe - 1.

• The reader now starts on the keyframe's exact stream tick (KeyframeIndex::exact_time, crates/video-decode/src/avassetreader.rs:384, used by reader_start_time, :411). Without a keyframe index it keeps the old start.

6. A reused reader answers from an older, unrelated seek

A request the reader has passed, or that falls in a gap, is answered from the newest cached frame at or before it. That frame is only valid if this reader vended it in its current run; otherwise it is a leftover from an earlier seek (seek storm frame 349 showed 237).

• reuse_would_serve_stale_frame (avassetreader.rs:1430) tracks the start of the current vend run and resets the reader when the cached answer predates it, or when nothing at or before the request is cached.

Wrong frames per case

Oracle suite, full tier, wrong frames per stream (checked frames in brackets). macOS on Blacksmith macos-15 runners, 3 full runs each on main and on this branch; Linux on ubuntu-24.04.

case	platform	main	this PR
export/jitter-camera-120s/full/mp460/avf	macOS	camera 5146 ahead, up to 2232 frames (7199)	0
playback/jitter-camera-120s/full/seq60-from0/avf	macOS	camera 2618 ahead, up to 1726 frames	0
export/jitter-camera-120s/full/mp460/ffmpeg	Linux, macOS	camera 1551 stale, up to 90 frames	0
playback/jitter-camera-120s/full/seq60-from0/ffmpeg	Linux, macOS	camera 1551 stale, up to 90 frames	0
export/jitter-camera-40s/full/mp460/avf (fast tier)	macOS	camera 531 ahead, up to 298 frames	0
export/camera-pts-late/full/mp430/avf	macOS	camera 127 ahead (179)	0
export/backward-hold/backward-jumps/mp460/avf	macOS	camera 1 stale (3177)	0
export/basic/transition/mp460/avf	macOS	screen 1 stale (273)	0
playback/basic/transition/seq60-from0/avf	macOS	screen 1 stale (273)	0
playback/basic/cuts-speeds-reorder/realtime30-from25/avf	macOS	1 to 2 stale per stream, every run	0
seek/seek-gop/keyframes60/avf	macOS	3 stale per stream (58)	0
seek/seek-native/keyframes60/avf	macOS	8 stale + 1 ahead per stream (66)	0
seek/seek-gop/storm30/avf	macOS	20 stale + 3 ahead per stream (7497)	0
seek/seek-native/storm30/avf	macOS	20 to 28 stale + 7 to 11 ahead per stream	0
seek/vfr-holds/storm60/avf	macOS	screen 125, camera 50 (7506)	0
seek/vfr-holds/into-holds60/avf	macOS	camera 1 stale (240)	0
seek/vfr-holds/revisit60/avf	macOS	camera 1 stale (120)	0

The scripted decoder model (decoder::model, 3024 AVAssetReader and 2864 FFmpeg sequences, about 3.7 M requests) agrees: 27 AVAssetReader and 8 FFmpeg cases no longer fail, and 4 more lose failure classes. No case changes status on Windows, and no case that passes on main changes on any platform.

Exports that main already gets right are byte-identical (ffmpeg framemd5, decoded PCM SHA-256) across 53 MP4, GIF and MOV outputs (20,740 frames) of the fast export tier, both decoders. The only differing outputs are the four exports above that are wrong on main, plus export/backward-hold/backward-jumps/mp430/avf, where one output frame lands exactly on a camera sample boundary: main shows sample 1048, this PR shows 1049, the exact slot. The 60 fps export of the same timeline flags main's 1048 as stale.

Performance

Wall time per suite run on Apple silicon, both arms built on main with #2648; 4 interleaved pairs, median:

workload	main	this PR	wall	instructions
export/basic/*	23.85 s	23.88 s	+0.1%	-0.1%
export/jitter-camera-120s/*	76.33 s	75.06 s	-1.7%	-1.2%
seek/*/storm30/*, both decoders	295.0 s	275.6 s	-6.6%	-3.3%
seek/vfr-holds/storm60/*, both decoders	185.7 s	171.9 s	-7.4%	-7.4%

Confidence Score: 5/5

The PR appears safe to merge; no actionable new issues were found.

Summary

Fixes source-frame selection during macOS playback, seeking, and export.

• AVAssetReader requests now use frames from the right seek and reader.
• The shared decoder keeps recent gaps between video frames.
• Indexed AVAssetReader starts and resets on exact keyframe ticks.

Review was based on code inspection; no runtime tests were run.

_{Reviews (4) · Last reviewed commit: "test: remove the expectations the AVAsse..." · Reviewed by Greptile}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
cap-status	Ready	Preview	Oct 16, 2026 3:29pm UTC

[capy-ai]

Capy found no issues.

View 1 other finding in Capy.

[capy-ai]

Capy found no issues.

[richiemcilroy]

hey @greptileai, please re-review the PR

[richiemcilroy]

hey @greptileai, please re-review the PR

[capy-ai]

Capy found no issues.

[capy-ai]

Capy found no issues.

[richiemcilroy]

@greptileai review