fix(video): stop re-encoding earlier segments when extending videos

[lstein]

Summary

Extending a video repeatedly made artifacts build up in its earliest segments. Each Extend run decoded the whole source and re-encoded it twice, first in extract_video_range (even for an untrimmed source) and then in video_concat. Both used imageio's default quality, which is libx264 CRF 25. After N extensions the first segment had been through 2N lossy encodes, and its soundtrack through 2N AAC encodes.

The earlier footage is now carried forward without re-encoding wherever the files allow it:

• video_concat. When the first clip was encoded by this app at the output canvas and frame rate, everything up to its last usable IDR keyframe is stream-copied. Only the rest is encoded: the clip's tail, the transition and the later clips. The soundtrack works the same way. The first clip's own AAC packets are kept up to four frames before its outgoing transition, and only the remainder is encoded, with four frames of pre-roll so the seam stays at coding-noise level.
• extract_video_range. A whole-range extract at the source's own frame rate is a copy of the source file. A trim stream-copies the keyframe-aligned run of frames inside the range and encodes only the partial keyframe intervals at each end. Its soundtrack is treated the same way: the source's AAC packets for every frame wholly inside the range are copied, and only a short lead-in and tail (5–6 AAC frames each) are encoded. Both nodes use one verified splice, aac_splice.splice_soundtrack.
• AAC encoding. Soundtracks are encoded without perceptual noise substitution (-aac_pns 0). The decoder fills PNS bands from a random generator whose state carries across packets, so a lead-in that used PNS would change how the copied packets after it decode, and they could not be verified.
• make_mp4_writer. Every video node now encodes at CRF 18 and forces an IDR every 2 seconds, so later joins have cut points.

Anything that can't be spliced losslessly falls back to the previous full re-encode. That covers uploads, files from older Invoke versions, retimed joins, rotated or edit-list-trimmed sources, and open-GOP streams. Every splice is checked before it is used:

• Video: the encoder headers (avcC) must match, there must be a single timeline with a uniform frame duration, and the expected frame counts must appear.
• Audio: the copied head must decode back bit-exact, and the seam must be no noisier than freshly encoded audio.

Before and after, extending a test clip three times (CRF 25 → CRF 18 plus splicing):

	Before	After
Earliest segment, vs. the clip as first generated	−0.6 dB PSNR per encode, 2 encodes per extension (31.2 → 28.0 dB after 3 rounds)	bit-identical after every round
Frames between the last keyframe and each seam	re-encoded every round	re-encoded once (47 dB vs. the original), then copied
Earlier soundtrack	one AAC generation per encode	bit-identical, apart from a few AAC frames at each seam (~0.1 s at 48 kHz)

Related Issues / Discussions

Reported in discussion: repeated "extend video" runs showed growing artifacts in the earliest segments.

#9834, which was stacked on this branch, has been merged into it, so this PR now also includes the trimmed-source soundtrack splice described above.

Follow-up planned: copy the safe regions of every clip in a join, not just the first clip's head, and seek past copied prefixes instead of decoding them.

QA Instructions

• uv run --no-sync pytest -n 4: 13305 passed, 210 skipped, 8 xfailed. This run predates the final review round. Since then, the video/audio/metadata subset (-k "video or mp4 or metadata or wan or ltx2 or minimax or h264 or audio") has been rerun on the final code (with ingest and upload added): 1737 passed.
• uv tool run ruff@0.11.2 check invokeai tests and format --check: clean.
• New end-to-end tests run real ffmpeg:
  • tests/app/invocations/test_video_concat_splice.py: bit-exact head across 3 chained extensions (video and audio); container duration matches the video; spliced soundtrack matches a full re-encode under every transition, seam included; fallbacks for foreign encoders, retimes, edit-list trims, rotation and short remainders.
  • tests/app/invocations/test_video_frame_extract_range.py: whole-range copy; trims with bit-exact copied runs; short encoded ends keep the frame rate; open-GOP sources; the source's tags aren't carried over.
  • tests/app/util/test_h264_splice.py: the layout gates.
• I mutation-checked each guard: removing it, or setting MIN_ENCODED_FRAMES or the pre-roll to 1, fails its test.
• Browser playback, headless Chromium 153 (Playwright; canPlayType reports probably for H.264 + AAC). Files tested, all passed through the production metadata remux: a 5-round spliced extension chain, main's 5-round chain as a control, and a trim of the spliced chain that itself spliced. Each played to the end with no media errors, every frame decoded, 0 corrupted frames, and the full audio track decoded. Seeking to every frame and comparing an 8×8 block signature with ffmpeg's decode, every frame matched its own index (difference 0.025, against 0.77 for a neighbour), so there were no misplaced frames at any seam.
• Benchmark: the Extend graph, 832×480 81-frame clips at 16 fps with audio, 5 rounds. Per round, main took 1.56–2.63 s wall and 3.1–8.6 s of ffmpeg CPU; this branch took 1.45–1.66 s and 2.4–2.9 s. On main the first segment fell from 37.5 to 26.3 dB (vs. as-generated) over the 5 rounds; on this branch it held at 48.2 dB, with bit-identical audio.
• Manual check in the real app:
  1. Generate a Wan or LTX-2 clip.
  2. Extend it three or four times, using each result as the next initial video.
  3. Scrub the first segment: it should look identical to the original generation, with no added blocking or softening, and the audio should have no clicks at the joins.

Review

Material findings, all resolved:

• Edit-list trims (e.g. ffmpeg -c copy / LosslessCut) listed more samples than they decode, so a splice made the node fail. Such tracks are now refused.
• A rotation stored as metadata on a square clip would have been applied to the copied frames but not the re-encoded ones. Such tracks are now refused.
• 1–2 frame encoded parts have a shorter B-frame delay, which distorted the joined timestamps (wrong probed fps, and the next Extend was refused). Encoded parts are now kept at 0 or at least 3 frames, and the joined timeline is checked.
• Open-GOP I-frames are sync samples but not clean cut points. Cuts are now made only at IDR pictures, checked by reading the NAL type.
• The audio seam glitched with one frame of encoder pre-roll and a cut too close to the transition. It now uses two frames of pre-roll plus a two-frame margin, and the seam is checked at runtime against an ordinary encode of the same samples. An earlier version of this check compared against the following audio, and rejected every clean seam before a silent continuation (e.g. a Wan clip).
• Smaller fixes:
  • The avcC comparison now runs before the long encode, avoiding a wasted second encode.
  • The whole-range copy strips the source's tags.
  • Decoders are closed promptly so they don't hold the process-wide decode slot.
  • The version-1 mvhd offset is fixed.
  • Truncated-table bounds checks are added.
  • Failed probes aren't cached.

From maintainer review, all resolved:

• Allocation and CPU bounds: sample-table counts from uploads are bounded before anything is allocated (a crafted stsz with no stss could exhaust memory). Sample-entry box lists are bounded too: concatenating them was quadratic, and listing the mp4a children amplified memory.
• Whole sample entry compared: eligibility compares the full sample entry, not just avcC: colr/pasp/clap etc., with the encoder-version name masked. The copied head and the joined file must keep it, which catches a pasp ffmpeg derives from a tkhd display size.
• Cancellation: splice ffmpeg children are killed and reaped on cancellation, and both nodes re-check cancellation before saving.
• Probe check: every spliced result is probed for fps and duration.

From the second maintainer review, all resolved:

• Byte-size bounds: sample entries over 64 KiB (avc1) or 4 KiB (mp4a) are refused before any of their bytes are read, so a huge free box or a padded avcC/esds is never loaded into memory.
• Bounded walks: every container walk (moov, trak, mdia, minf, stbl, edts) stops after 64 child boxes, and avcC is found within the bounded child list. A box flood therefore can't hold a worker for minutes. Measured before the fix: ~240 s for a 1 GiB flood.
• Cancellable decodes: one cancellable ffmpeg runner (video_audio.run_ffmpeg_process) now backs the splice copies, the audio decodes (including the splice's validation decodes), the audio mux, and the whole-range copy's tag remux.

From the third maintainer review, all resolved:

• Whole-range copy matches a trim's soundtrack: the copy is taken only when no stream plays past the last frame (probed duration, end trims applied), and it carries only the first audio track. Audio that outlasts the video or starts late now goes through the ordinary path, which still stream-copies the video and cuts the audio to the range.
• macOS CI: the RLIMIT_AS allocation test runs only on Linux, which enforces that limit.

Remaining limitations:

• One join's seam window is a few AAC frames (~0.1 s at 48 kHz). There the copied and new audio overlap, and error is comparable to one AAC generation.
• Across 3 sample rates × 4 kinds of content × silent and audible continuations × 3 transitions, every join spliced with a clean seam. The runtime seam check is a coarse safety net, though: it compares RMS over the two-frame window. With the pre-roll deliberately set to 1 it rejected most glitchy seams, but not all, so its correctness rests mainly on the pre-roll and margin.
• The bounds checks for truncated boxes are not exercised by a crafted file.
• Uploads and pre-existing videos are re-encoded once, on their first extension.

Compatibility / Rollout

No persisted-state or API changes.

• Larger files: CRF 18 makes new video files larger than CRF 25 did.
• Older videos: videos made by earlier versions don't match the new encoder settings, so their first extension re-encodes them once, as before. Extensions after that are spliced. Older soundtracks were encoded with PNS, so a trim of one that starts mid-clip usually re-encodes its audio once, too.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Meaningful regression coverage added / updated where needed; obsolete tests/code removed
• Persisted-state and API changes include required migrations / compatibility validation
• Relevant performance/efficiency opportunities considered; material claims have evidence
• Material review findings resolved and relevant checks rerun
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[JPPhoto]

First, a high-level overview of what I think the aims should be and where this PR stands:

This is a useful step toward stopping quality loss during repeated extensions, but it solves a narrower problem.

Aim	PR 9833
Copy safe regions and encode only transition bridges across the whole timeline	Partial. It copies only a prefix of the first clip. It re-encodes that clip’s tail and every later clip. It also decodes the copied prefix to skip it.
Preserve existing cuts, fades, frame counts, and audio behavior	Mostly. The legacy timeline and blending path remains, and the new tests cover frame counts and audio seams.
Safely decide which media can be spliced	Partial. It checks IDR boundaries, codec configuration, edit lists, rotation, and sample counts. It does not check all relevant timing and container metadata, such as PTS/DTS cadence, color metadata, and sample aspect ratio.
Fall back safely when splicing is unsupported or uncertain	Partial. Several cases fall back to full encoding, but the sample-table parser can allocate based on an unchecked input count, and cancellation cannot stop the new splice subprocesses promptly.
Validate browser playback and demonstrate smart-render performance	No. The PR says browser playback was not checked. It reports quality comparisons, but not a representative speed or memory benchmark for the broader smart-render approach.

So it meets part of the quality-preservation goal for chained extensions. It does not yet meet the broader goal of planning a timeline across multiple clips, copying every safe region, and encoding only the transition bridges.

Next, the code review...

Merge blockers:

• invokeai/app/util/h264_splice.py:317: When an MP4 has no stss box, the parser builds tuple(range(sample_count)) from the input’s 32-bit stsz count before applying _MAX_SAMPLES. A crafted, probeable H.264 MP4 can trigger a huge allocation in the application process. H.264 MP4 uploads can reach storage without transcoding (invokeai/app/api/routers/videos.py:494-503), and concat calls this parser at invokeai/app/invocations/video_concat.py:323. Effect: memory exhaustion and possible process termination. Likelihood: crafted malformed media; ordinary clips are unaffected. Recovery: restart the application. Test: pass an oversized stsz count with no stss through concat preflight in a bounded subprocess; verify rejection before a count-sized allocation.

Other findings/issues:

• invokeai/app/invocations/video_concat.py:327: Splice eligibility compares avcC but ignores container color and pixel-aspect metadata. MP4 colr and pasp properties are separate from the codec configuration (FFmpeg MP4 muxer, sample aspect handling). A remuxed input can retain the same avcC while adding non-default metadata; the copied head can then supply track metadata for the re-encoded continuation. Effect: changed color or aspect in the output. Likelihood: plausible for remuxed or uploaded media; normal app output is unaffected. Recovery: reject mismatched metadata and use the full-encode path. Test: remux an eligible source with colr or pasp, confirm avcC is unchanged, and compare the joined continuation with a forced full encode.

• invokeai/app/util/h264_splice.py:349: Splice subprocesses block in subprocess.run(timeout=600) without cancellation. Cancellation during video copy or concat waits for the process; during AAC splicing, cancellation can also go unchecked through subsequent stages and reach save. Effect: canceled work can keep a worker occupied for minutes and may save an unwanted output. Likelihood: plausible for long clips or slow storage. Recovery: wait for completion or timeout; delete any output saved after cancellation. Test: cancel during video and AAC splice subprocesses; verify prompt child termination and reaping, no later stages or save, and temporary-file cleanup.

Suggestions:

• Consider treating this as an incremental extension optimization. It copies only the first clip’s prefix; _encode_joined() decodes and discards that prefix, then encodes the first clip’s tail and every later clip. The broader smart-render outcome needs safe copy spans planned across clips, with encoding limited to transition bridges.

[lstein]

Thanks for the thorough review. All three findings are confirmed and fixed in 9a607cc, each with a regression test that fails against the previous code.

Blocker: unbounded allocation. Confirmed: an upload that is already an H.264/AAC MP4 is stored as-is, so the sample tables are the uploader's. A file claiming ~4 billion samples with no stss dies with MemoryError at the line you named.

• Bounds before allocation: the sample count is bounded before anything is sized by it. It must be ≤ 2²⁰, the size table must actually hold the entries, and the samples must fit in the file.
• Sync samples only: byte offsets are computed only for the sync samples, from one cumulative-size array, and each must lie inside the file.
• Keyframe cap: keyframes examined per track are capped at 2¹⁴, and a track with no stss above that is refused.
• Bounded entry reads: the follow-up review found two more amplifiers in code this fix had added: concatenating a sample entry's child boxes was quadratic, and the mp4a children were materialised in full. Both entries are now read with a bounded box count.
• Tests: the crafted file is parsed in a child process capped at 4 GiB of address space (size-table and uniform-size variants), and tracemalloc checks a sample entry stuffed with 200k boxes.

Colour / pixel aspect. Confirmed: remuxing one of our outputs with a BT.2020/PQ colr, or a 16:9 pasp, leaves avcC byte-identical.

• What is compared now: the sample entry's fixed fields (with the encoder-version compressor name masked) and every child box except btrt, length-prefixed. They're compared against a one-frame probe of the app's own encoder, so colr/pasp/clap etc. fall back to a full encode.
• The copies are checked too: the copied head (or middle) and the joined file must keep the source's exact sample entry. ffmpeg can derive presentation boxes while copying (a tkhd display size that disagrees with the coded size becomes a pasp), and that would otherwise stretch the new clip.
• Audio: mp4a entries with anything besides esds/btrt are refused.

Cancellation.

• run_ffmpeg: it checks for cancellation before spawning, then polls the child every 0.1 s, killing and reaping it on cancellation or timeout. It's threaded through every copy, join and audio-splice call.
• Before save: both nodes check for cancellation immediately before save().
• Tests: they assert that the stage itself raises CanceledException (concat video copy, concat audio splice, trim copy), and that a running ffmpeg is killed promptly. All four fail if run_ffmpeg ignores cancellation.
• Still uncancellable (pre-existing): audio extraction, the full-encode audio mux, and the metadata remux. They can no longer lead to a save after cancellation.

Timing. Every spliced result is also probed: its fps and duration must match the expected timeline. This is in addition to the existing checks (uniform frame duration, sample counts, IDR-only cuts).

Browser playback. Checked in headless Chromium 153 (Playwright), which reports probably for avc1.64001f, mp4a.40.2. Files tested, all passed through the production metadata remux: a 5-round spliced extension chain, the same chain on main as a control, and a trim of the spliced chain that itself spliced.

• Playback: each played to the end with no media errors, every frame decoded (476/476, 254/254), 0 corrupted frames, and the full audio track decoded.
• Frame identity: seeking to every frame and comparing an 8×8 block signature against ffmpeg's decode, every frame matched its own index. Signature difference was 0.025 against 0.77 for the neighbouring frame, so there were no misplaced frames at any seam.

Benchmark. Extend graph (extract_video_range of the whole source → video_concat, crossfade 2), 832×480, 81-frame clips at 16 fps with audio, 5 rounds:

round	main: wall / ffmpeg CPU	main: 1st segment vs as-generated	PR: wall / ffmpeg CPU	PR: 1st segment vs as-generated
1	1.56 s / 3.06 s	37.5 dB	1.45 s / 2.37 s	48.2 dB
3	2.08 s / 5.58 s	30.3 dB	1.53 s / 2.71 s	48.2 dB
5	2.63 s / 8.56 s	26.3 dB	1.66 s / 2.89 s	48.2 dB

On this branch the first segment's audio is bit-identical every round; on main it drifts. The PR's remaining per-round growth is the copied prefix still being decoded to skip it, which is your scope point.

Scope. Agreed, this is an incremental extension optimisation. It fixes the reported compounding loss in the earliest segments: afterwards each segment takes one re-encode, as the new clip, and is then carried forward. The timeline-wide version will be a separate follow-up PR. It plans copy spans for every clip (for Extend, the new clip after its first IDR past the transition, so it never loses a generation either) and seeks past copied prefixes rather than decoding them. #9834, stacked on this, already applies the same splice to trimmed sources' audio.

[JPPhoto]

Merge blockers:

• invokeai/app/util/aac_splice.py:168 (same issue in invokeai/app/util/h264_splice.py:347 and :362): The splice readers load uploader controlled esds, avcC, and sample entry boxes into memory without a byte limit. A large free child in an H.264 entry is copied into multiple byte strings; the AAC reader also reads its full esds payload. Accepted uploads can be up to 1 GiB. Effect: a crafted first clip can exhaust backend memory and potentially terminate the server. Likelihood: requires a crafted uploaded MP4. Recovery: restart the server if the process is killed. Test: use oversized free and padded esds boxes; confirm the parser refuses them without payload sized allocation.

Other findings/issues:

• invokeai/app/util/h264_splice.py:343: find_box() scans for avcC before the later 33 box limit is applied. Putting avcC after many small boxes forces the parser to scan them all before rejecting the entry. Effect: a crafted upload can hold the invocation worker in an uncancellable scan. Likelihood: unusual, malformed sample entries. Recovery: wait for the scan to finish. Test: put a box flood before avcC and confirm scanning stops at a fixed bound. This is unusual and probably can be ignored - your call.

• invokeai/app/util/aac_splice.py:342 and invokeai/app/invocations/video_concat.py:496: The new AAC validation decodes call extract_audio_pcm(), which runs ffmpeg with a 600 second timeout and has no cancellation callback. Canceling during either decode leaves the child running until it finishes or times out. Effect: the canceled job continues occupying a worker; the save guard prevents output from being saved. Likelihood: plausible with long audio inputs. Test: cancel after each validation decode starts; confirm the child is killed and reaped promptly and nothing is saved.

[lstein]

Thanks — all three are fixed in ca9837f, each with a regression test that fails against the previous code. #9834, which was stacked on this branch, has been merged into it, so the trim-audio splice is now part of this PR. The description is updated to cover it.

Blocker: unbounded box reads.

• Size cap before any read: a sample entry is refused by its header size before any of its bytes are read: over 64 KiB for avc1, over 4 KiB for mp4a. Every child box (avcC, esds, a free box…) lies inside the entry, so nothing in it is read beyond those caps. This app's own entries are 130–134 bytes for video at every resolution tried, and under 100 bytes for audio.
• Tests: a 16 MiB free child, a 16 MiB-padded avcC, and a 16 MiB-padded esds are each refused with tracemalloc peak under 1 MiB.

avcC scan.

• Entry level: avcC is now looked up within the bounded (≤32) child list.
• Container level: the follow-up review found the same pattern one level up, so I fixed that too. Every container walk (moov, trak, mdia, minf, stbl, edts) now stops after 64 child boxes. A flood at the front of stbl previously cost each parse roughly 240 s per GiB, since it was walked once per table looked up. It's now refused after a fixed look.
• Tests: a 2M-box flood before avcC, and a 200k-box flood at the front of stbl (video and audio), each parse with a bounded number of seeks.

Validation decodes.

• One cancellable runner: video_audio.run_ffmpeg_process polls the child, and kills and reaps it on cancellation or timeout. It now backs extract_audio_pcm, mux_audio_into_video, the splice copies, and the whole-range copy's tag remux (write_mp4_tags).
• Wired through: is_canceled reaches every decode and mux on the splice and save paths: the inputs' audio extraction, both AAC validation decodes, and the full-encode mux.
• Tests: the cancellation test is parametrized over the video copy, the audio splice, the input audio decode and the splice's validation decode. Each asserts that the stage itself raises CanceledException, and nothing is saved.
• Smaller changes: run_ffmpeg_process keeps only the last 8 KiB of stderr, and the source ADTS copy stops at the last packet the splice uses.

[JPPhoto]

The earlier findings about oversized parser reads, unbounded box walks, and uncancellable validation decodes are fixed!

Merge blockers:

• macOS Python tests fail. In test_h264_splice.py:140, the subprocess unconditionally sets RLIMIT_AS to 4 GiB. On the macOS runner its hard limit is lower, so both parameterizations fail with ValueError: current limit exceeds maximum limit, before the parser is tested. The Python test run reports 13,680 passed and 2 failed. Set a limit supported by the runner or handle this platform's limit.

Other findings/issues:

• P2 - Full-range copy can preserve audio past the video's duration. video_frame_extract_range.py:235 bypasses audio rebuilding for a whole-range copy. _save_unchanged remuxes every audio stream unchanged (mp4_metadata.py:254), while reporting duration from the video frame count. A compatible H.264 file with 5 seconds of video and 10 seconds of AAC audio therefore keeps the extra audio, unlike the normal path, which trims audio to the selected frame range. Gate the copy on matching audio duration or trim the copied soundtrack; add coverage for longer source audio.

[lstein]

Thanks. Both findings are confirmed and fixed in 4b52ab1.

macOS Python tests. The macOS runner's hard RLIMIT_AS is below 4 GiB, and macOS doesn't enforce that limit on allocations anyway, so the subprocess test proves nothing there. It now runs only on Linux, where the CI Linux jobs still exercise it.

Full-range copy keeping audio past the video. Confirmed: 2.5 s of H.264 with 5 s of AAC came out with all 5 s.

• The gate: the whole-file copy is taken only when the probed duration, which is the longest stream's with end trims applied, is within half a frame (+10 ms probe rounding) of the video's. Otherwise the ordinary path runs: it still stream-copies the video (the keyframe-aligned run covers every frame) and cuts the audio to the range.
• Also covered: audio that starts late and so ends past the last frame (1 s delay → +0.94 s), and extra audio tracks. The copy now carries only the first audio track, as the ordinary path does.
• Why not decoded length: a first attempt compared the decoded audio length. The bundled ffmpeg doesn't apply the AAC end trim when decoding, so this app's own outputs decode up to a packet long, and about a third of MiniMax H3 clip lengths (24 fps / 32 kHz) lost the copy. The probed duration of app outputs is within ±5 ms across 8–120 fps and 16–48 kHz.
• Tests: longer audio, delayed audio and a second audio track each check that nothing plays past the last frame, that there is one audio track, and that the video is still copied. App outputs at H3 and 16 kHz shapes still get the bit-exact whole-file copy. Each fails against no gate, the decoded-length gate, or carrying every track.

[JPPhoto]

@lstein I am making a test for the following issue, expect it shortly:

• invokeai/app/invocations/video_frame_extract_range.py:241: If imageio probing fails and the supported OpenCV fallback runs, _probe() sets duration from video frame count and FPS alone (invokeai/app/util/video_decode_worker.py:163). A full-range H.264 clip with longer AAC audio then passes the gate, and _save_unchanged() preserves the extra audio. Effect: audio outlasts the output video and its reported duration. Likelihood: uncommon; requires OpenCV to open a spliceable clip after imageio probing fails. Recovery: manually trim or re-encode the output. Test: force the fallback with an H.264/AAC fixture whose audio outlasts its video; verify full-range extraction cuts audio at the video end.

[JPPhoto]

Good to merge.