Skip to content

Renderer: dispose live frames before re-alloc on source switch#704

Open
lrohrmann wants to merge 1 commit into
SuRGeoNix:masterfrom
lrohrmann:fix/renderer-dispose-frames-on-source-switch
Open

Renderer: dispose live frames before re-alloc on source switch#704
lrohrmann wants to merge 1 commit into
SuRGeoNix:masterfrom
lrohrmann:fix/renderer-dispose-frames-on-source-switch

Conversation

@lrohrmann

Copy link
Copy Markdown
Contributor

Summary

Re-opening a source on a Player (i.e. switching streams via Open/OpenAsync
without a full Stop()/renderer reset) leaks native GPU memory. The process's
committed memory grows on every switch and is only released when the device is
destroyed (Stop/dispose). On integrated Intel GPUs (where "VRAM" is system RAM) this
ratchets up until the machine runs out of commit charge.

Root cause

A VideoFrame can leave the VideoCache and never be Dispose()d — e.g. the frame
held by the renderer/screamer at the moment of a source switch, seek, or hand-off.
VideoFrame has no finalizer, so the native objects it owns — the D3D11
Texture/ShaderResourceView/VideoProcessorInputView and the pinned AVFrame
are released only when the COM wrappers are finalized by the GC.

Because the managed heap stays tiny during playback, Gen2 collections almost never
run, so those finalizers almost never run. Each switch therefore leaves the previous
source's frames (and their surfaces) committed while the new decoder allocates its
own → the concurrent peak is old + new, and committed memory never recovers.

A heap dump of a leaking process shows thousands of VideoFrame and
ID3D11VideoProcessorInputView instances that are unrooted (no GC root) but whose
native resources were never freed, while only a handful of ID3D11Texture2D remain
(HW frames share one texture array) — i.e. the managed wrappers are dead garbage and
the native memory leaked.

Reproduction (current master)

  1. Take any sample that drives a Player (e.g. the FlyleafPlayer sample).
  2. Repeatedly switch the source on the same player without calling Stop()
    between switches — e.g. alternate player.OpenAsync(A) / player.OpenAsync(B)
    on a ~1s timer, or loop a playlist of a few items.
  3. Watch the process's Commit size (Task Manager → add the "Commit size" column,
    or VMMap "Private Committed"). Do not rely on the Working Set — it is trimmed
    by the OS and hides the leak.
  4. Committed memory rises by roughly one source's worth of surfaces per switch and
    never drops while the player keeps running.
    • Most visible with large software-decoded frames (e.g. big MJPEG/JPEG stills,
      several megapixels) and on Intel integrated GPUs; hardware-decoded streams leak
      more slowly but still ratchet up.
    • Calling Stop() (full renderer reset) before each Open releases the memory —
      confirming the leak is specific to the in-place re-open path.

Fix

Track every frame the renderer hands out through the single FillPlanes choke point,
and dispose any survivors deterministically:

  • in Player.OpenInternal, before the new decoder allocates (so the freed
    surfaces are reused instead of accumulating), and
  • on renderer teardown (DisposeLocal).

A Flush() after disposal lets D3D11 actually destroy the released resources (there
is no Present on a paused/just-switched player to trigger deferred destruction).

Changes

  • Renderer.Device.cs: per-renderer live-frame registry (TrackFrame/UntrackFrame/
    DisposeLiveFrames) + FlushContext(); DisposeLiveFrames() is also called from
    DisposeLocal.
  • Renderer.VP.cs: FillPlanes becomes a small wrapper around the (now private)
    fillPlanes delegate that records each returned frame.
  • VideoFrame: gains an Owner back-reference and untracks itself on Dispose().
  • Player.Open.cs: dispose cached + escaped frames and flush before decoder.Open(...).

No public API change (the affected members are internal). The per-frame tracking is
a HashSet add/remove guarded by a lock — negligible overhead on the frame path.

Testing

Verified on an Intel integrated GPU (committed-memory growth is most pronounced there)
with a setup of several concurrent Players, repeatedly switching one player's source
between an H.264 RTSP stream (hardware-decoded) and large multi-megapixel MJPEG stills
(software-decoded).

  • Committed memory (Task Manager "Commit size" / VMMap "Private Committed", not
    Working Set): before the fix it rose by roughly one source's worth of surfaces on
    every switch and never recovered (~tens of MB per switch with the large SW frames,
    unbounded over ~30 switches). After the fix it stays bounded — it plateaus at roughly
    a single source's peak instead of ratcheting up.
  • Heap dump cross-check (WinDbg + SOS): with the fix, !dumpheap -stat -type VideoFrame and -type ID3D11VideoProcessorInputView no longer show thousands of
    accumulating instances after repeated switches (they stay at a small steady count),
    and !gcroot on a sampled survivor confirms old frames are actually disposed rather
    than left as unrooted objects whose native resources were never released.
  • Both decode paths: confirmed for the HW (D3D11 VideoProcessor) and SW frame
    paths; the regression was reproducible and is resolved on both.
  • No rendering/playback regression: source switching, seeking and normal playback
    behave as before; no black frames or flicker introduced by the deterministic disposal
    • flush.

Re-opening a source without a full Stop()/Renderer reset left the previous source's decoded frames alive. Frames that escape the VideoCache (switch/seek/screamer hand-off) were freed only by non-deterministic COM finalizers, keeping the D3D11 device referenced so the driver never reclaimed the native surface memory -> committed memory grew unbounded per switch (observed on Intel iGPU).

Track every frame the renderer returns and dispose survivors deterministically in OpenInternal (before the new decoder allocates) and on teardown, then Flush so D3D11 destroys them.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant