Skip to content

fix(media): break OutboundClockSync Arc cycle leaking ~70 KB/call - #266

Open
ftong2010 wants to merge 1 commit into
restsend:mainfrom
ftong2010:fix/media-outbound-clock-cycle
Open

fix(media): break OutboundClockSync Arc cycle leaking ~70 KB/call#266
ftong2010 wants to merge 1 commit into
restsend:mainfrom
ftong2010:fix/media-outbound-clock-cycle

Conversation

@ftong2010

@ftong2010 ftong2010 commented Sep 1, 2026

Copy link
Copy Markdown

Problem

Under sustained load (50 cps), rustpbx memory grows without bound — roughly
400–780 MB/min, reaching the 8 GiB container limit in ~10 minutes. About
70 KB per call is allocated and never freed.

Before fix
99d66539620c6e0d1a0ffd1ed88bef93

After fix
afd1dcd70d26993f034b1abe42da7f67

Root cause

image

Memory Profiling point to session building process strongly. With step by step metrics collection and analyzing, root cause is identified

Commit 5f4a3106 ("fix(media): stop WebRTC Opus IVR pops and harden egress
timeline", 2026-08-21, jinti) added the OutboundClockSync RTP observer and
registered it on the PeerConnection with a strong pc reference:

pc.rtp_observers → OutboundClockSync → pc      (cycle #1)

That alone keeps the PeerConnection alive. Even after replacing pc with
the RtpSender, the observer is also attached to the RTP transport
(attach_registered_observers), and the sender holds that same transport:

transport.observers → OutboundClockSync → sender → sender.transport = transport   (cycle #2)

Rust reference counting never collects either cycle, so the RtpTransport
and its IngressTap observer (also registered on the transport) are retained
forever.

Fix

Hold a Weak<RtpSender> in OutboundClockSync instead of a strong
PeerConnection/RtpSender, upgrading on use in on_egress. A Weak does
not keep the sender alive, so once the PeerConnectionInner drops, the sender
drops, the transport's last strong ref goes, and the transport + tap free.

Verification

  • Instrumented RtpTransport/IngressTap lifecycle counters: before the fix
    3600 created / 0 dropped; after, 3600 / 3600.
  • jemalloc stats.allocated stays flat across repeated 1,800-call runs
    (~62 MB) instead of growing ~70 KB/call.
  • 50 cps × 10 min with recording + CDR + RWI enabled: memory plateaus at
    ~470 MiB (was unbounded).

The OutboundClockSync RTP observer was registered on the PeerConnection
holding a strong 'pc', and later a strong 'RtpSender'. Because the observer
is also attached to the RTP transport (attach_registered_observers) and the
sender holds that same transport, this forms the cycle:

  transport.observers -> OutboundClockSync -> sender -> sender.transport

which reference counting never frees, retaining the RtpTransport + its
IngressTap observer (~70 KB) per call. Introduced by 5f4a310.

Hold a Weak<RtpSender> instead and upgrade on use.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant