fix(media): break OutboundClockSync Arc cycle leaking ~70 KB/call - #266
Open
ftong2010 wants to merge 1 commit into
Open
fix(media): break OutboundClockSync Arc cycle leaking ~70 KB/call#266ftong2010 wants to merge 1 commit into
ftong2010 wants to merge 1 commit into
Conversation
The OutboundClockSync RTP observer was registered on the PeerConnection holding a strong 'pc', and later a strong 'RtpSender'. Because the observer is also attached to the RTP transport (attach_registered_observers) and the sender holds that same transport, this forms the cycle: transport.observers -> OutboundClockSync -> sender -> sender.transport which reference counting never frees, retaining the RtpTransport + its IngressTap observer (~70 KB) per call. Introduced by 5f4a310. Hold a Weak<RtpSender> instead and upgrade on use.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Under sustained load (50 cps),
rustpbxmemory grows without bound — roughly400–780 MB/min, reaching the 8 GiB container limit in ~10 minutes. About
70 KB per call is allocated and never freed.
Before fix

After fix

Root cause
Memory Profiling point to session building process strongly. With step by step metrics collection and analyzing, root cause is identified
Commit
5f4a3106("fix(media): stop WebRTC Opus IVR pops and harden egresstimeline", 2026-08-21, jinti) added the
OutboundClockSyncRTP observer andregistered it on the
PeerConnectionwith a strongpcreference:That alone keeps the
PeerConnectionalive. Even after replacingpcwiththe
RtpSender, the observer is also attached to the RTP transport(
attach_registered_observers), and the sender holds that same transport:Rust reference counting never collects either cycle, so the
RtpTransportand its
IngressTapobserver (also registered on the transport) are retainedforever.
Fix
Hold a
Weak<RtpSender>inOutboundClockSyncinstead of a strongPeerConnection/RtpSender, upgrading on use inon_egress. AWeakdoesnot keep the sender alive, so once the
PeerConnectionInnerdrops, the senderdrops, the transport's last strong ref goes, and the transport + tap free.
Verification
RtpTransport/IngressTaplifecycle counters: before the fix3600 created / 0 dropped; after, 3600 / 3600.
jemallocstats.allocatedstays flat across repeated 1,800-call runs(~62 MB) instead of growing ~70 KB/call.
~470 MiB (was unbounded).