Add custom video encoder factory support via SDK-owned protocols - #1109
Open
hiroshihorie wants to merge 12 commits into
Open
Add custom video encoder factory support via SDK-owned protocols#1109hiroshihorie wants to merge 12 commits into
hiroshihorie wants to merge 12 commits into
Conversation
Adds LiveKitSDK.set(videoEncoderFactory:) which accepts an implementation of the new public VideoEncoderFactory and VideoEncoder protocols instead of exposing LKRTCVideoEncoderFactory, keeping LiveKitWebRTC out of the public API surface. New public types mirror the WebRTC encoder interface: VideoCodecInfo, VideoEncoderSettings, VideoEncoderQpThresholds, EncodedVideoFrame and VideoEncoderStatus. Internal adapters bridge them to the RTC types, and the custom factory is wrapped in the simulcast adapter the same way the default factory is. Known gaps: the optional factory surface (encoderSelector, queryCodecSupport, implementations) is not bridged, and H265 codec-specific info falls back to generic on macOS because the prebuilt macOS slice does not ship RTCCodecSpecificInfoH265.h.
WebRTC 150.7871.01 ships the H265 codec specific info header in the macOS slice, so the generic fallback is no longer needed.
WebRTC assigns each frame an RTP timestamp in the 90kHz clock before it hands the frame to an encoder, and an encoded frame has to carry that same value back. VideoFrame only exposed the capture time in nanoseconds, so a custom encoder had no way to read it. Add rtpTimestamp to VideoFrame, populate it when converting from the WebRTC type, and write it back when converting to the WebRTC type so a frame that passes through a VideoProcessor keeps it. The existing initializer stays as is and defaults the value to 0.
Frames arrive as NV12 CVPixelBuffers and I420VideoBuffer has no public initializer, so an encoder that needs planar data had no way to get it. Add toI420() on VideoBuffer and VideoFrame, which copies the pixel data when the buffer is not already I420. I420VideoBuffer now holds the WebRTC I420 buffer protocol rather than its concrete class, since that is what the conversion returns, and gains width and height so the planes can be read without consulting the frame. Document the video buffer types and their accessors, which were public but undocumented.
The adapter handed the encoder a new closure on every setCallback call and captured the WebRTC block directly, so a cleared callback could still be invoked from an encoder thread. The block now lives in a small locked box, the encoder gets one stable forwarding closure, and releasing the encoder clears the box first. Frame types were built with compactMap, which silently shortened the array and misaligned the per stream requests. They are mapped one to one now, with anything unknown treated as a delta. A nil qp landed on the native side as 0, which reads as a perfect quantizer to the quality scaler, so it is now sent as -1 for unknown. The generic codec info object is allocated once instead of per frame, and supported codecs are computed once at factory construction. A custom factory is now paired with the built in encoders as the simulcast fallback, so an encoder reporting fallbackSoftware keeps the stream alive. A frame with a buffer the SDK cannot map reports that same status rather than an invalid parameter, which would drop every frame.
Fill in the pieces an encoder implementation needs and document the public types that were missing docstrings. VideoEncoderSettings gains a memberwise initializer so an encoder can be driven from a test. EncodedVideoFrame can now report the encode start and finish times and an NTP timestamp, which feed the encode time stats. VideoEncoderStatus gains the two remaining WebRTC codes and prints a readable name. VideoCodecInfo can resolve its SDP name to a VideoCodec. Setting a factory that advertises no codecs is now rejected up front rather than leaving video unpublishable, and the warning on the setter explains that any use of the SDK initializes the peer connection factory.
WebRTC only consults supports_native_handle when deciding whether to crop or scale a native frame for a simulcast layer. It never converts frames to I420 before handing them to the encoder, so the docstring now points encoders at toI420() instead.
WebRTC's frame encode metadata writer overwrites encode start and finish times, NTP time and the timing flags for every encoded image it matches to a source frame, and marks timing invalid otherwise. Values set by a custom encoder never reach stats or the wire, so the fields are removed rather than shipped as inert API.
The RTP packetizer for H264, VP8 and VP9 reads a typed video header that only exists when the encoded image carries matching codec specific info. A generic info object leaves the header empty and the packetizer aborts on the first frame. The adapter now synthesizes non interleaved H264 or H265 info from the codec the encoder was created for when a frame has none, and set(videoEncoderFactory:) rejects factories that advertise VP8 or VP9 since their layer info cannot be bridged yet. Also documents that a custom factory only takes over the codecs it lists, since the simulcast factory keeps advertising the built in ones.
WEBRTC_VIDEO_CODEC_TARGET_BITRATE_OVERSHOOT is 5, not -14. Adds the positive NO_OUTPUT and OK_REQUEST_KEYFRAME codes an encode call may return. Documents that the bridge always reports an alignment of 1 and hardware acceleration, and that startEncode may run before setCallback.
The underlying conversion only handles NV12, 32BGRA and 32ARGB. Other formats hit a debug assertion and return undefined pixel data in release builds, so toI420() now returns nil for them as its docstring promises.
hiroshihorie
marked this pull request as ready for review
September 4, 2026 12:19
hiroshihorie
requested review from
pblazej and
xianshijing-lk
as code owners
September 4, 2026 12:19
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
The adapter forwarded whatever codec specific info an encoder returned. An H264 encoder returning H265 info left the RTP video header empty and the H264 packetizer aborted, the same path as the generic info case. The info is now always built for the codec the encoder was created for, taking only the packetization mode from the frame. The nil default also hardcoded non interleaved packetization. It now follows the packetization-mode parameter of the negotiated codec, so a factory that advertises mode 0 gets single NAL unit packetization.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
LiveKitSDK.set(videoEncoderFactory:)with SDK-ownedVideoEncoderFactory/VideoEncoderprotocols, so apps can supply their own encoder (e.g. a software H264 encoder) withoutLiveKitWebRTCtypes in the public API. Supersedes #1082, thanks @sergeyphi for the PR and the detailed testing notes.Addresses the requests from that thread:
VideoFrame.rtpTimestampcarries the 90kHz RTP timestamp WebRTC assigns beforeencode, andEncodedVideoFrame.rtpTimestampmust be copied from it.VideoBuffer.toI420()/VideoFrame.toI420()give access to I420 planes for NV12 camera frames.supportsNativeHandleonly gates crop/scale in WebRTC, so this is the route to planar data.VideoEncoderSettingshas a public init for testing.FrameEncodeMetadataWriteroverwrites them for every frame it matches by RTP timestamp, so encoder-provided values never reach stats or the wire.Behavior notes, verified against the WebRTC source:
settime since their layer info cannot be bridged through the ObjCRTCCodecSpecificInfotypes, and a generic info object makes the RTP packetizer abort.codecSpecificInfofollow the negotiatedpacketization-modeparameter, non interleaved when absent..fallbackSoftware.resolutionAlignmentis not honored yet due to a bug in the ObjC bridge (requested_resolution_alignment = resolutionAlignment > 0 ?: 1), and the bridge reports every encoder as hardware accelerated. Both documented, fork fixes to follow separately.Swift only for now; ObjC encoders need a thin Swift shim. Compile verified on macOS and iOS Simulator, no runtime test yet.