Resolve a default audio session from engine state when no policy was pushed - #1182
Merged
Conversation
…pushed On iOS the audio engine refuses to enable recording unless the audio session category permits input, and livekit_client owns that session. The native engine observer only applied a configuration that Dart had pushed, and the only push site in automatic mode was Room.connect. Any recording that started earlier (pre-connect audio, a pre-join microphone preview, an engine start driven from native before the Flutter side exists) ran against the app-default soloAmbient category and failed with kAudioEngineErrorAudioSessionInvalidCategory (-9001), reported as AudioProcessingException(applyFailed). The observer now resolves a built-in playAndRecord preset from engine state when nothing has been pushed and automatic management is on, matching the Swift SDK's AudioSessionEngineObserver, which derives the session from engine state alone. The Dart-pushed policy becomes an override rather than a prerequisite. Dart passes preferSpeakerOutput so the preset picks the same mode the Dart policy would. Audio device module results -9000, -9001 and -4100 now get their own error codes and surface as TrackCreateException or the new AudioSessionException instead of an audio processing failure.
…ping helpers AudioManager prefers speaker output by default, so the built-in preset now defaults to videoChat as well. Otherwise the connect-time push would switch the live session from voiceChat to videoChat. Also simplifies effectiveConfigurationLocked and the Dart error mapping, and shortens the changeset entries.
hiroshihorie
marked this pull request as ready for review
August 27, 2026 16:47
hiroshihorie
requested review from
cloudwebrtc and
xianshijing-lk
as code owners
August 27, 2026 16:47
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
cloudwebrtc
approved these changes
Aug 28, 2026
Contributor
|
I am able to confirm this also fixes the issue 👍 |
MaxHeimbrock
approved these changes
Aug 28, 2026
…fault preset RTCAudioSessionConfiguration.webRTC() returns the process-wide shared instance. Mutating it rewrote WebRTC's global default config on the first pre-connect engine start, leaked the preset into later partial policy pushes (which merge into that same object) and into flutter_webrtc's own session handling, and let the object escape the observer's lock while the platform thread could mutate it concurrently. Build the preset on a copy.
The built-in native preset only runs while no policy has been pushed (cachedConfiguration == nil), but the only writer of the native preferSpeakerOutput field was updatePolicy, which sets cachedConfiguration in the same locked section and is never cleared. The preset could therefore only ever observe the compiled-in default, and a non-default preference always arrives as a pushed policy whose mode already carries it. Drop the Dart parameter, channel key, Swift field and the tests asserting them, and hardcode the preset's videoChat mode with a comment explaining why.
…e audio doc AudioSessionException is a new exported public class; repo convention labels additive API with a minor changeset (the dart-apitool CI gate only catches breaking changes, not additions). Also update doc/audio.md, which still promised AudioProcessingException for capture-time setup failures that the start path now surfaces as TrackCreateException / AudioSessionException.
The built-in preset also fires for playout-only engine starts before any policy is pushed (applying a playback session where the observer was previously a no-op). The changeset only advertised the recording case, so integrators could not anticipate the playout-side behavior change.
…e start With nothing pushed, willEnableEngine previously always proceeded (the observer was a no-op). Applying the new default preset made it able to return -4100 and roll the engine start back when the session reconfigure was rejected, even where the app's own already-valid session would have sufficed. Tolerate apply failures for the preset (log and proceed; the ADM's own pre-enable checks still gate recording on an unsuitable category), and keep failing hard only for a policy the app actually pushed.
register(with:) built a fresh LKAudioEngineObserver every time, so a second Flutter engine registering in the same process (add-to-app, FlutterEngineGroup) silently reset the pushed policy, management mode and activation flag to their defaults. Before the built-in preset that reset was a silent no-op; now it would make the next engine lifecycle event apply the preset and activate a session the app may own. Share one observer process-wide (matching the process-wide session it manages) and rebind only the notification channel per registration, guarding channel access with the existing lock since it is now mutable.
handleSetMicrophoneMuteMode and handleStopLocalRecording still hand-built generic FlutterErrors, so the same ADM failure surfaced differently depending on entry point: a mute-mode change can rebuild the engine and hit the same permission / audio session pre-enable checks (-9000/-9001) as a recording start. Route both sites through flutterError(forAudioEngineResult:), and map the codes in AudioManager.setMicrophoneMuteMode like setEngineAvailability does.
The native built-in preset duplicates the Dart default policy with only a keep-in-sync comment tying the two, so any change to the Dart automatic branch would silently diverge for pre-connect capture. Have LocalAudioTrack.startCapture push the resolved policy first (cache-only while the engine is idle in automatic mode), so Flutter-driven starts always use the real Dart policy and the native preset is reduced to a fallback for engine starts before the Flutter side exists (CallKit killed-state wake).
The codes that map to AudioSessionException cannot occur on macOS: the invalid-category check (-9001) is compiled out under !TARGET_OS_OSX in the audio device module, and the configure failure (-4100) is only produced inside !os(macOS) blocks. Saying 'Apple platforms' over-promised for macOS integrators, where only the permission mapping (-9000) is live.
hiroshihorie
added a commit
that referenced
this pull request
Sep 1, 2026
Flutter counterpart of client-sdk-swift #1085, matching its final merged behavior. Builds on #1182. ## Why webrtc-sdk/webrtc#265 (first shipped in `m144.7559.12`) removed the blocking mic permission request from the AudioEngine device. The pre-enable check is now passive: it returns `kAudioEngineErrorInsufficientDevicePermission` (-9000) instead of prompting, so requesting permission is the SDK's job. flutter-webrtc still pins `144.7559.10`, so this is not load-bearing yet. It is harmless there, since `getUserMedia` in flutter-webrtc already prompts and the status is resolved before the device's blocking path runs. Once flutter-webrtc bumps past `.12` and the pin here follows, this is what keeps the current behavior. ## What Flutter already had flutter-webrtc's `getUserMedia` calls `AVCaptureDevice requestAccessForMediaType:` and waits for the answer, so every livekit_client mic path (publish, `restartTrack` on unmute, pre-connect audio) already prompted before the audio device saw the track. That part of #1085 needs no port. #1182 already maps -9000 to `TrackCreateException` for the direct ADM entry points (`setEngineAvailability`, `startLocalRecording`). ## What this adds The one behavior from #1085 that was missing: only prompt while the app can show the alert. - Native `ensureMicrophoneAccess` in `LiveKitPlugin.swift`: `authorized` passes, `denied`/`restricted` fail, `notDetermined` requests access. On iOS the request is only made while `UIApplication.shared.applicationState == .active`. An inactive or backgrounded app (locked screen, CallKit wake, app switcher) has the alert deferred by the system, and awaiting it would suspend `getUserMedia` and the `_publishRunner` behind it, blocking camera and screen share publishes for as long as the app stays there. Failing fast lets the next foreground attempt prompt normally. macOS can present the prompt regardless, so it always requests. No app extension concern here, the plugin is app-only. - The gate is skipped while engine input availability is disabled (`setEngineAvailability`, the CallKit flow), mirroring the same late fix in #1085: the audio device module defers opening input entirely and runs no permission check there, so gating would turn a working background connect into a `deviceAccessDenied` failure. The check reads the plugin's tracked availability value, so it also covers gating done natively before the Flutter engine exists. - `LocalTrack.createStream` calls it for `AudioCaptureOptions` on Apple platforms before `getUserMedia`. That is the Flutter choke point: `LocalAudioTrack.create()`, `restartTrack()` and `PreConnectAudioBuffer.startRecording()` all reach it. Since the prompt in Flutter happens at `getUserMedia` rather than at capture start, the gate sits in front of that instead of in `startCapture` as in Swift. - Failures surface as `TrackCreateException` through the `deviceAccessDenied` code introduced in #1182. - Docs for `withPreConnectAudio` and `PreConnectAudioBuffer.startRecording` now say permission is requested at recording start but only while the app is active, so callers running at app launch should request it up front (matching the final #1085 wording). `Native.setEngineAvailability` documents that permission is not requested there and must be granted before input availability is restored. ## Testing - `flutter analyze`, `flutter test`, `dart format --set-exit-if-changed`, `import_sorter --exit-if-changed` clean. - Unit tests cover `Native.ensureMicrophoneAccess` (no-op when unimplemented, propagates `deviceAccessDenied`). The `createStream` gate is behind `lkPlatformIsApple()` and not reachable from unit tests. - The example app builds for iOS (device SDK) and macOS with the change, re-verified after the rebase onto `main`. On-device run against a fresh install (first-launch prompt) and a CallKit background wake still to do. Refs CLT-3243, client-sdk-swift#1085
rokk4
added a commit
to rokk4/client-sdk-flutter
that referenced
this pull request
Sep 1, 2026
Brings in three upstream commits: 8827c32 Resolve a default audio session from engine state when no policy was pushed (livekit#1182) f014f69 Request microphone permission before audio capture starts (livekit#1183) b0e5db2 Declare the UTF-8 byte length in the sendText stream header (livekit#1184) Upstream touched none of the files this fork patches, so the only conflict was pubspec.lock (matcher, meta, test_api - all SDK-vendored). Resolved by re-resolving with Flutter 3.47.2, the SDK the consuming app builds against. The lock now differs from upstream by two lines: our flutter_webrtc 1.6.0+hotfix.1 pin, which the podspecs' WebRTC-SDK 144.7559.10 depends on. The two audio commits are Apple-platform only (LiveKitPlugin.swift); they do not overlap the Android audio-routing work in the app.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On iOS the WebRTC audio engine refuses to enable recording unless the audio session category permits input. Its pre-enable check returns
kAudioEngineErrorAudioSessionInvalidCategory(-9001), which the SDK reported asAudioProcessingException(applyFailed): Audio engine returned error code: -9001.livekit_clientowns the iOS audio session since #1108 (LiveKitPlugin.swiftdisables flutter_webrtc's session management at registration). The native engine observer (LKAudioEngineObserver.willEnableEngine) is the right hook and runs before the engine's check, but it only applied a configuration that Dart had pushed. In automatic mode the only push site wasRoom.connect, so anything that started recording earlier met an empty cache, the observer returned "proceed" with the session stillsoloAmbient, and the engine rolled back:Room.withPreConnectAudio, whichSessionOptions.preConnectAudioenables by default, so the Flutter agent starter failed on every "Start call"setEngineAvailabilityon a CallKit killed-state wakeBecause the preconnect throw happens before
connect, the cache was never seeded and the failure repeated on every attempt.How the Swift SDK handles this
AudioSessionEngineObserver.engineWillEnablederives the session configuration from the requested engine state alone (playAndRecordpresets while recording,playbackfor playout only), synchronously inside the engine's enable call. Nothing is configured "before connect". The engine asks, the observer configures, the engine starts. The Flutter plugin already has the same observer in the same place, it just had no built-in policy.Fix
LKAudioEngineObserver.effectiveConfigurationLockednow resolves a built-inplayAndRecordpreset (allowBluetooth | allowBluetoothA2DP | allowAirPlay,videoChat) whenever nothing has been pushed and automatic management is on. The existing playout-onlyplaybackbranch applies to it as well. Manual mode still leaves the session alone. The Dart-pushed policy becomes an override rather than a prerequisite, and for the defaultAudioSessionOptions.communicationit pushes the same values, so the connect-time push does not change the live session.The preset is built on a copy of the shared
RTCAudioSessionConfiguration.webRTC()object, and it is best-effort: if applying it fails, the engine start proceeds and the ADM's own pre-enable checks still gate recording, so apps whose own session was already valid are not newly rolled back with -4100. Only a policy the app actually pushed keeps failing the engine start hard. The mode is fixed tovideoChatbecause it matches the Dart default speaker preference, and a non-default preference always arrives as a pushed policy whose mode already carries it, so the preset can never observe anything else.LocalAudioTrack.startCaptureadditionally pushes the resolved Dart policy to native before recording starts (cache-only while the engine is idle in automatic mode). Flutter-driven starts therefore always use the real Dart policy, and the built-in preset only stands in for engine starts that happen before the Flutter side exists, for example the plugin's staticsetEngineAvailabilityon a CallKit killed-state wake.The engine observer is also shared across plugin registrations now: a second Flutter engine registering in the same process (add-to-app,
FlutterEngineGroup) previously reset the pushed policy and management mode silently, which the preset would have turned into an unwanted session activation. Only the notification channel is rebound per registration.Error mapping
Audio device module results now get their own error codes on the
startLocalRecording,setEngineAvailability,setMicrophoneMuteModeandstopLocalRecordingchannels, mirroring client-sdk-swift'scheckAdmResult. A mute-mode change can rebuild the engine and hit the same pre-enable checks as a recording start, so all entry points surface the same failure the same way:InsufficientDevicePermissiondeviceAccessDeniedTrackCreateExceptionAudioSessionInvalidCategoryaudioSessionInvalidCategoryAudioSessionException(new)FailedToConfigureAudioSessionaudioSessionConfigureFailedAudioSessionException(new)applyFailed,setEngineAvailability, ...)AudioSessionExceptionis a new public class with its own changeset entry.Relation to #1179
@MaxHeimbrock's #1179 diagnosed this first and fixes it by pushing the policy from
LocalAudioTrack.startCapture(prepareRecording()). This PR ends up covering both layers:startCapturepushes the resolved policy like #1179 does (without a Dart-side flag tracking native cache state), and the native observer is additionally self-sufficient like Swift's, so engine enables that never pass through Dart are covered too. With this merged,prepareRecording()becomes redundant.Testing
Start callfailed with -9001 on every attempt. Patched,startCapture()succeeds two seconds before the connect-time policy push, the preconnect buffer is sent to the agent, and the call connects.flutter analyze,flutter test,dart format --set-exit-if-changed,import_sorter --exit-if-changedclean.lkPlatform()has no seam), same limitation as Configure the audio session before recording starts, not only on connect #1179 noted.Fixes #1165
Refs #1042