fix(mobile): stabilize scene realtime audio flow - #142
Conversation
![fennoai[bot]](https://avatars.githubusercontent.com/in/1924301?s=60&v=4){kind=small}
There was a problem hiding this comment.
Reviewed the Android PCM tap, JS capture bridge, realtime barge-in cancellation, and read-to-speak recorder handoff. The focused controller/screen/audio tests (43 tests), TypeScript check, and clean Expo Android prebuild pass. The new capture path still has two lifecycle issues that can exhaust memory or silently skip scoring after a native failure. An Android Kotlin compile was attempted but the Gradle distribution download was blocked by the environment's TLS certificate validation.
| sampleRate = rate | ||
| } | ||
| if (format != audioFormat || channels != channelCount || rate != sampleRate) return | ||
| segment.write(data) |
There was a problem hiding this comment.
[P2] Bound the PCM captured while waiting for speech
enableSceneInputAfterRecordingStarts() starts this segment before the learner begins speaking, so every WebRTC microphone frame is appended here for the entire time the scene sits ready. Because ByteArrayOutputStream has no cap or streaming sink, an idle screen accumulates about 96 KB/s at 48 kHz mono PCM16 (roughly 173 MB in 30 minutes), and takeSegment() then creates several additional full-size copies during decode/resample/encoding. This can OOM the mobile process or produce an unexpectedly huge upload after a delayed response; use a bounded pre-roll/ring buffer or stream the segment to a file with a duration/size limit.
| stopPromise = Promise.resolve(nativeTap.stopSegment()); | ||
| } | ||
| if (stopPromise) await stopPromise; | ||
| const uri = finalized ? await nativeTap.takeSegment() : null; |
There was a problem hiding this comment.
[P2] Clear capture state when native take fails
If nativeTap.takeSegment() rejects (the native module explicitly rejects file creation/write failures), execution skips the two reset lines below. RealtimeSessionController.takeTurnAudioUri() catches that rejection and returns null, leaving this object with finalized === true; the next learner turn's start() therefore returns early and is not recorded. A rejected stopPromise is worse because every later take() rethrows the same promise. Reset the JS state in a finally block so a transient native failure cannot disable subsequent turn scoring.
2 participants
Summary
Root cause
Mobile scene practice opened WebRTC at 48 kHz for realtime conversation and AudioStudio at 16 kHz for WAV scoring at the same time. On the OnePlus test device the two Android recording sessions competed for the microphone route, which caused premature VAD stops, delayed interruption, short or invalid scoring audio, and the initial AI preparing state.
Validation
No Web or backend implementation files are changed.