feat: Spatius avatar plugin by 3DRX · Pull Request #5821 · livekit/agents

3DRX · 2026-05-23T15:02:06Z

Hi LiveKit team! 👋

This PR introduces the Spatius avatar plugin.

Working demos:
👉 Spatius LiveKit Agents Demo
👉 Product home: https://spatius.ai/

User context:
We had 400+ users on SpatialReal platform, and Spatius AI has already picked up 50+ signups in its first few weeks. Around 70% are developers, and most of them are already building on LiveKit — so having the plugin in the main repo would directly help them discover the integration and offer a new affordable live avatar option.
Happy to address any review feedback or run additional tests.

Thanks for your time!

devin-ai-integration

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 4 additional findings.

3DRX · 2026-06-02T18:30:09Z

@tinalenguyen Hi, sorry for the buzz. I just updated this PR, and resolved all the review and CI checks. It's ready from my point of view, please have a look when you have time :)

Thank you so much for your time.

3DRX · 2026-06-28T13:48:28Z

To answer questions in #5014

Also, is there a reason why livekit_egress doesn't accept a JWT token instead of LiveKit API secret and key? It would be good to scope credentials to each room that way. I also don't think you need to set audio_output to False in the RoomOptions with that approach

The initial implementation of our LiveKit integration generates livekit token in Spatius' service, which is not good for security. We've changed it since, now the token generation is in this livekit agents plugin.

I tested it out and I wasn't able to see the avatar join the room at all, but audio was published. Could you pull from main and bump the versions to see if that helps?

I've rebased again, pull from main and bump the versions. For my testing it works. https://docs.spatius.ai/livekit-agents/overview Is the documentation we write to help with the setup (a bit more complicated than video streaming avatar, since we do the rendering on client side).

mgznyuan · 2026-07-07T12:28:35Z

@tinalenguyen gentle ping on this one — whenever you get a moment. Just wanted to make sure this doesn't get buried😄

mahimairaja · 2026-07-07T18:50:16Z

Great work! @3DRX @mgznyuan

tinalenguyen

hi, apologies about the delayed review, i left a comment to fix the py.typed file

i also tried it out and the video track was only a black screen, i tested out all the public avatars and i didn't see any changes. could you try to reproduce? i only passed the avatar ID as an argument, ideally the plugin only needs that (or none with a default avatar ID) and works out of the box

tinalenguyen · 2026-07-09T05:22:47Z

@@ -0,0 +1 @@
+


Suggested change

Thanks, fixed this :)

Hi Tina, have you tried our livekit demo? https://github.com/spatius-ai/spatius-avatar-demo/tree/main/platform-integrations/livekit-agents-demo/livekit-agents-reference-demo Maybe this can be helpful for your review.

3DRX · 2026-07-09T06:18:06Z

hi, apologies about the delayed review, i left a comment to fix the py.typed file

i also tried it out and the video track was only a black screen, i tested out all the public avatars and i didn't see any changes. could you try to reproduce? i only passed the avatar ID as an argument, ideally the plugin only needs that (or none with a default avatar ID) and works out of the box

Hi @tinalenguyen. Thanks for reviewing this, to make it clear, it is expected to only receive black frames when attempting to playback the track as video. We actually renders the avatar at client side so it’s motion data (not video frames) we are streaming through livekit. We have a client side rendering SDK and a LiveKit etc adapter that helps to bridge between rtc track and client rendering pipeline. Please checkout the example project and this documentation https://docs.spatius.ai/livekit-agents/client.

By doing the rendering on client side, we are able to achieve significantly lower cost and bandwidth requirement.

longcw · 2026-07-10T10:00:02Z

it's interesting to run the rendering on client side, would like to test it as well.

longcw

I tested with the frontend in https://github.com/spatius-ai/spatius-avatar-demo/tree/main/platform-integrations/livekit-agents-demo/livekit-agents-reference-demo/frontend.

so basically the plugin needs a specific frontend to show the avatar since the rendering is in the client side. I think it's better to mention this in the plugin's readme and agent example.

longcw · 2026-07-11T02:12:36Z

+
+        self._agent_session = agent_session
+        self._original_audio_output = agent_session.output.audio
+        self._original_audio_tail = self._get_audio_sink_proxy_tail(agent_session.output.audio)


maybe it's not needed inside the plugin if finding the original tail is for switching it back in aclose.

longcw · 2026-07-11T02:16:17Z

+
+        segment.pushed_duration += frame.duration
+        self._active_req_id = req_id
+        self._schedule_active_segment_idle_end()


the audio is segmented in like 20ms per frame, this may cancel the task and recreate it per frame? could this be optimized in some other way?

longcw · 2026-07-11T02:24:37Z

I also noticed that the plugin may send multiple playback completed events in a single reply, see the logs below

2026-07-11 10:20:05,725 - DEBUG livekit.plugins.spatius - Spatius avatar first audio frame {"request_id": "20260711022005_xAPSg5JsBpVT", "pid": 28844, "job_id": "AJ_HNQ8eG7Pivit", "room": "voice-agent-room"}
2026-07-11 10:20:07,184 - WARNING livekit.plugins.spatius - Avatar segment end marker missing; forcing finalization {"request_id": "20260711022005_xAPSg5JsBpVT", "idle_timeout": 1.0, "pid": 28844, "job_id": "AJ_HNQ8eG7Pivit", "room": "voice-agent-room"}
2026-07-11 10:20:08,300 - DEBUG livekit.plugins.spatius - Spatius avatar first audio frame {"request_id": "20260711022008_mQlPWLm07f_W", "pid": 28844, "job_id": "AJ_HNQ8eG7Pivit", "room": "voice-agent-room"}
2026-07-11 10:20:09,955 - WARNING livekit.agents - playback_finished called more times than playback segments were captured {"pid": 28844, "job_id": "AJ_HNQ8eG7Pivit", "room": "voice-agent-room"}
2026-07-11 10:20:09,956 - DEBUG livekit.plugins.spatius - Spatius avatar segment playback completed {"request_id": "20260711022000_Ke3EIjJUqJKO", "reason": "timeout", "interrupted": false, "playback_position": 7.392000000000004, "pushed_duration": 7.392000000000004, "pid": 28844, "job_id": "AJ_HNQ8eG7Pivit", "room": "voice-agent-room"}
2026-07-11 10:20:09,956 - WARNING livekit.plugins.spatius - Avatar segment completion timed out, assuming playback finished {"request_id": "20260711022000_Ke3EIjJUqJKO", "timeout": 7.780106067657471, "pid": 28844, "job_id": "AJ_HNQ8eG7Pivit", "room": "voice-agent-room"}
2026-07-11 10:20:10,010 - WARNING livekit.agents - playback_finished called more times than playback segments were captured {"pid": 28844, "job_id": "AJ_HNQ8eG7Pivit", "room": "voice-agent-room"}
2026-07-11 10:20:10,011 - DEBUG livekit.plugins.spatius - Spatius avatar segment playback completed {"request_id": "20260711021957__Exu8cOlOfXJ", "reason": "timeout", "interrupted": false, "playback_position": 10.391999999999998, "pushed_duration": 10.391999999999998, "pid": 28844, "job_id": "AJ_HNQ8eG7Pivit", "room": "voice-agent-room"}

not sure if it's from the network connection issue or some race conditions.

3DRX · 2026-07-12T06:03:43Z

Thanks @longcw — I pushed 668a57b16 to address the review feedback:

documented the required Spatius client-side renderer in both the plugin README and agent example, with links to the client guide and reference frontend
removed the original audio-tail discovery/restoration logic
replaced the per-frame idle task cancellation/recreation with one persistent idle watchdog
changed implicit EOF handling to flush QueueAudioOutput, so the resulting AudioSegmentEnd keeps LiveKit playback segment accounting aligned with Spatius request boundaries
grouped provider request IDs under each LiveKit audio segment and now emit playback completion only once per LiveKit segment, including provider/timeout races

The one-second implicit EOF is still needed for realtime providers such as Gemini Live that can stop producing audio without sending an explicit EOF. I tested the updated implementation with our Gemini Live end-to-end demo and it is working as expected.

longcw · 2026-07-13T05:07:23Z

@3DRX I think I see the issue: the error I mentioned above only happens when using a slow non-streaming TTS (e.g. the default openai TTS), where the audio is generated sentence by sentence and there can have 1s delay between sentences.

when the delay is greater than ACTIVE_SEGMENT_IDLE_END_SECONDS (1s), the plugin fires the warning and hard cuts the generation. it works well when it uses a faster TTS.

but I think it needs to be fixed from the plugin: the framework is making sure one AudioSegmentEnd (= one flush) ⇒ exactly one notify_playback_finished, that should also be applied to gemini live. so maybe a timeout if not needed?

3DRX · 2026-07-13T05:15:58Z

@longcw Got it, I will test it against the latest Gemini live plugin. I agree that ideally the avatar plugin shouldn’t need to detect implicit audio end. We have this because some users are having the issue regarding Gemini live in livekit agents 1.5 era so we add this as a temporary workaround.

3DRX · 2026-07-13T14:05:48Z

@longcw Got it, I will test it against the latest Gemini live plugin. I agree that ideally the avatar plugin shouldn’t need to detect implicit audio end. We have this because some users are having the issue regarding Gemini live in livekit agents 1.5 era so we add this as a temporary workaround.

I tried to remove the ACTIVE_SEGMENT_IDLE_END_SECONDS (1s) logic and test it against various TTS providers, and I find this suitable for most TTS providers but not for gemini live. The problem is when using gemini live, AudioSegmentEnd came in much later than the last audio segment (for example when the user's prompt is "tell me a story", AudioSegmentEnd can be 20+ seconds later than the last audio segment). And by reading the google plugin code, seems like it only close the channel and emitting AudioSegmentEnd until received turn_complete from gemini API, which won't arrive as soon as audio segment ends. Instead, it arrives at estimated playback end time, which is certainly much later then audio generation finished time.

Internally, Spatius rely on the EOF marker (end=True) to flush internal audio buffer & for inferencing, so we need an audio segment end signal that's soon after final audio chunk arrives. This means it's incompatable with current gemini live's AudioSegmentEnd sending behavior.

3DRX · 2026-07-13T14:10:30Z

@longcw Seems like generationComplete is a more suitable signal from gemini live API to be used for AudioSegmentEnd. But since this is a behavior change, is it likely to be changed in the google plugin right now? If not, does it make sense for us to detect if gemini live is used in AgentSession and just keep this ACTIVE_SEGMENT_IDLE_END_SECONDS (1s) behavior specifically for gemini live?

3DRX · 2026-07-17T06:14:00Z

Fixed transcription sync by using signal from lk.playback_started and lk.playback_finished, and also defaults to sending PCM audio to spatius.

Address avatar review feedback: - wire the Spatius SDK on_error/on_close callbacks so a dropped ingress connection tears the session down and releases any pending playback, instead of hanging AgentSession.wait_for_playout() - arm a clear-buffer timeout that finishes the segment locally when the worker never confirms an interrupt with a playback-finished RPC - forward every playback-finished RPC straight to the base AudioOutput, which already does 1:1 segment accounting, and drop the redundant per-segment bookkeeping (_SegmentState, pending deque, finalize lock, interrupted-position estimation) - drop the user_state_changed interrupt; clear_buffer is the sole interrupt signal - validate configuration before super().start() so misconfigurations fail fast without leaving background tasks running - collapse teardown into a single _closed guard and keep a reference to the scheduled aclose task - drop the manual exception-chain unwinding in favor of raise-from

devin-ai-integration Bot reviewed May 23, 2026

View reviewed changes

3DRX force-pushed the add-spatius-avatar-plugin branch from 00544ad to 62653e3 Compare June 2, 2026 07:16

This comment was marked as resolved.

Sign in to view

3DRX force-pushed the add-spatius-avatar-plugin branch from 62653e3 to 5971642 Compare June 2, 2026 18:06

3DRX changed the title ~~Add Spatius avatar plugin~~ feat: Spatius avatar plugin Jun 2, 2026

This comment was marked as resolved.

Sign in to view

3DRX force-pushed the add-spatius-avatar-plugin branch from 5971642 to 1aeadcb Compare June 2, 2026 18:21

3DRX force-pushed the add-spatius-avatar-plugin branch from 1aeadcb to afe1254 Compare June 3, 2026 05:43

3DRX force-pushed the add-spatius-avatar-plugin branch from afe1254 to dc83c03 Compare June 10, 2026 16:18

This comment was marked as resolved.

Sign in to view

3DRX force-pushed the add-spatius-avatar-plugin branch from dc83c03 to 1adb288 Compare June 13, 2026 05:51

3DRX requested a review from a team as a code owner June 13, 2026 05:51

tinalenguyen reviewed Jul 9, 2026

View reviewed changes

This comment was marked as resolved.

Sign in to view

longcw reviewed Jul 11, 2026

View reviewed changes

longcw mentioned this pull request Jul 15, 2026

fix(google): close realtime output streams on generation_complete #6431

Merged

Add Spatius avatar plugin

fd7a25f

3DRX added 12 commits July 18, 2026 19:28

chore: sync spatius avatar updates

87c3fd8

chore: bump spatius plugin version

e78c84b

fix(spatius): align avatar playback segments

a7972ed

chore: remove Spatius avatar example

7c26514

fix(spatius): finalize on audio segment end

4a90d14

feat(spatius): sync playback from avatar RPCs

6cdb18d

fix(spatius): identify agent for playback RPCs

874aca3

fix(spatius): permit avatar playback RPCs

664c4d9

fix(spatius): keep interruption on SDK transport

b1abed5

fix(spatius): default to PCM audio

3a9d8f1

chore(spatius): align plugin version with 1.6.6

90c9ded

longcw approved these changes Jul 20, 2026

View reviewed changes

cursor Bot mentioned this pull request Jul 21, 2026

docs: daily engineering digest 2026-07-21 Seventhen/agents#36

Draft

Uh oh!

Conversation

3DRX commented May 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

✅ Devin Review: No Issues Found

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

3DRX commented Jun 2, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

3DRX commented Jun 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mgznyuan commented Jul 7, 2026

Uh oh!

mahimairaja commented Jul 7, 2026

Uh oh!

tinalenguyen left a comment

Choose a reason for hiding this comment

Uh oh!

tinalenguyen Jul 9, 2026

Choose a reason for hiding this comment

Uh oh!

3DRX Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

mgznyuan Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

3DRX commented Jul 9, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

longcw commented Jul 10, 2026

Uh oh!

longcw left a comment

Choose a reason for hiding this comment

Uh oh!

longcw Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

longcw Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

longcw commented Jul 11, 2026

Uh oh!

3DRX commented Jul 12, 2026

Uh oh!

longcw commented Jul 13, 2026

Uh oh!

3DRX commented Jul 13, 2026

Uh oh!

3DRX commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

3DRX commented Jul 13, 2026

Uh oh!

3DRX commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

3DRX commented May 23, 2026 •

edited

Loading

3DRX commented Jun 28, 2026 •

edited

Loading

mgznyuan Jul 10, 2026 •

edited

Loading

3DRX commented Jul 13, 2026 •

edited

Loading