Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
4d6acfe
feat: live voice conversations for Fast sessions
mrubens Sep 1, 2026
e7e280f
feat: continuous transcription with client-side turn detection
mrubens Sep 1, 2026
5756284
fix: speak replies on turn settle, guard voice start races, cover voi…
mrubens Sep 4, 2026
dbdb735
fix: voice cutoff from server timestamps, stop voice on structured input
mrubens Sep 4, 2026
b0e3a8f
perf: speak replies as they stream instead of after the turn settles
mrubens Sep 4, 2026
8c6e5b0
feat: start a voice-mode Fast session from the home and New Session c…
mrubens Sep 4, 2026
ddfd669
feat: connect Fast voice sessions through GPT-Live-1
mrubens Sep 10, 2026
8785802
Merge remote-tracking branch 'origin/develop' into feat/live-voice
mrubens Sep 10, 2026
c97783c
fix: keep Live delegations and Fast results correlated
mrubens Sep 10, 2026
3bb804c
fix: make voice usable end to end and give GPT-Live workspace context
mrubens Sep 10, 2026
4c6ff54
refactor: clean voice transcripts with the deployment helper model
mrubens Sep 10, 2026
89ae696
feat: require the dedicated voice key and start new voice sessions fr…
mrubens Sep 10, 2026
6b41f6b
feat: read Fast replies verbatim as they stream and never drop an utt…
mrubens Sep 10, 2026
1c1f8f9
feat: make voice a transcribed call on the Session
mrubens Sep 10, 2026
eb9ac7d
Merge remote-tracking branch 'origin/develop' into feat/live-voice
mrubens Sep 10, 2026
021d792
feat: stream the call into the transcript and attribute streamed repl…
mrubens Sep 10, 2026
b097150
fix: keep spoken acknowledgements after the request they answer
mrubens Sep 10, 2026
5033cef
fix: drop GPT-Live sound annotations from the call transcript
mrubens Sep 10, 2026
f731310
fix: hold spoken acknowledgements on an explicit in-flight request count
mrubens Sep 10, 2026
83805bc
fix: drop annotation fragments split across spoken turns
mrubens Sep 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion apps/docs/docs.json
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,8 @@
"goal-mode",
"fast-sessions",
"memory",
"file-attachments"
"file-attachments",
"voice"
]
},
{
Expand Down
1 change: 1 addition & 0 deletions apps/docs/environment-variables.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -372,6 +372,7 @@ as per-task auth tokens or workspace paths.
| `R_ALLOWED_EMAILS` | Optional | Comma-separated email allowlist for deployments that restrict sign-in by email. |
| `R_ELEVENLABS_API_KEY` | Optional | ElevenLabs API key for narrated feature-demo videos. The key stays on the control plane; sandboxes reach text-to-speech only through an authenticated Roomote endpoint. A key scoped to text-to-speech only is sufficient and recommended. |
| `R_ELEVENLABS_VOICE_ID` | Optional | ElevenLabs voice ID used for feature-demo narration. Required alongside the API key for narration to be available. |
| `R_VOICE_OPENAI_API_KEY` | Optional | OpenAI key dedicated to GPT-Live voice conversations. Falls back to `OPENAI_API_KEY` when unset and requires project access to `gpt-live-1`. The key stays on the control plane; browsers receive only a server-negotiated WebRTC session answer. |

During Microsoft Teams setup, Roomote uses the Microsoft Entra app values for
the Teams bot by default. Use **Show advanced config** after the Directory
Expand Down
68 changes: 68 additions & 0 deletions apps/docs/voice.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
---
title: Voice
icon: audio-lines
description: Talk naturally with a Fast Session using GPT-Live.
---

Voice turns a Fast Session into a natural, full-duplex spoken conversation.
GPT-Live handles listening, speaking, and interruptions while the actual Fast
Session handles questions and work with its selected model, tools, context, and
durable transcript. You can move between voice and text without switching to a
separate voice-only agent.

## Enabling voice

Voice uses OpenAI GPT-Live-1 and is available on deployments that set
`R_VOICE_OPENAI_API_KEY` to a key from an OpenAI project with GPT-Live access.
Voice is opt-in: the deployment's general `OPENAI_API_KEY` is not used, so
enabling OpenAI for task inference does not turn voice on.

When the voice key is not configured the voice button does not appear. The key
stays on the control plane. The browser sends its WebRTC connection offer to Roomote
and receives only the negotiated session answer; it never receives the API key.

## Using voice

1. Select the voice button in a composer: in an open Session, or on the home
page and the **New Session** dialog. From the home page or dialog a new
Session is created and the call starts inside it.
2. Grant microphone access when the browser asks. A short rising tone
confirms the call is open; a falling tone marks the end. A **Call started**
marker appears in the Session.
3. Talk to Roomote the way you would on a phone call. It acknowledges each
request in a few words, hands the work to the Fast Session, and reports the
result out loud when it lands. Greetings, thanks, and small talk are
answered directly without starting Fast work.
4. Speak at any time to interrupt. Roomote keeps listening while it speaks,
and follow-ups go back through the same Fast Session. You can also type in
the composer during the call.
5. Use the in-call controls to mute your microphone, silence Roomote's audio
without muting yourself, or end the call. The button stays highlighted
while the call is active, and a **Call ended** marker records its length.

Voice input requires a browser with microphone and WebRTC support, which
includes current Chrome, Edge, Safari, and Firefox.

## How the transcript works

A voice call is transcribed into the Session as the record of what was said.
Your speech appears as your messages, and what Roomote said out loud appears
as its replies. The Fast Session's work, such as tool calls, launched tasks,
and reports, appears between those turns exactly as it does in a typed
Session, so the timeline shows both the conversation and the work behind it.

During a call the Fast Session returns its results to the voice rather than
writing them as chat replies; Roomote then reports them in its own words,
keeping numbers, names, paths, and link labels exact. If a result cannot be
spoken, for example because the call drops, it is written to the transcript
as a normal reply so nothing is lost. Typed messages sent during a call are
answered in writing as usual.

Each spoken request is cleaned up (filler words, false starts, and misheard
terms) by the deployment's helper model before it reaches the Fast Session.
GPT-Live is told which repositories, environments, and integrations the Fast
Session can reach, so it recognises their names, and the same names guide the
cleanup so a misheard repository name is corrected to the real one.

Ending the call stops the microphone; Fast work already started remains
visible in the Session and follows the normal Session lifecycle.
73 changes: 73 additions & 0 deletions apps/web/src/app/(authenticated)/home/Home.client.test.tsx

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading