Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
@@ -1 +1,4 @@
# Copy this file to assets/.env and set your token server sandbox ID,
# found in your LiveKit Cloud project's Settings page.
# Without it, the app connects to a default agent.
LIVEKIT_SANDBOX_ID=<your-sandbox-id>
3 changes: 0 additions & 3 deletions .github/workflows/test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,6 @@ jobs:
with:
channel: "stable"

- name: Create empty .env file
run: touch .env

- name: Install dependencies
run: flutter pub get

Expand Down
28 changes: 19 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,10 @@ This template is compatible with iOS, macOS, Android, and web. It is free for yo

## Getting started

First, you'll need a LiveKit agent to speak with. Try our starter agent for [Python](https://github.com/livekit-examples/agent-starter-python), [Node.js](https://github.com/livekit-examples/agent-starter-node), or [create your own from scratch](https://docs.livekit.io/agents/start/voice-ai/).

Second, you need a token server. The easiest way to set this up is with a [token server](https://docs.livekit.io/frontends/authentication/tokens/sandbox-token-server/) and the [LiveKit CLI](https://docs.livekit.io/home/cli/cli-setup/).

First, enable the token server from your project's **Options** on the [Settings](https://cloud.livekit.io/projects/p_/settings/project) page in LiveKit Cloud and copy the `sandboxId`.

Then, run the following command to automatically clone this template and connect it to LiveKit Cloud.
Run the following command to automatically clone this template and connect it to LiveKit Cloud.

```bash
lk app create --template agent-starter-flutter --sandbox <token_server_sandbox_id>
lk app create --template agent-starter-flutter
```

This will create a new Flutter project in the current directory. Install dependencies and run the app:
Expand All @@ -30,8 +24,24 @@ flutter run

Note: You may need to configure signing certificates in Xcode if building to a real iOS device.

The app is configured to connect to the LiveKit homepage agent by default, which you can also try at [livekit.com](https://www.livekit.com). To point the app at your own agent (see [Connect to your agent](#connect-to-your-agent)).

> [!NOTE]
> To setup without the LiveKit CLI, clone the repository and then either create a `.env` with a `LIVEKIT_SANDBOX_ID` (from your project's **Options** on the [Settings](https://cloud.livekit.io/projects/p_/settings/project) page), or modify `lib/controllers/app_ctrl.dart` to replace the `SandboxTokenSource` with your own token source implementation (development-only hardcoded credentials are also supported there).
> To setup without the LiveKit CLI, clone the repository and then either create the `assets/.env` file manually from a copy of `.env.example`. The env file is optional: without any configuration, the app connects to a default agent — the same one featured on the [LiveKit homepage](https://livekit.io) — so you can try it out right away.

## Connect to your agent

To switch from the default agent to your own, you first need a LiveKit agent to speak with. For a no-code setup, use the [Agent Builder](https://docs.livekit.io/agents/start/builder/). For more customization, try our starter agent for [Python](https://github.com/livekit-examples/agent-starter-python), [Node.js](https://github.com/livekit-examples/agent-starter-node), or [create your own from scratch](https://docs.livekit.io/agents/start/voice-ai/).

Second, you need a token server. For development, the easiest option is the [sandbox token server](https://docs.livekit.io/frontends/authentication/tokens/sandbox-token-server/): enable it from your project's **Options** on the [Settings](https://cloud.livekit.io/projects/p_/settings/project) page in LiveKit Cloud and copy the `sandboxId`.

Then fill the `LIVEKIT_SANDBOX_ID` in your `assets/.env`:

```swift
LIVEKIT_SANDBOX_ID=<your-sandbox-id>
```

or modify `lib/controllers/app_ctrl.dart` to replace the `SandboxTokenSource` with your own token source implementation (development-only hardcoded credentials are also supported there).

## Feature overview

Expand Down
11 changes: 8 additions & 3 deletions lib/controllers/app_ctrl.dart
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ import 'package:logging/logging.dart';
import 'package:uuid/uuid.dart';
import 'package:flutter_dotenv/flutter_dotenv.dart';

final String homepageAgentTokenEndpoint = 'https://livekit.com/api/homepage-agent/token';

enum AppScreenState { welcome, agent }

enum AgentScreenState { visualizer, transcription }
Expand Down Expand Up @@ -47,12 +49,15 @@ class AppCtrl extends ChangeNotifier {
}

final sandboxId = dotenv.env['LIVEKIT_SANDBOX_ID']?.replaceAll('"', '');
if (sandboxId == null || sandboxId.isEmpty) {
throw StateError('LIVEKIT_SANDBOX_ID is not set and no hardcoded token is configured.');
sdk.EndpointTokenSource tokenSource;
if (sandboxId == null || sandboxId.isEmpty || sandboxId == '<your-sandbox-id>') {
tokenSource = sdk.EndpointTokenSource(url: Uri.parse(homepageAgentTokenEndpoint));
} else {
tokenSource = sdk.SandboxTokenSource(sandboxId: sandboxId);
}

return sdk.Session.fromConfigurableTokenSource(
sdk.SandboxTokenSource(sandboxId: sandboxId).cached(),
tokenSource,
options: sdk.SessionOptions(room: room),
);
}
Expand Down
3 changes: 2 additions & 1 deletion lib/main.dart
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@ import 'app.dart';

// Load environment variables before starting the app
// This is used to configure the LiveKit sandbox ID for development
// The file is optional; without it the app connects to a default agent (see app_ctrl.dart)
void main() async {
await dotenv.load(fileName: '.env');
await dotenv.load(fileName: 'assets/.env', isOptional: true);
runApp(const VoiceAssistantApp());
}
167 changes: 91 additions & 76 deletions lib/screens/agent_screen.dart
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import 'package:provider/provider.dart';
import '../controllers/app_ctrl.dart';
import '../support/agent_selector.dart';
import '../widgets/agent_layout_switcher.dart';
import '../widgets/agent_status_indicator.dart';
import '../widgets/camera_toggle_button.dart';
import '../widgets/message_bar.dart';

Expand Down Expand Up @@ -115,84 +116,98 @@ class AgentScreen extends StatelessWidget {
isCameraVisible: appCtrl.isUserCameEnabled,
isScreenshareVisible: appCtrl.isScreenshareEnabled,
),
builder: (ctx, agentLayoutState, child) => AgentLayoutSwitcher(
layoutState: agentLayoutState,
// agentViewBuilder: (ctx) => AgentTrackView(),
buildAgentView: (ctx) => const AgentTrackView(),
buildCameraView: (ctx) => Container(
clipBehavior: Clip.hardEdge,
decoration: BoxDecoration(
borderRadius: BorderRadius.circular(15),
),
child: components.MediaDeviceContextBuilder(
builder: (context, roomCtx, mediaDeviceCtx) => components.ParticipantSelector(
filter: (identifier) => identifier.isVideo && identifier.isLocal,
builder: (context, identifier) => Stack(
children: [
components.VideoTrackWidget(
fit: sdk.VideoViewFit.cover,
noTrackBuilder: (ctx) => Container(color: Theme.of(ctx).cardColor),
),
Positioned(
right: 10,
bottom: 10,
child: CameraToggleButton(
onTap: () => mediaDeviceCtx.toggleCameraPosition(),
),
),
],
),
builder: (ctx, agentLayoutState, child) => Stack(
children: [
_buildLayoutSwitcher(ctx, agentLayoutState),
// In transcription mode the chat placeholder shows the agent
// status instead.
if (!agentLayoutState.isTranscriptionVisible)
const Positioned(
left: 0,
right: 0,
bottom: 110,
child: Center(child: AgentStatusIndicator(hideWhenConnected: true)),
),
],
),
),
);

Widget _buildLayoutSwitcher(BuildContext ctx, AgentLayoutState agentLayoutState) => AgentLayoutSwitcher(
layoutState: agentLayoutState,
// agentViewBuilder: (ctx) => AgentTrackView(),
buildAgentView: (ctx) => const AgentTrackView(),
buildCameraView: (ctx) => Container(
clipBehavior: Clip.hardEdge,
decoration: BoxDecoration(
borderRadius: BorderRadius.circular(15),
),
child: components.MediaDeviceContextBuilder(
builder: (context, roomCtx, mediaDeviceCtx) => components.ParticipantSelector(
filter: (identifier) => identifier.isVideo && identifier.isLocal,
builder: (context, identifier) => Stack(
children: [
components.VideoTrackWidget(
fit: sdk.VideoViewFit.cover,
noTrackBuilder: (ctx) => Container(color: Theme.of(ctx).cardColor),
),
Positioned(
right: 10,
bottom: 10,
child: CameraToggleButton(
onTap: () => mediaDeviceCtx.toggleCameraPosition(),
),
),
],
),
),
buildScreenShareView: (ctx) => Container(
alignment: Alignment.center,
decoration: BoxDecoration(
color: Colors.green.withValues(alpha: 0.3),
),
),
buildScreenShareView: (ctx) => Container(
alignment: Alignment.center,
decoration: BoxDecoration(
color: Colors.green.withValues(alpha: 0.3),
),
child: const Text('Screenshare View'),
),
transcriptionsBuilder: (ctx) => Column(
mainAxisSize: MainAxisSize.max,
crossAxisAlignment: CrossAxisAlignment.stretch,
children: [
Expanded(
child: GestureDetector(
onTap: () => ctx.read<AppCtrl>().messageFocusNode.unfocus(),
child: Consumer<sdk.Session>(
builder: (context, session, _) {
if (session.messages.isEmpty) {
return _AgentStatusPlaceholder(isAgentConnected: session.agent.isConnected);
}
return components.ChatScrollView(
session: session,
padding: const EdgeInsets.symmetric(horizontal: 16, vertical: 20),
physics: const BouncingScrollPhysics(),
messageBuilder: (context, message) => Padding(
padding: const EdgeInsets.only(bottom: 12),
child: _MessageBubble(message: message),
),
);
},
),
),
child: const Text('Screenshare View'),
),
transcriptionsBuilder: (ctx) => Column(
mainAxisSize: MainAxisSize.max,
crossAxisAlignment: CrossAxisAlignment.stretch,
children: [
Expanded(
child: GestureDetector(
onTap: () => ctx.read<AppCtrl>().messageFocusNode.unfocus(),
child: Consumer<sdk.Session>(
builder: (context, session, _) {
if (session.messages.isEmpty) {
return _AgentListeningPlaceholder(canListen: session.agent.canListen);
}
return components.ChatScrollView(
session: session,
padding: const EdgeInsets.symmetric(horizontal: 16, vertical: 20),
physics: const BouncingScrollPhysics(),
messageBuilder: (context, message) => Padding(
padding: const EdgeInsets.only(bottom: 12),
child: _MessageBubble(message: message),
),
);
},
),
),
),
Padding(
padding:
EdgeInsets.only(left: 16, right: 16, bottom: max(0, MediaQuery.of(ctx).viewInsets.bottom - 80)),
child: Selector<AppCtrl, bool>(
selector: (ctx, appCtx) => appCtx.isSendButtonEnabled,
builder: (ctx, isSendEnabled, child) => MessageBar(
focusNode: ctx.read<AppCtrl>().messageFocusNode,
isSendEnabled: isSendEnabled,
controller: ctx.read<AppCtrl>().messageCtrl,
onSendTap: () => ctx.read<AppCtrl>().sendMessage(),
),
),
Padding(
padding: EdgeInsets.only(left: 16, right: 16, bottom: max(0, MediaQuery.of(ctx).viewInsets.bottom - 80)),
child: Selector<AppCtrl, bool>(
selector: (ctx, appCtx) => appCtx.isSendButtonEnabled,
builder: (ctx, isSendEnabled, child) => MessageBar(
focusNode: ctx.read<AppCtrl>().messageFocusNode,
isSendEnabled: isSendEnabled,
controller: ctx.read<AppCtrl>().messageCtrl,
onSendTap: () => ctx.read<AppCtrl>().sendMessage(),
),
],
),
),
),
],
),
);
}
Expand Down Expand Up @@ -246,10 +261,10 @@ class _MessageBubble extends StatelessWidget {
}
}

class _AgentListeningPlaceholder extends StatelessWidget {
const _AgentListeningPlaceholder({required this.canListen});
class _AgentStatusPlaceholder extends StatelessWidget {
const _AgentStatusPlaceholder({required this.isAgentConnected});

final bool canListen;
final bool isAgentConnected;

@override
Widget build(BuildContext context) {
Expand All @@ -262,13 +277,13 @@ class _AgentListeningPlaceholder extends StatelessWidget {
Icon(Icons.graphic_eq, size: 32, color: colorScheme.primary.withValues(alpha: 0.7)),
const SizedBox(height: 12),
Text(
'Agent is listening',
isAgentConnected ? 'Agent is listening' : 'Waiting for agent',
style: textTheme.titleMedium?.copyWith(
fontWeight: FontWeight.w600,
),
textAlign: TextAlign.center,
),
if (!canListen)
if (isAgentConnected)
Padding(
padding: const EdgeInsets.only(top: 6),
child: Text(
Expand Down
31 changes: 3 additions & 28 deletions lib/screens/welcome_screen.dart
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import 'package:livekit_client/livekit_client.dart' as sdk;
import 'package:provider/provider.dart';
import 'package:url_launcher/url_launcher.dart' show launchUrl;
import '../controllers/app_ctrl.dart' as ctrl;
import '../widgets/agent_status_indicator.dart';
import '../widgets/button.dart' as buttons;

class WelcomeScreen extends StatelessWidget {
Expand Down Expand Up @@ -50,34 +51,8 @@ class WelcomeScreen extends StatelessWidget {
],
),
),
// Agent listening indicator
Consumer<sdk.Session>(
builder: (ctx, session, child) => AnimatedOpacity(
opacity: session.agent.canListen ? 1.0 : 0.0,
duration: const Duration(milliseconds: 300),
child: Container(
padding: const EdgeInsets.symmetric(vertical: 8, horizontal: 16),
child: Row(
mainAxisSize: MainAxisSize.min,
children: [
Icon(
Icons.mic,
color: Colors.green,
size: 18,
),
const SizedBox(width: 8),
const Text(
'Agent is listening',
style: TextStyle(
color: Colors.green,
fontWeight: FontWeight.w500,
),
),
],
),
),
),
),
// Agent status indicator
const AgentStatusIndicator(),
Consumer2<ctrl.AppCtrl, sdk.Session>(
builder: (ctx, appCtrl, session, child) {
final isProgressing =
Expand Down
Loading
Loading