Add Landmarks app: A multimodal trip planner using Gemini Live by peterfriese · Pull Request #250 · FirebaseExtended/firebase-video-samples

peterfriese · 2026-07-02T13:57:29Z

This PR introduces the new Landmarks app, a functional prototype demonstrating a live conversation feature using video and audio powered by Gemini.

Key Features:

Multimodal Live Conversation: Talk to an AI travel guide with real-time audio and video context.
Optimized for Performance:
- Zero Choke Ups: Implemented an automatic cache purge on startup and silenced high-frequency logging to prevent MDB_MAP_FULL storage errors.
- Low Latency: Optimized for European users by routing through europe-west1 (Belgium).
- Immediate Visuals: The model receives visual context immediately upon camera activation.
- Balanced Throughput: Adjusted audio buffer sizes (16k) and video resolution (VGA, 0.5 FPS, 0.4 quality) for a stable, high-performance demo.
Robust UI Controls: Fully functional Mute and Pause controls that stay in sync with background media loops.
Intelligent Speech Handling: Disabled barge-in and restored transcription continuity for a more natural conversational flow.

…xamples.foundationmodels.TripPlanner

- Disabled AI barge-in by forcing silence when AI is speaking. - Fixed UI buttons (Mute, Pause) by using escaping closures for state. - Reduced audio packet frequency (4k -> 16k buffer) to lower data pressure. - Reduced video overhead: VGA resolution, 0.5 FPS, 0.4 JPEG quality. - Implemented immediate first-frame delivery for visual context. - Optimized for Europe by switching backend to europe-west1. - Silenced high-frequency console logs to prevent lag. - Implemented auto-cache purge on startup to prevent MDB_MAP_FULL errors.

gemini-code-assist

Code Review

This pull request introduces a comprehensive Trip Planner iOS application showcasing guided generation and tool calling with Google's Gemini models via Firebase AI Logic. It includes features like live itinerary planning, real-time audio/video streaming, and Google Maps/Search integration. The code review feedback identifies several critical improvements, including fixing barge-in silence logic, offloading synchronous disk operations to a background thread during app launch, marking ModelData as @mainactor for Swift 6 concurrency safety, and ensuring proper connection state updates when the live session ends. Additionally, optimizations were suggested to remove dead code, redundant delays, and SwiftUI view hierarchy anti-patterns.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

gemini-code-assist · 2026-07-02T14:00:29Z

+        if isUserSpeaking {
+            await session.sendAudioRealtime(pcmData)
+        } else if !isAISpeaking {
+            let silence = Data(count: pcmData.count)
+            await session.sendAudioRealtime(silence)
+        }


The current barge-in prevention logic does not send silence when isAISpeaking is true because of the else if !isAISpeaking guard on line 60. Instead, it sends nothing. To actually force silence transmission when the AI is speaking (as described in the PR description), you should send silence whenever the user is not speaking.

Suggested change

if isUserSpeaking {

await session.sendAudioRealtime(pcmData)

} else if !isAISpeaking {

let silence = Data(count: pcmData.count)

await session.sendAudioRealtime(silence)

}

if isUserSpeaking {

await session.sendAudioRealtime(pcmData)

} else {

let silence = Data(count: pcmData.count)

await session.sendAudioRealtime(silence)

}

gemini-code-assist · 2026-07-02T14:00:30Z

+    private func clearFirebaseCache() {
+        let fileManager = FileManager.default
+        guard let cacheDir = fileManager.urls(for: .cachesDirectory, in: .userDomainMask).first else { return }
+
+        let foldersToClear = ["google-sdks-events", "google-app-measurement"]
+
+        for folder in foldersToClear {
+            let folderUrl = cacheDir.appendingPathComponent(folder)
+            if fileManager.fileExists(atPath: folderUrl.path) {
+                try? fileManager.removeItem(at: folderUrl)
+                print("DEBUG: Cleared internal Firebase storage at: \(folder)")
+            }
+        }
+    }


Executing synchronous file system operations (clearFirebaseCache()) on the main thread during app launch can block the main thread and potentially cause watchdog crashes if the directory is large or the disk is slow. It is highly recommended to run these operations asynchronously on a background queue.

private func clearFirebaseCache() { DispatchQueue.global(qos: .background).async { let fileManager = FileManager.default guard let cacheDir = fileManager.urls(for: .cachesDirectory, in: .userDomainMask).first else { return } let foldersToClear = ["google-sdks-events", "google-app-measurement"] for folder in foldersToClear { let folderUrl = cacheDir.appendingPathComponent(folder) if fileManager.fileExists(atPath: folderUrl.path) { try? fileManager.removeItem(at: folderUrl) print("DEBUG: Cleared internal Firebase storage at: \\(folder)") } } } }

gemini-code-assist · 2026-07-02T14:00:30Z

+@Observable
+class ModelData {


The ModelData class is not isolated to @MainActor, but its properties (such as itineraryPlanners) are accessed and mutated from the main actor (UI). This is a data race hazard in Swift 6. Marking the entire class as @MainActor ensures thread safety and conforms to Swift 6 concurrency requirements.

Suggested change

@Observable

class ModelData {

@Observable

@MainActor

class ModelData {

gemini-code-assist · 2026-07-02T14:00:30Z

+            // 3. Handshake and Processing
+            state = .connected
+            await sendInitialGreeting(session: session)
+            try await startProcessingResponses()


When startProcessingResponses() returns normally (e.g., when the connection is closed by the server), the connect() method exits without updating the state to .disconnected. This causes the UI to still show "Live" or "Connected" even though the session is dead. You should call await disconnect() when the processing loop finishes.

Suggested change

try await startProcessingResponses()

try await startProcessingResponses()

await disconnect()

gemini-code-assist · 2026-07-02T14:00:30Z

+        let startTime = Date()
+        Logging.general.log("FindPointsOfInterestMapsTool: call called with query: \(arguments.naturalLanguageQuery), category: \(arguments.pointOfInterest.rawValue)")
+
+        let model = FirebaseAI.firebaseAI(backend: .vertexAI(location: "global"))


The PR description states that the backend was switched to europe-west1 (Belgium) for lower latency in Europe. However, FindPointsOfInterestMapsTool still uses "global" as the location for Vertex AI. This should be updated to "europe-west1" to maintain consistency and optimize latency.

Suggested change

let model = FirebaseAI.firebaseAI(backend: .vertexAI(location: "global"))

let model = FirebaseAI.firebaseAI(backend: .vertexAI(location: "europe-west1"))

gemini-code-assist · 2026-07-02T14:00:30Z

+    @ObservationIgnored
+    nonisolated(unsafe) private var isExtractingFrames = false
+    @ObservationIgnored
+    nonisolated(unsafe) private var lastExtractedFrame: UIImage?


The private variable isExtractingFrames is written to but never read, making it dead code. Additionally, since CameraManager is @MainActor, lastExtractedFrame is already isolated to the main actor and does not need the nonisolated(unsafe) annotation. Please remove isExtractingFrames and its writes (on lines 129 and 136), and remove the unsafe annotation from lastExtractedFrame.

Suggested change

@ObservationIgnored

nonisolated(unsafe) private var isExtractingFrames = false

@ObservationIgnored

nonisolated(unsafe) private var lastExtractedFrame: UIImage?

@ObservationIgnored

private var lastExtractedFrame: UIImage?

gemini-code-assist · 2026-07-02T14:00:30Z

+                                Rectangle()
+                                    .fill(Color.clear)
+                                    .frame(height: 200)
+                                    .overlay(
+                                        AsyncImage(url: url) { image in
+                                            image
+                                                .resizable()
+                                                .scaledToFill()
+                                        } placeholder: {
+                                            ProgressView("Loading image...")
+                                                .frame(maxWidth: .infinity, maxHeight: .infinity)
+                                                .background(Color.gray.opacity(0.1))
+                                        }
+                                    )
+                                    .clipped()
+                                    .clipShape(RoundedRectangle(cornerRadius: 12))
+                                    .padding([.horizontal, .bottom], 4)
+                            }


Wrapping AsyncImage in an .overlay of a clear Rectangle with a fixed height is an anti-pattern in SwiftUI that unnecessarily complicates the view hierarchy. You can apply the frame, clipping, and padding directly to the AsyncImage component.

AsyncImage(url: url) { image in image .resizable() .scaledToFill() } placeholder: { ProgressView("Loading image...") .frame(maxWidth: .infinity, maxHeight: .infinity) .background(Color.gray.opacity(0.1)) } .frame(height: 200) .clipped() .clipShape(RoundedRectangle(cornerRadius: 12)) .padding([.horizontal, .bottom], 4) }

gemini-code-assist · 2026-07-02T14:00:30Z

+        if FirebaseApp.app() == nil {
+            try await Task.sleep(nanoseconds: 1_000_000_000)
+            guard self.connectionId == currentId else { throw ApplicationError("Cancelled") }
+            throw ApplicationError("Firebase missing GoogleService-Info.plist")
+        }


The 1-second sleep in checkFirebaseConfiguration is redundant and unnecessarily delays error reporting when Firebase is not configured. The error should be thrown immediately.

if FirebaseApp.app() == nil { throw ApplicationError("Firebase missing GoogleService-Info.plist") }

peterfriese added 4 commits July 2, 2026 12:10

Add Apple Foundation Models trip planner sample

0ed886d

Update README.md with Firebase setup instructions and bundle ID

c78004a

Update project settings and configure bundle ID com.google.firebase.e…

fb83582

…xamples.foundationmodels.TripPlanner

gemini-code-assist Bot reviewed Jul 2, 2026

View reviewed changes

peterfriese changed the title ~~Optimize Live Itinerary for performance and stability~~ Add Landmarks app: A multimodal trip planner using Gemini Live Jul 2, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add Landmarks app: A multimodal trip planner using Gemini Live#250

Add Landmarks app: A multimodal trip planner using Gemini Live#250
peterfriese wants to merge 4 commits into
mainfrom
peterfriese/landmarks

peterfriese commented Jul 2, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot Jul 2, 2026

Uh oh!

gemini-code-assist Bot Jul 2, 2026

Uh oh!

gemini-code-assist Bot Jul 2, 2026

Uh oh!

gemini-code-assist Bot Jul 2, 2026

Uh oh!

gemini-code-assist Bot Jul 2, 2026

Uh oh!

gemini-code-assist Bot Jul 2, 2026

Uh oh!

gemini-code-assist Bot Jul 2, 2026

Uh oh!

gemini-code-assist Bot Jul 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	try await startProcessingResponses()
	try await startProcessingResponses()
	await disconnect()

	let model = FirebaseAI.firebaseAI(backend: .vertexAI(location: "global"))
	let model = FirebaseAI.firebaseAI(backend: .vertexAI(location: "europe-west1"))

Uh oh!

Conversation

peterfriese commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot Jul 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

peterfriese commented Jul 2, 2026 •

edited

Loading