feat: Add voice command support - #147
Open
khedkaravani-rgb wants to merge 3 commits into
Open
Conversation
🤖 CodeAnt AI — Review Status
|
github-actions
Bot
added
ECSoC26
codeant-ai
Bot
added
the
size:XL
📝 WalkthroughWalkthroughEarthTwin updates orbital rendering and Cesium lifecycle management, adds direct hover/click picking and bookmark navigation, and integrates browser speech recognition with voice commands. The HUD, legend, metrics, and bookmark controls are also revised. ChangesEarthTwin interaction and voice controls
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Operator
participant EarthTwin
participant SpeechRecognition
participant VoiceCommands
participant CesiumViewer
Operator->>EarthTwin: Start voice control
EarthTwin->>SpeechRecognition: Start listening
SpeechRecognition->>EarthTwin: Return transcript
EarthTwin->>VoiceCommands: Execute transcript
VoiceCommands->>EarthTwin: Invoke matched action
EarthTwin->>CesiumViewer: Fly camera or update view
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![codeant-ai[bot]](https://avatars.githubusercontent.com/in/646884?s=60&v=4){kind=small}
Comment on lines
+505
to
+513
| const handleVoiceCommand = useCallback( | ||
| (transcript: string) => { | ||
| executeVoiceCommand(transcript, { | ||
| trackISS: handleTrackISS, | ||
| showDebris: handleShowDebris, | ||
| openCollisionCenter: handleOpenCollisionCenter, | ||
| zoomToIndia: handleZoomToIndia, | ||
| toggleSpaceWeather: handleToggleSpaceWeather, | ||
| }); |
There was a problem hiding this comment.
Suggestion: The result returned by executeVoiceCommand is discarded, so the command handler never exposes its success or failure message. The UI only displays the raw transcript and recognition errors; users receive no command execution feedback despite the command executor producing a result. Capture the result and render or toast its message. [incomplete implementation]
Severity Level: Major ⚠️
- ⚠️ Recognized commands provide no execution confirmation.
- ⚠️ Unrecognized commands provide no diagnostic message.
- ⚠️ Accessibility feedback is limited to raw transcripts.Steps of Reproduction ✅
1. Start voice recognition through `EarthTwin` at
`frontend/src/components/EarthTwin.tsx:834-849`.
2. The speech hook receives a recognized transcript and calls `onCommand(spokenText)` at
`frontend/src/hooks/useSpeechRecognition.ts:95-99`.
3. `handleVoiceCommand` invokes `executeVoiceCommand` at
`frontend/src/components/EarthTwin.tsx:505-513`, but does not assign or render its
returned `VoiceCommandResult`.
4. The executor creates success and failure messages at
`frontend/src/lib/voiceCommands.ts:26-29`, `frontend/src/lib/voiceCommands.ts:51-54`, and
`frontend/src/lib/voiceCommands.ts:82-85`; the component renders only the raw transcript
and recognition error at `frontend/src/components/EarthTwin.tsx:859-869`, so command
success or “not recognized” feedback is never shown.
(Use Cmd/Ctrl + Click for best experience)
Prompt for AI Agent 🤖
This is a comment left during a code review.
**Path:** frontend/src/components/EarthTwin.tsx
**Line:** 505:513
**Comment:**
*Incomplete Implementation: The result returned by `executeVoiceCommand` is discarded, so the command handler never exposes its success or failure message. The UI only displays the raw transcript and recognition errors; users receive no command execution feedback despite the command executor producing a result. Capture the result and render or toast its message.
Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 7
🧹 Nitpick comments (6)
frontend/src/components/EarthTwin.tsx (3)
336-337: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueMove
autoRotateRefabove its first consumer.
autoRotateRefis declared at Line 337 but referenced byhandleTrackISS(Line 164) andhandleZoomToIndia(Line 196). It works today because those bodies only execute after render, but a future eager call during render would hit a TDZ error. Hoist it next to the other refs at Lines 141-144.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@frontend/src/components/EarthTwin.tsx` around lines 336 - 337, Move the autoRotateRef declaration from its current location to the existing refs section near the top of the component, before handleTrackISS and handleZoomToIndia reference it. Keep the ref initialized to true and remove the original declaration to avoid duplicates.
242-291: 🚀 Performance & Scalability | 🔵 Trivial | ⚖️ Poor tradeoffFull
removeAll()+ rebuild every 5 minutes discards conjunction polylines and churns entities.
viewer.entities.removeAll()at Line 246 wipes everything, then ~500 point entities plus polylines are recreated. This is a visible flicker and a GC spike on each refresh. Consider updatingposition/point.coloron the existing entities keyed bycatalog_numberand only adding/removing the delta.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@frontend/src/components/EarthTwin.tsx` around lines 242 - 291, Replace the full-rebuild flow around viewer.entities.removeAll() with incremental synchronization keyed by obj.catalog_number: reuse existing entities by updating their position and point styling, add entities for new objects, and remove entities no longer present. Preserve conjunction polylines and other unrelated viewer entities, and ensure stale entries are removed from entityMap without clearing the entire viewer collection.
93-118: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueLocally redefined color/size helpers duplicate the shared visualization module.
These were previously imported. Keeping a local copy means legend, tooltip, and any other consumer can drift apart. Consider re-exporting from the shared module instead.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@frontend/src/components/EarthTwin.tsx` around lines 93 - 118, Remove the local CATEGORY_COLORS, getPointSize, and getOutlineWidth definitions from EarthTwin and reuse the corresponding exports from the shared visualization module. Update imports or re-exports as needed so the legend, tooltip, and visualization consumers all use the single shared implementation.frontend/src/lib/voiceCommands.ts (1)
45-48: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRedundant alternatives.
command.includes("collision center")already matches"open collision center", andcommand.includes("space weather")already matches"toggle space weather". The second clause in each pair is dead.Also applies to: 70-73
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@frontend/src/lib/voiceCommands.ts` around lines 45 - 48, Remove the redundant “open collision center” alternative from the collision-center condition and the redundant “toggle space weather” alternative from the corresponding space-weather condition in the voice command matching logic. Retain the broader “collision center” and “space weather” includes checks so both command forms continue to match.frontend/src/hooks/useSpeechRecognition.ts (2)
61-71: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win
onCommandin the dependency array tears down andabort()s recognition on every identity change.Any consumer that passes an inline callback will recreate the
SpeechRecognitioninstance on each render, aborting an in-flight session. KeeponCommandin a ref and drop it from the deps so the instance is created once.♻️ Proposed refactor
+ const onCommandRef = useRef(onCommand); + + useEffect(() => { + onCommandRef.current = onCommand; + }, [onCommand]); + useEffect(() => { if (!isSupported) { return; }setTranscript(spokenText); - onCommand(spokenText); + onCommandRef.current(spokenText);- }, [isSupported, onCommand]); + }, [isSupported]);Also applies to: 116-116
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@frontend/src/hooks/useSpeechRecognition.ts` around lines 61 - 71, Update the speech recognition effect in useSpeechRecognition so onCommand is stored in a ref and read from that ref by recognition callbacks, rather than included as an effect dependency. Remove onCommand from the dependency array so the SpeechRecognition instance is created once and in-flight recognition is not aborted when callback identity changes.
3-44: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUse the Web Speech DOM types for recognition
Add
@types/dom-speech-recognitionand import the standardSpeechRecognition,SpeechRecognitionEvent, andSpeechRecognitionErrorEventtypes instead of the shallow*Likeinterfaces. Local augmentations can still add thewebkitprefix, butVoiceRecognitionLikeis unnecessary because the event object shape comes from the DOM types.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@frontend/src/hooks/useSpeechRecognition.ts` around lines 3 - 44, Replace the local SpeechRecognitionEventLike, SpeechRecognitionErrorEventLike, and SpeechRecognitionInstance type definitions in useSpeechRecognition with the standard SpeechRecognition, SpeechRecognitionEvent, and SpeechRecognitionErrorEvent types from `@types/dom-speech-recognition`, and add that dependency. Retain only the global Window augmentation needed for webkitSpeechRecognition, using the standard constructor type; remove the unnecessary VoiceRecognitionLike-style aliases and update handlers to use the imported DOM event types.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@frontend/src/components/EarthTwin.tsx`:
- Around line 157-179: Update handleTrackISS to locate the ISS entity in
entitiesRef using catalog number 25544, then fly to or track that entity instead
of the hardcoded coordinates. Preserve the existing viewer validity guard and
rotation behavior, and correct the callback body indentation.
- Around line 850-872: Move the voice feedback overlay out of the flex controls
row and position it relative to the intended voice-control container so it
appears near the button without affecting control spacing. Add role="status" and
aria-live="polite" to the overlay container so listening updates, transcripts,
and errors are announced to screen readers; preserve the existing conditional
rendering and visual messages.
- Around line 181-215: Update handleShowDebris to navigate to the declared
dashboard debris route, /dashboard/debris, instead of /Debris; leave the other
navigation handlers unchanged.
- Around line 956-961: Remove the duplicate BookmarkModal render gated by
isBookmarkModalOpen near the earlier JSX block, and retain the keyed
BookmarkModal render that handles editingBookmark. Ensure opening the modal
mounts only the editing-aware dialog.
In `@frontend/src/hooks/useSpeechRecognition.ts`:
- Around line 118-125: Update startListening to wrap
recognitionRef.current.start() in error handling, catching synchronous
InvalidStateError from rapid repeated starts and passing the caught failure to
setError instead of allowing it to escape the click handler. Preserve the
existing guard and clear any prior error before attempting to start recognition.
- Around line 85-99: Update the speech recognition effect around
recognition.onresult to store the transcript-clear timeout in a ref, clear any
existing timer before scheduling a new one, and schedule it after
setTranscript(spokenText). Clear the active timeout during the effect cleanup to
prevent state updates after unmount.
In `@frontend/src/lib/voiceCommands.ts`:
- Around line 14-86: Update handleVoiceCommand in EarthTwin.tsx to capture the
VoiceCommandResult returned by executeVoiceCommand and surface its message
through the HUD overlay, including the failure message for unrecognized
commands. Preserve the existing raw transcript display and successful command
execution behavior.
---
Nitpick comments:
In `@frontend/src/components/EarthTwin.tsx`:
- Around line 336-337: Move the autoRotateRef declaration from its current
location to the existing refs section near the top of the component, before
handleTrackISS and handleZoomToIndia reference it. Keep the ref initialized to
true and remove the original declaration to avoid duplicates.
- Around line 242-291: Replace the full-rebuild flow around
viewer.entities.removeAll() with incremental synchronization keyed by
obj.catalog_number: reuse existing entities by updating their position and point
styling, add entities for new objects, and remove entities no longer present.
Preserve conjunction polylines and other unrelated viewer entities, and ensure
stale entries are removed from entityMap without clearing the entire viewer
collection.
- Around line 93-118: Remove the local CATEGORY_COLORS, getPointSize, and
getOutlineWidth definitions from EarthTwin and reuse the corresponding exports
from the shared visualization module. Update imports or re-exports as needed so
the legend, tooltip, and visualization consumers all use the single shared
implementation.
In `@frontend/src/hooks/useSpeechRecognition.ts`:
- Around line 61-71: Update the speech recognition effect in
useSpeechRecognition so onCommand is stored in a ref and read from that ref by
recognition callbacks, rather than included as an effect dependency. Remove
onCommand from the dependency array so the SpeechRecognition instance is created
once and in-flight recognition is not aborted when callback identity changes.
- Around line 3-44: Replace the local SpeechRecognitionEventLike,
SpeechRecognitionErrorEventLike, and SpeechRecognitionInstance type definitions
in useSpeechRecognition with the standard SpeechRecognition,
SpeechRecognitionEvent, and SpeechRecognitionErrorEvent types from
`@types/dom-speech-recognition`, and add that dependency. Retain only the global
Window augmentation needed for webkitSpeechRecognition, using the standard
constructor type; remove the unnecessary VoiceRecognitionLike-style aliases and
update handlers to use the imported DOM event types.
In `@frontend/src/lib/voiceCommands.ts`:
- Around line 45-48: Remove the redundant “open collision center” alternative
from the collision-center condition and the redundant “toggle space weather”
alternative from the corresponding space-weather condition in the voice command
matching logic. Retain the broader “collision center” and “space weather”
includes checks so both command forms continue to match.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 83b0affc-98e3-47c8-93b8-0dd10a758ee7
📒 Files selected for processing (3)
frontend/src/components/EarthTwin.tsxfrontend/src/hooks/useSpeechRecognition.tsfrontend/src/lib/voiceCommands.ts
13 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
User description
Summary
This PR adds voice command support to the globe dashboard, allowing users to control key navigation and globe actions using spoken commands.
Changes
useSpeechRecognitionhook.This improves accessibility and provides a more intuitive hands-free way to interact with the strategic globe dashboard.
Related Issue
Fixes #139
Type of Change
Screenshots / Screen Recordings
Screen.Recording.2026-07-27.231310.mp4
Testing Performed
Breaking Changes
N/A
Checklist
ECSoC26 Submission
ECSoC26-L1– BeginnerECSoC26-L2– IntermediateECSoC26-L3– AdvancedCodeAnt-AI Description
Add hands-free globe controls and saved views to the orbital dashboard
What Changed
Impact
✅ Hands-free globe navigation✅ Saved and shareable orbital views✅ Clearer satellite selection and day/night visibility💡 Usage Guide
Checking Your Pull Request
Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.
Talking to CodeAnt AI
Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:
This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.
Example
Preserve Org Learnings with CodeAnt
You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:
This helps CodeAnt AI learn and adapt to your team's coding style and standards.
Example
Retrigger review
Ask CodeAnt AI to review the PR again, by typing:
Check Your Repository Health
To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.
Summary by CodeRabbit
Greptile Summary
This PR adds browser speech recognition and connects spoken phrases to dashboard navigation and Cesium camera actions.
Confidence Score: 2/5
This PR is not safe to merge until the stacked bookmark dialogs, false ISS-tracking behavior, and removed keyboard globe controls are fixed.
The current code still mounts two bookmark dialogs under one state flag, labels a flight to fixed zero coordinates as ISS tracking, and replaces the accessible globe interaction system with mouse-only handlers.
Files Needing Attention: frontend/src/components/EarthTwin.tsx, frontend/src/lib/voiceCommands.ts
Important Files Changed
Sequence Diagram
Reviews (3): Last reviewed commit: "Fix iss command" | Re-trigger Greptile