Terminal autocomplete for shell prompts and Claude Code TUI#790
Terminal autocomplete for shell prompts and Claude Code TUI#790akramj13 wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
Adds an opt-in “Terminal Autocomplete (Beta)” pipeline that enables Cotabby suggestions in terminal environments using cooperative shell hooks over a local Unix-domain socket, with a fail-closed Claude Code TUI OCR fallback. This fits Cotabby’s architecture by introducing terminal-specific source adapters and keeping the existing suggestion coordinator as the central orchestration point, while avoiding unreliable AX text reads in terminal surfaces unless an authoritative source is active.
Changes:
- Introduces a secure local IPC server + bundled shell hook resources (zsh/bash/fish) to stream prompt buffer + cursor state into Cotabby.
- Adds terminal/TUI focus arbitration and OCR-based Claude Code prompt extraction, publishing authoritative focus snapshots into the existing suggestion pipeline.
- Adds terminal-specific prompting, normalization, and acceptance behavior (paste-based insertion; whole-line command replacement mode for natural-language instructions).
Reviewed changes
Copilot reviewed 59 out of 59 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| TERMINAL_AUTOCOMPLETE.md | User-facing setup + security/privacy/limitations documentation for terminal autocomplete. |
| README.md | Adds terminal autocomplete feature mention + link to detailed doc. |
| project.yml | Bundles shell-integration scripts into app resources. |
| scripts/shell-integration/cotabby.zsh | zsh hook to report ZLE buffer/cursor over Unix socket. |
| scripts/shell-integration/cotabby.bash | bash hook + readline key wrapping for per-keystroke buffer reporting (Bash 4+). |
| scripts/shell-integration/cotabby.fish | fish hook + key bindings wrapping for buffer reporting. |
| Cotabby/App/Core/CotabbyAppEnvironment.swift | Composes terminal subsystem (IPC + OCR + focus arbitration) and wires it into the coordinator. |
| Cotabby/App/Core/AppDelegate.swift | Starts/stops terminal integration with the opt-in setting and screen-recording permission changes. |
| Cotabby/App/Coordinators/SuggestionCoordinator.swift | Adds terminal/TUI observer hooks and terminal scheduling state. |
| Cotabby/App/Coordinators/SuggestionCoordinator+Input.swift | Adjusts prediction scheduling to avoid polling terminal AX and to feed TUI OCR observer. |
| Cotabby/App/Coordinators/SuggestionCoordinator+Prediction.swift | Adds terminal replacement mode handling (no streaming for replacements; fail-closed validation). |
| Cotabby/App/Coordinators/SuggestionCoordinator+Acceptance.swift | Implements terminal paste insertion mode + whole-line terminal replacement acceptance. |
| Cotabby/App/Coordinators/SuggestionCoordinator+Lifecycle.swift | Threads terminal integration gating into lifecycle evaluator calls. |
| Cotabby/Models/FocusModels.swift | Extends focus snapshot with terminal metadata + terminal role helpers + focus sequence retagging. |
| Cotabby/Models/SuggestionModels.swift | Adds request/result mode and terminal replacement SuggestionKind plumbing. |
| Cotabby/Models/SuggestionSubsystemContracts.swift | Extends insertion contract for terminal paste + whole-line terminal replacement. |
| Cotabby/Models/SuggestionSettingsData.swift | Reframes integrated-terminal toggle as terminal autocomplete master opt-in (source-verified). |
| Cotabby/Models/SuggestionSettingsModel.swift | Updates setting documentation for terminal opt-in semantics. |
| Cotabby/Models/TerminalIntegrationPaths.swift | Defines per-product socket/hook filesystem layout (incl. sun_path constraints). |
| Cotabby/Models/TerminalFocusModels.swift | Adds terminal IPC message model + authoritative terminal focus snapshot/state types. |
| Cotabby/Models/TerminalAwareFocusModel.swift | Arbitrates between AX focus and authoritative terminal (hook/OCR) focus for suggestions. |
| Cotabby/Services/Suggestion/ContextBuffer.swift | Ensures terminal identity changes (elementIdentifier/role) advance generation safely. |
| Cotabby/Services/Suggestion/SuggestionInserter.swift | Adds terminal paste mode + ordered backspace-then-paste whole-line replacement with shared clipboard restore. |
| Cotabby/Services/Suggestion/SuggestionOverlayPresenter.swift | Hides overlay until terminal caret geometry is non-zero/trustworthy. |
| Cotabby/Services/Terminal/TerminalIntegrationService.swift | Implements IPC session model + bounded same-user socket server + message validation. |
| Cotabby/Services/Terminal/TerminalIntegrationResourceInstaller.swift | Installs bundled shell hooks into per-product Application Support with private permissions. |
| Cotabby/Services/Terminal/TerminalGeometryResolver.swift | Reads terminal window frame/title via AX for OCR coordinate mapping and cheap TUI signals. |
| Cotabby/Services/Terminal/TerminalWindowScreenshotService.swift | Captures frontmost terminal window (or embedded pane region) via ScreenCaptureKit for OCR. |
| Cotabby/Services/Terminal/ShellPromptGeometryCoordinator.swift | Performs debounced OCR to anchor shell prompt row once, then tracks caret arithmetically. |
| Cotabby/Services/Terminal/TuiContextReader.swift | Reduces OCR output to Claude Code prompt text + cursor estimate using fail-closed heuristics. |
| Cotabby/Services/Terminal/TuiContextCoordinator.swift | Owns debounced/permissioned Claude Code OCR lifecycle and injects snapshots into effective focus. |
| Cotabby/Services/Runtime/OpenAICompatibleSuggestionEngine.swift | Threads request mode through result reporting. |
| Cotabby/Services/Runtime/LlamaSuggestionEngine.swift | Threads request mode through result reporting. |
| Cotabby/Services/Runtime/FoundationModelSuggestionEngine.swift | Threads request mode through result reporting. |
| Cotabby/Support/FoundationModelPromptRenderer.swift | Adds terminal-specific instructions/prompt shapes for Foundation Models path. |
| Cotabby/Support/ProcessTreeInspector.swift | Adds lightweight Darwin process ancestry inspection for TUI detection. |
| Cotabby/Support/SuggestionAvailabilityEvaluator.swift | Adds terminal integration gating (opt-in + authoritative live source required). |
| Cotabby/Support/SuggestionRequestFactory.swift | Adds terminal request mode selection + terminal prompt rendering and token floor for replacements. |
| Cotabby/Support/SuggestionSessionReconciler.swift | Invalidates sessions when terminal source identity changes. |
| Cotabby/Support/SuggestionTextNormalizer.swift | Adds terminal replacement scaffolding removal (fences/Command:/$). |
| Cotabby/Support/TerminalAppDetector.swift | Expands terminal host detection to embedded-terminal hosts + terminal host helper. |
| Cotabby/Support/TerminalCommandIntentPolicy.swift | Deterministic classifier for “natural language command replacement” mode. |
| Cotabby/Support/TerminalCompletionPromptRenderer.swift | Terminal-specific base-model prompt shaping (shell continuation vs Claude Code vs replacement). |
| Cotabby/Support/TerminalFocusAdapter.swift | Adapts hook-sourced terminal snapshots into FocusedInputSnapshot form. |
| Cotabby/Support/TerminalPromptAnchor.swift | OCR prompt anchoring model + caret geometry calculation utilities. |
| Cotabby/Support/TuiSessionDetector.swift | Cheap pre-OCR classifier for Claude Code via title/process signals. |
| Cotabby/Support/TuiFocusAdapter.swift | Adapts OCR-sourced Claude Code prompt into FocusedInputSnapshot form. |
| Cotabby/Services/Visual/ScreenTextExtractor.swift | Publishes OCR lines with geometry (positionedLines) alongside existing hygiene pipeline output. |
| Cotabby/UI/Settings/SettingsIndex.swift | Renames/retargets settings label/description/keywords for terminal autocomplete. |
| Cotabby/UI/Settings/Panes/AppsPaneView.swift | Adds opt-in toggle UI + shell setup command copying UI with quoting. |
| CotabbyTests/TerminalAppDetectorTests.swift | Updates expected disable reason string for terminal gating. |
| CotabbyTests/SuggestionAvailabilityEvaluatorTests.swift | Updates gating expectations: opt-in alone doesn’t authorize raw terminal AX. |
| CotabbyTests/SuggestionStateHelperTests.swift | Ensures terminal session identity participates in generation advancement. |
| CotabbyTests/SuggestionCoordinatorTestSupport.swift | Updates test inserter to record insertion modes. |
| CotabbyTests/SuggestionCoordinatorAcceptanceTests.swift | Adds tests for terminal paste insertion and whole-line command replacement acceptance. |
| CotabbyTests/SyntheticReplacePlannerTests.swift | Adds tests ensuring terminal replacement ordering is delete-then-paste and never submits commands. |
| CotabbyTests/TerminalIntegrationModelTests.swift | Adds pure-model tests for terminal offsets, replacement intent, prompting, and resource install perms. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| } catch { | ||
| logger.error("Terminal integration failed to start: \(error.localizedDescription)") | ||
| server = nil | ||
| isRunning = false | ||
| return false | ||
| } |
| var bytes = [UInt8](repeating: 0, count: 4_096) | ||
| while true { | ||
| let count = Darwin.read(descriptor, &bytes, bytes.count) | ||
| if count > 0 { | ||
| clients[descriptor]?.buffer.append(contentsOf: bytes[0..<count]) | ||
| guard processFrames(for: descriptor) else { |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| guard suggestionInserter.replaceTerminalLine( | ||
| deletingCharacterCount: originalText.count, | ||
| with: session.fullText | ||
| ) else { |
There was a problem hiding this comment.
When a terminal command-replacement prompt contains emoji or another multi-unit character, this path deletes originalText.count graphemes even though the inserter sends repeated Backspace events. The terminal can leave part of the original instruction in the buffer before pasting the generated command, so the user may execute a corrupted command after pressing Return.
| guard suggestionInserter.replaceTerminalLine( | |
| deletingCharacterCount: originalText.count, | |
| with: session.fullText | |
| ) else { | |
| guard suggestionInserter.replaceTerminalLine( | |
| deletingCharacterCount: originalText.utf16.count, | |
| with: session.fullText | |
| ) else { |
| for event in events { | ||
| event.post(tap: .cgAnnotatedSessionEventTap) |
There was a problem hiding this comment.
This whole-line terminal replacement posts Backspace and Cmd-V events at .cgAnnotatedSessionEventTap, while the other synthetic insert and replace paths use .cghidEventTap and register the expected synthetic keydown burst. If the input monitor only observes the HID tap path, the synthetic Cmd-V or Backspace events can be treated as real input during the accept callback, which can re-enter acceptance handling or interleave with the user's held Tab while the prompt is being deleted and pasted.
Terminal Autocomplete USAGE / REVIEWER GUIDE (Beta)Cotabby can provide inline autocomplete in dedicated terminal apps and integrated terminal panes. Enable and set up
Settings generates paths for the exact app build being used. This matters during development: zsh is recommended. It uses How it worksShell prompts and terminal TUIs use separate source adapters:
Both sources feed the existing suggestion coordinator. The normal accept shortcuts still apply; At an end-of-line shell prompt, a short imperative English instruction such as Security and privacy boundaries
Current limitations
|
Closes #762
Summary
This PR brings Cotabby's inline autocomplete to terminal environments
while preserving the reliability improvements introduced in #656.
Instead of relying on Accessibility APIs inside terminals, this
implementation uses cooperative shell integration over a local
Unix-domain socket. When shell integration is unavailable, Claude Code
can fall back to OCR-based prompt detection. Suggestions are only
enabled when a verified live integration source exists, so unsupported
terminals continue behaving exactly as they do today.
The feature is opt-in and disabled by default.
What's included
Shell integration
socket.
directly from Settings.
Secure local IPC
0700runtime directory0600Unix socketTerminal autocomplete
active.
source is available.
Accessibility text insertion.
acceptance.
Claude Code support
unavailable.
the active terminal.
Natural language command translation
Terminal prompts that are clearly natural-language instructions are
treated as command translation requests rather than autocomplete
continuations.
For example:
becomes
or
becomes
Accepting the suggestion replaces the entire line but does not execute
the command. The user must still press Return manually.
Normal shell commands (
git status,cd,ls, flags, pipes,redirects, etc.) continue using standard inline autocomplete.
Settings
Adds a new opt-in Terminal Autocomplete (Beta) setting.
The Settings UI includes commands for enabling shell integration in zsh,
bash, and fish.
Testing
Verified:
Static validation:
git diff --checkNotes
A bug discovered during implementation was also fixed:
sun_pathlength limit (~104 bytes). Theruntime socket now lives in the user's private temporary directory
while installation resources remain in Application Support,
preventing IPC failures on long home-directory paths.
Another regression fixed during review ensured terminal command
replacement is atomic. Previously, queued backspaces could race with
paste, causing the accepted command to be partially deleted. Replacement
now always deletes the original input before pasting the generated
command.
Greptile Summary
This PR adds opt-in terminal autocomplete support. The main changes are:
Confidence Score: 4/5
The terminal command replacement path needs a fix before merging.
Cotabby/App/Coordinators/SuggestionCoordinator+Acceptance.swift; Cotabby/Services/Suggestion/SuggestionInserter.swift
Important Files Changed
Sequence Diagram
%%{init: {'theme': 'neutral'}}%% sequenceDiagram participant Shell as Shell integration participant IPC as TerminalIntegrationService participant Focus as Terminal focus model participant Coord as SuggestionCoordinator participant Insert as SuggestionInserter Shell->>IPC: Prompt and cursor update IPC->>Focus: Verified terminal snapshot Focus->>Coord: Focused input context Coord->>Coord: Build autocomplete or command replacement Coord->>Insert: Accept via paste or line replacement Insert->>Shell: Backspace burst then paste Coord->>IPC: Optimistic terminal buffer update%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%% sequenceDiagram participant Shell as Shell integration participant IPC as TerminalIntegrationService participant Focus as Terminal focus model participant Coord as SuggestionCoordinator participant Insert as SuggestionInserter Shell->>IPC: Prompt and cursor update IPC->>Focus: Verified terminal snapshot Focus->>Coord: Focused input context Coord->>Coord: Build autocomplete or command replacement Coord->>Insert: Accept via paste or line replacement Insert->>Shell: Backspace burst then paste Coord->>IPC: Optimistic terminal buffer updateReviews (1): Last reviewed commit: "fix(terminal): fix command replacement o..." | Re-trigger Greptile