Add Debug-only MCP integration for AI-assisted UI inspection#26
Open
TastyHeadphones wants to merge 2 commits into
Open
Add Debug-only MCP integration for AI-assisted UI inspection#26TastyHeadphones wants to merge 2 commits into
TastyHeadphones wants to merge 2 commits into
Conversation
Introduce `lookinside-mcp`, an optional MCP server that lets AI agents inspect a running Debug build's UI through the same Peertalk plumbing the macOS app uses — hierarchy, search, element details, screenshots, highlight, layout/accessibility diagnostics, and a one-shot bug report. Two new SPM units: LookinMCPCore (headless inspection client, JSON shaping, diagnostics, secure-text redaction) and lookinside-mcp (executable using modelcontextprotocol/swift-sdk over stdio). Reuses LookinCore data models and the existing in-app LookinServer; license gate is bypassed because it is enforced client-side, not by the in-process server.
Codex CLI and Claude Code both use the canonical `<cli> mcp add <name> [--env K=V] -- <command> [args]` form. The previous claude-code snippet omitted the `--` separator, which works in practice but isn't the documented syntax. Aligns both clients on the same shape and documents the env-var passthrough.
aac150e to
d432066
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an optional MCP server,
lookinside-mcp, so AI agents (Claude Desktop, Claude Code, Codex CLI, Cursor, Windsurf, VS Code, …) can inspect a running Debug build's UI through LookInside — hierarchy, screenshots, search, highlight, layout/accessibility diagnostics, and a one-shot bug report.What's new
Two new SPM units, layered:
LookinMCPCore(Swift library) — headless inspection client built onLookinCore. Includes aHierarchyProviderseam with two implementations:LiveLookinClient(Peertalk over loopback to a runningLookinServer) andFileHierarchyProvider(reads.lookinsnapshot files for offline analysis). Also contains the canonical JSON shapes, hierarchy index, element search, layout/accessibility diagnostics, bug-report builder, and aSecureTextRedactorthat strips secure-text-field contents at the model boundary.lookinside-mcp(executable) — stdio MCP server using modelcontextprotocol/swift-sdk. One file per tool, registered via a smallToolRegistry, so adding a tool later is one new conformance.11 tools shipped:
health_check,list_apps,current_screen,get_hierarchy,search_elements,get_element,capture_screenshot,highlight_element,diagnose_layout,diagnose_accessibility,export_bug_report.Architecture
The MCP server is a parallel consumer of
LookinServeralongside the macOS LookInside.app. The license handshake inLKConnectionManager.mis purely client-side —Sources/LookinServer/Server/Connection/LKS_RequestHandler.mhas no license check — so a separate Debug tooling client can speak the protocol directly without dragging the auth helper into a CLI.Safety / Debug-only
LookinServer, so there's nothing to talk to.UITextField.isSecureTextEntry,NSSecureTextField) are redacted bySecureTextRedactorinsideJSONShape.nodeso every current and future tool inherits the protection.Notes & limitations
macOSminimum from 11 to 13, required by the swift-sdk transport. SPM has no per-target deployment target; this is the cleanest option. iOS/tvOS minimums are unchanged.highlight_elementrequires a new server-side request type that doesn't exist yet; it returns{ ok: false, reason: "…" }for now rather than silently lying. Tracked as a follow-up.ToolRegistrypattern.Install + try
./Scripts/build-mcp-server.sh ./build/lookinside-mcp health ./build/lookinside-mcp print-config claude-desktop # or claude-code, codex, cursor, windsurf, vscodeThen paste the snippet (or run the printed command) into your client's MCP config and ask:
Test plan
swift build --product lookinside-mcp— cleanswift test— 15/15 tests pass (hierarchy index, JSON shape, secure-text redaction, element search, layout/a11y diagnostics, provider error paths)./build/lookinside-mcp health—no_targetpath renders correctly with nonzero exit./build/lookinside-mcp print-config <client>— emits absolute-path snippets for all 6 supported clientsinitialize→notifications/initialized→tools/listreturns all 11 tools with valid JSON SchemaLookinServerembedded (requires user environment)export_bug_reportagainst a real screenlookinside-mcp serve --snapshot some.lookinfrom a captured archiveDocs
healthinterpretation, common errors