Skip to content

Add Debug-only MCP integration for AI-assisted UI inspection#26

Open
TastyHeadphones wants to merge 2 commits into
LookInsideApp:mainfrom
TastyHeadphones:claude/eager-robinson-c731e7
Open

Add Debug-only MCP integration for AI-assisted UI inspection#26
TastyHeadphones wants to merge 2 commits into
LookInsideApp:mainfrom
TastyHeadphones:claude/eager-robinson-c731e7

Conversation

@TastyHeadphones
Copy link
Copy Markdown

@TastyHeadphones TastyHeadphones commented May 14, 2026

Summary

Adds an optional MCP server, lookinside-mcp, so AI agents (Claude Desktop, Claude Code, Codex CLI, Cursor, Windsurf, VS Code, …) can inspect a running Debug build's UI through LookInside — hierarchy, screenshots, search, highlight, layout/accessibility diagnostics, and a one-shot bug report.

What's new

Two new SPM units, layered:

  • LookinMCPCore (Swift library) — headless inspection client built on LookinCore. Includes a HierarchyProvider seam with two implementations: LiveLookinClient (Peertalk over loopback to a running LookinServer) and FileHierarchyProvider (reads .lookin snapshot files for offline analysis). Also contains the canonical JSON shapes, hierarchy index, element search, layout/accessibility diagnostics, bug-report builder, and a SecureTextRedactor that strips secure-text-field contents at the model boundary.
  • lookinside-mcp (executable) — stdio MCP server using modelcontextprotocol/swift-sdk. One file per tool, registered via a small ToolRegistry, so adding a tool later is one new conformance.

11 tools shipped: health_check, list_apps, current_screen, get_hierarchy, search_elements, get_element, capture_screenshot, highlight_element, diagnose_layout, diagnose_accessibility, export_bug_report.

Architecture

AI agent ↔ MCP client ↔ lookinside-mcp ─ Peertalk (47164–47179) ─► LookinServer (Debug)
                              │
                              └── LookinMCPCore (JSON shapes, search, diagnostics)

The MCP server is a parallel consumer of LookinServer alongside the macOS LookInside.app. The license handshake in LKConnectionManager.m is purely client-side — Sources/LookinServer/Server/Connection/LKS_RequestHandler.m has no license check — so a separate Debug tooling client can speak the protocol directly without dragging the auth helper into a CLI.

Safety / Debug-only

  • Release builds don't embed LookinServer, so there's nothing to talk to.
  • No write-side request types are wired in: no arbitrary selector invocation, no method dispatch, no shell exec.
  • Secure text field contents (UITextField.isSecureTextEntry, NSSecureTextField) are redacted by SecureTextRedactor inside JSONShape.node so every current and future tool inherits the protection.
  • stdio transport only; no HTTP listener.

Notes & limitations

  • Bumps the SwiftPM macOS minimum from 11 to 13, required by the swift-sdk transport. SPM has no per-target deployment target; this is the cleanest option. iOS/tvOS minimums are unchanged.
  • highlight_element requires a new server-side request type that doesn't exist yet; it returns { ok: false, reason: "…" } for now rather than silently lying. Tracked as a follow-up.
  • No tap/scroll/type/property-mutation tools in v1 by design — easy to add later via the same ToolRegistry pattern.

Install + try

./Scripts/build-mcp-server.sh
./build/lookinside-mcp health
./build/lookinside-mcp print-config claude-desktop  # or claude-code, codex, cursor, windsurf, vscode

Then paste the snippet (or run the printed command) into your client's MCP config and ask:

Inspect the current screen and tell me what UI issues you see.

Test plan

  • swift build --product lookinside-mcp — clean
  • swift test — 15/15 tests pass (hierarchy index, JSON shape, secure-text redaction, element search, layout/a11y diagnostics, provider error paths)
  • ./build/lookinside-mcp healthno_target path renders correctly with nonzero exit
  • ./build/lookinside-mcp print-config <client> — emits absolute-path snippets for all 6 supported clients
  • End-to-end stdio MCP handshake — initializenotifications/initializedtools/list returns all 11 tools with valid JSON Schema
  • Connect from Claude Desktop / Claude Code / Codex against a Simulator Debug build with LookinServer embedded (requires user environment)
  • Verify export_bug_report against a real screen
  • Verify the offline path: lookinside-mcp serve --snapshot some.lookin from a captured archive

Docs

Introduce `lookinside-mcp`, an optional MCP server that lets AI agents
inspect a running Debug build's UI through the same Peertalk plumbing
the macOS app uses — hierarchy, search, element details, screenshots,
highlight, layout/accessibility diagnostics, and a one-shot bug report.

Two new SPM units: LookinMCPCore (headless inspection client, JSON
shaping, diagnostics, secure-text redaction) and lookinside-mcp
(executable using modelcontextprotocol/swift-sdk over stdio). Reuses
LookinCore data models and the existing in-app LookinServer; license
gate is bypassed because it is enforced client-side, not by the
in-process server.
Codex CLI and Claude Code both use the canonical `<cli> mcp add <name>
[--env K=V] -- <command> [args]` form. The previous claude-code snippet
omitted the `--` separator, which works in practice but isn't the
documented syntax. Aligns both clients on the same shape and documents
the env-var passthrough.
@TastyHeadphones TastyHeadphones force-pushed the claude/eager-robinson-c731e7 branch 2 times, most recently from aac150e to d432066 Compare May 14, 2026 02:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant