da-mcp — Crossplatform Desktop Automation MCP Server

A Model Context Protocol (MCP) server that lets AI agents (OpenCode, Claude Desktop, etc.) interact with a local desktop environment: screenshots, OCR with UI-element classification, mouse/keyboard control, and program launch — on Linux, macOS, and Windows.

For AI agents and automated installers: Do NOT chain apt install … && npm install && npm run build by hand. Use the bundled installer scripts — they handle the platform-specific prerequisites (tesseract, xdotool, Node 22+) and surface native-binding / TCC failures as actionable errors instead of silent broken builds. See Install (automated / AI agents) below.

Features

20 tools registered under the da_* namespace:

Capture

• da_screenshot — Capture full screen or a specific display as PNG.
• da_ocr — Run OCR (Tesseract) on a screenshot and return structured text + UI element classification.
• da_list_displays — List connected displays with id, bounds, scale factor.

Input

• da_get_mouse_position — Read current cursor position (Linux X11 uses xdotool getmouselocation --shell, Wayland uses ydotool, Windows uses PowerShell + user32!GetCursorPos). macOS is stubbed in v1.0.0 — surfaces a "not implemented" error (tracked by #19).
• da_move_mouse — Move the cursor to (x, y).
• da_click — Click at (x, y) with optional button (left/right/middle/back/forward) and count.
• da_click_text — OCR-then-click: find a UI element by visible text (exact or fuzzy) and click its center. Returns NOT_FOUND if no match.
• da_find_text — Same OCR+match pipeline as da_click_text but stops before the click — returns the bbox/center/confidence so the agent can decide what action to take (click vs. drag vs. right-click).
• da_double_click — Convenience wrapper for double-click.
• da_drag — Drag from (x1, y1) to (x2, y2).
• da_draw_path — Trace a multi-point mouse path with optional Modifier[] held throughout (try/finally guarantees modifier cleanup). Used for freeform shapes (circles, signatures) and for constrained drawing in Paint (modifiers:["shift"]).
• da_scroll — Scroll wheel at (x, y) by (dx, dy).
• da_type — Type a string at the current focus.
• da_key — Press a single key or chord (e.g. Ctrl+C).

Stability / verification

• da_wait_for_window — Poll da_window_list until a window with a matching title appears (substring/exact/regex). Use after da_launch to wait for the app to paint before clicking inside.
• da_wait_for_text — Poll the OCR text-match pipeline until text appears on screen. Use after any state-changing action to confirm the new state is visible before continuing.
• da_verify_pixels — Poll the screen until a pixel-level predicate holds: {kind:"color", rgb, minCount} (count matching pixels) or {kind:"diff", baseline, threshold} (fraction differing from a baseline PNG). E.g. wait until 200+ red pixels appear on the canvas region after drawing a circle in Paint.

Launch

• da_launch — Launch a program by name or path; returns a spawn handle with PID + POSIX signal exit codes (SIGINT=130, SIGTERM=143, SIGHUP=129, SIGKILL=137, SIGQUIT=131, SIGABRT=134).

Window

• da_window_list — Enumerate all visible top-level windows (hwnd/pid/title/bounds/visibility). Cross-platform: wmctrl (Linux X11 + Wayland via XWayland), osascript + System Events (macOS), PowerShell + user32!EnumWindows (Windows).
• da_window_focus — Bring a window to the foreground by hwnd, pid, or title match (exact / regex / substring, case-insensitive). Title matching uses pure-JS resolver; multi-window Paint-style flows return a NOT_FOUND error when nothing matches.

Skills (for OpenCode / Claude Desktop agents)

This repo ships a generic desktop-orchestration skill that any agent can install to drive the 16 da_* tools through a 6-step loop: Orient → Observe → Locate → Act → Verify → Iterate. It is application-agnostic — Paint, browsers, IDEs, dialogs, file managers, native apps.

Source-of-truth location: docs/skills/da-ui-orchestrator.md. The file is deliberately kept out of .opencode/skills/ so that this repo doesn't auto-load as an OpenCode skill on its own — it stays a portable artefact that you copy into your client.

Install the skill on your machine:

# OpenCode:
mkdir -p ~/.config/opencode/skills/da-ui-orchestrator
cp docs/skills/da-ui-orchestrator.md ~/.config/opencode/skills/da-ui-orchestrator/SKILL.md

# Claude Code:
mkdir -p ~/.agents/skills/da-ui-orchestrator
cp docs/skills/da-ui-orchestrator.md ~/.agents/skills/da-ui-orchestrator/SKILL.md

Restart your MCP client — da-ui-orchestrator will appear in available_skills. After git pull on this repo, re-copy the file to refresh.

UI element classification (OCR post-processing)

The da_ocr classifier tags each detected text region with one of:

Category	Examples
button	"OK", "Cancel", "Apply"
input	Text fields, search boxes
label	Static descriptive text
checkbox	"☑ Enable", "☐ Dark mode"
radio	"◉ Local", "○ Network"
menu	Top-level menu headers ("File", "Edit")
menu-item	Dropdown entries ("New", "Open…")
icon	Toolbar / sidebar icons

Architecture

• Language: TypeScript 7.0 (strict, ESM, Node 22+); exactOptionalPropertyTypes, noUncheckedIndexedAccess, noFallthroughCasesInSwitch all on. Exact version pins (no ^/~).
• MCP SDK: v2 (@modelcontextprotocol/server@2.0.0) over StdioServerTransport (production) and InMemoryTransport (tests).
• Module layout (250 LOC ceiling per file):
  • Screenshot — src/screenshot/{png,backends,index,types}.ts. PNG validation/encoding isolated in png.ts; backend dispatch (screenshot-desktop → OS CLI shell-out: scrot/grim/screencapture/PowerShell BitBlt) in backends.ts. No native NAPI binary — see #12.
  • OCR — src/ocr/{cli,index,mock,parse,wasm,types,classify,classify-rules}.ts. CLI backend (runCli), WASM fallback (runWasm), parser, mock; orchestrator in index.ts rethrows as OCR_FAILED when both backends fail.
  • Input — src/input/{routing,mouse,keyboard,scroll,drag,types,index}.ts plus per-OS backends mouse-{macos,windows}.ts, keyboard-{macos,windows}.ts, scroll-{macos,windows}.ts, clipboard.ts. Shared routing helpers (runCli, resolveRouting, requireTool, isMockMode, validateCoords, Routing) in routing.ts. Linux paths shell out to xdotool/ydotool/wtype. Windows path uses PowerShell + user32 (keybd_event, mouse_event, SetCursorPos, GetCursorPos); Unicode text goes via clipboard + Ctrl+V. macOS is a stub in v1.0.0 (tracked by #19) — surfaces a "not implemented in #13" error. No native NAPI binary — see #13.
  • Launch — src/launch/{launch,types}.ts. open(1) + child_process.spawn (shell:false); SIGNAL_EXIT_CODES map for POSIX signal mapping.
  • Platform — src/platform/{detect,types}.ts. detectPlatform() returns { os, display, tools, home }; assertPlatformSupported() throws PLATFORM_INIT_FAILED on unsupported combos.
  • Server — src/server.ts. Registers 20 tools, wraps handler results into CallToolResult with structuredContent (Buffers stripped to number[] for JSON-safety), installs SIGINT/SIGTERM shutdown.
  • Server instructions — src/server-instructions.ts. Exports SERVER_INSTRUCTIONS, a string surfaced to the AI agent via the MCP instructions field (MCP spec, ServerOptions.instructions). Tells the agent it IS the orchestrator — call the 14 da_* tools directly through the MCP client, do NOT write an orchestrator script that imports/spawns the server. Edit this string to update the agent-facing announcement.
  • Window — src/window/{types,list,list-linux,list-macos,list-windows,focus,resolve,index}.ts. Per-OS list backends (wmctrl / osascript / PowerShell+EnumWindows) keep each file under the 250 LOC ceiling; pure-JS matchOne resolver in resolve.ts for title matching.

Backend dispatch

Capability	Primary	Fallback 1	Fallback 2
Screenshot (Linux X11)	screenshot-desktop (X11)	scrot	—
Screenshot (Linux Wayland)	screenshot-desktop (XWayland portal)	grim	—
Screenshot (macOS)	screenshot-desktop (CG via wrapper)	screencapture	—
Screenshot (Windows)	screenshot-desktop (GDI wrapper)	PowerShell BitBlt (windowsCliBackend, ships in backends.ts)	—
Input (Linux X11)	xdotool CLI	—	—
Input (Linux Wayland)	ydotool CLI	wtype (keyboard only)	—
Input (Windows)	PowerShell + user32 (keybd_event, mouse_event, SetCursorPos, GetCursorPos); Unicode text via clipboard + Ctrl+V	—	—
Input (macOS)	Stub in v1.0.0 — surfaces "not implemented in #13" error (tracked by #19)	—	—
OCR (any OS)	tesseract CLI	tesseract.js@7 WASM	—
Window list + focus (Linux)	wmctrl CLI (X11 + XWayland on Wayland)	—	—
Window list + focus (macOS)	osascript + System Events	—	—
Window list + focus (Windows)	PowerShell + user32!EnumWindows / SetForegroundWindow	—	—

Every spawnSync/spawn call uses shell:false. Permission-gated errors (e.g. macOS ScreenCaptureKit, Windows access denied) are detected via a tightened pattern (screen…permission, screencapturekit, access is denied) and re-thrown as DaMcpError('PERMISSION_DENIED').

Install (automated / AI agents)

If you are an AI agent setting this up on behalf of a user, or running scripted/CI installation, use the bundled installer scripts. Do not improvise a manual npm install chain — manual install bypasses the prompts that catch missing system tools and PATH issues, and turns platform-specific gotchas into silent broken builds.

OS	Command
Linux	sudo ./scripts/install-system-deps.sh && npm ci && npm run build && DA_MCP_TEST_MODE=mock npm test
macOS	./scripts/install-macos.sh
Windows	powershell -ExecutionPolicy Bypass -File scripts/install-windows.ps1

What each script does:

• Verifies / installs system prerequisites — tesseract, xdotool/ydotool (Linux), Node.js 22+ (Xcode CLT only needed if Homebrew is missing it — handled via brew install if so)
• Runs npm ci — locked, reproducible install. Avoid npm install (which resolves ranges and is slower)
• Builds TypeScript with npm run build
• Runs DA_MCP_TEST_MODE=mock npm test so the build is verified before you declare success
• Prints the MCP client config snippet to drop into Claude Desktop / OpenCode / etc.

If a script fails, read its output — the next step is printed at the end of each failure path. Do not retry by hand without first understanding what the script detected.

For manual / sandboxed installs where you cannot run the scripts, see Install below.

Install (Windows, single-binary)

For Windows users, a self-contained single-binary release is available — no Node.js, no npm install, no build step. The binary embeds Node 22 + the bundled JavaScript via Node SEA (scripts/build-sea.sh); the recommended system dependency is tesseract on $PATH (for fast OCR — see OCR backend fallback below).

# Download latest release asset from GitHub
curl.exe -L -o da-mcp.exe https://github.com/cioinside/da-mcp/releases/latest/download/da-mcp-win32-x64.exe

# First run — prints CLI help and exits 0
.\da-mcp.exe help

# Run stdio MCP server (default — point your MCP client at this binary)
.\da-mcp.exe

# Optional: install Tesseract OCR CLI for fast OCR (auto-elevates via UAC)
.\da-mcp.exe install-tesseract

Caveats:

• Unsigned binary — postject strips Authenticode when injecting the SEA blob, so Windows SmartScreen will warn on first launch. Click "More info" → "Run anyway".
• No native NAPI deps — screenshot-desktop, MCP SDK, zod, tesseract.js are all inlined in the binary. Only tesseract is recommended (for fast OCR; not required — see OCR backend fallback).
• Linux + macOS binaries are not part of the v1.0.0 release — see BUILD.md for the local cross-platform build path (Windows SEA build runs on windows-latest CI only).

OCR backend fallback

da_ocr tries backends in order; the first one that succeeds is used:

Order	Backend	Speed	Requires	Used when
1	Tesseract CLI (tesseract on $PATH)	~0.5–2 s / screenshot	tesseract installed	Recommended path
2	tesseract.js WASM (in-binary, with pre-bundled eng.traineddata)	~5–15 s / screenshot	Nothing — runs offline	Tesseract not installed
3	tesseract.js WASM (downloads traineddata)	~15–30 s on first call, then ~5–15 s	Internet on first call	tesseract.js fallback if no pre-bundled data

If all backends fail, da_ocr returns OCR_FAILED with a multi-line remediation hint pointing you to the install command for your platform.

Install Tesseract for fast OCR (recommended):

OS	Command
Windows	winget install UB-Mannheim.TesseractOCR (or choco install tesseract). For the single-binary release, you can also run .\da-mcp.exe install-tesseract — it auto-detects winget/choco and re-launches itself elevated via UAC if needed.
macOS	brew install tesseract
Linux	apt install tesseract-ocr (Debian/Ubuntu) / dnf install tesseract (Fedora) / pacman -S tesseract (Arch)

Configure the tessdata cache directory with DA_MCP_TESSDATA_DIR (default ./tessdata — relative to the process CWD).

For source installs (Linux/macOS/Windows dev workflow), continue to Install below.

Install

# System dependencies (apt/dnf/brew; see scripts/install-system-deps.sh)
sudo ./scripts/install-system-deps.sh

# npm deps
npm install

# Build
npm run build

# Verify type-check (strict mode)
npm run typecheck

# Run all tests (mock mode — skips real native calls)
DA_MCP_TEST_MODE=mock npm test

Upgrading

da-mcp upgrade is a single CLI command that self-updates whichever way you installed it — the entry point auto-detects binary vs source mode (process.execPath === process.argv[1]), so the same command works for both the single-binary release and a source checkout:

# Binary install (Windows single-binary release):
.\da-mcp.exe upgrade

# Source install (Linux/macOS/Windows dev workflow):
node /projects/da-mcp/dist/server-dispatch.js upgrade
# or:
npm run upgrade

Both modes accept --force (alias -f). Pass it to reinstall even when the version comparison says you're up to date, or (in source mode) to discard uncommitted local changes.

Binary mode (da-mcp.exe upgrade)

For a Node SEA single-binary install (e.g. Windows da-mcp-win32-x64.exe):

1. Query GitHub Releases — GET https://api.github.com/repos/cioinside/da-mcp/releases/latest returns the latest non-prerelease release with its asset list and sha256 digests.
2. Compare versions — the embedded build-time constant (process.env.DA_MCP_VERSION, injected by esbuild --define in scripts/build-sea.sh) is compared against tag_name. If the running version is already ≥ the release, the command is a no-op (unless --force).
3. Pick the matching asset — da-mcp-{platform}-{arch}[.exe] for the current process.platform + process.arch.
4. Download to ${execPath}.new.<ts> (sibling of the running binary, never overwriting it in place).
5. Verify sha256 if the asset has a digest: sha256:… field — mismatches abort before any rename.
6. Atomic replace — rename the running binary to ${execPath}.old.<ts> (Windows allows renaming a running executable; the process keeps its file handle), then rename the staged file to execPath. On failure, the staged file is left behind and the original is untouched.
7. Restart the service if one is registered via install-service (see below). If no service is installed, prints a one-line reminder to restart your MCP client manually.

The previous binary is kept at ${execPath}.old.<ts> so the operator can roll back manually if the new binary misbehaves on first launch.

Source mode (npm run upgrade)

For a source checkout (any OS):

1. Refuse dirty trees — git status --porcelain must be empty unless --force is passed.
2. git fetch origin <branch> + git reset --hard origin/<branch> — fast-forward to the latest commit on the current branch.
3. npm ci — locked, reproducible dependency install.
4. npm run build — TypeScript compile to dist/.
5. npm run typecheck — strict-mode smoke check.
6. Restart the service if one is registered, or print a reminder to restart the MCP client manually.

The command refuses to run on a detached HEAD — check out a branch first.

Run da-mcp as a system service (auto-restart)

For long-running installations, da-mcp registers as a managed service so that upgrade can bounce it without manual intervention:

# Install (one-shot, needs root / Administrator):
node /projects/da-mcp/dist/server-dispatch.js install-service

# Uninstall later:
node /projects/da-mcp/dist/server-dispatch.js uninstall-service

OS	Service type	Restart command used by upgrade
Linux	da-mcp.service (systemd, user unit)	systemctl --user restart da-mcp
macOS	com.da-mcp.daemon (launchd)	launchctl kickstart -k gui/$(id -u)/com.da-mcp.daemon
Windows	da-mcp (SCM service, runs under LocalSystem)	sc stop da-mcp && sc start da-mcp

Templates live in scripts/systemd/, scripts/launchd/, and scripts/windows/. Service installation requires elevated privileges (sudo / Run as Administrator). The default transport after install-service is HTTP with token auth (so multiple MCP clients can share one daemon); use DA_MCP_TRANSPORT=stdio if you prefer per-client stdio.

Run

stdio (default)

The server speaks MCP over stdio. Configure your MCP client to launch node /projects/da-mcp/dist/server-dispatch.js (or npx tsx src/server-dispatch.ts for dev).

HTTP (opt-in, token-protected)

Set DA_MCP_TRANSPORT=http to expose the server on http://0.0.0.0:3000/<token>. A 256-bit random token is generated on first start and persisted at:

OS	Token path
Linux	$XDG_CONFIG_HOME/da-mcp/token or ~/.config/da-mcp/token
macOS	~/Library/Application Support/da-mcp/token
Windows	%APPDATA%\da-mcp\token

The token file is created with mode 0o600 (owner-only). Rotate it any time:

node /projects/da-mcp/dist/server-dispatch.js token regenerate
# → http://0.0.0.0:3000/<43-char-base64url-token>
# (substitute the host's LAN IP for 0.0.0.0 when configuring the remote client)

Override defaults with env vars:

• DA_MCP_HTTP_HOST — bind address (default 0.0.0.0 — LAN-reachable, token-gated); supports IPv4, IPv6 ([::1]), and hostname
• DA_MCP_PORT — port (default 3000)
• DA_MCP_TOKEN_PATH — override token storage path

The URL is a bearer-style token — anyone with the token can call tools (mouse, keyboard, screenshot, launch). Default 0.0.0.0 bind means the daemon is reachable from any host that can route to this machine (LAN, VPN, public IP). The token is the sole auth — its 256-bit entropy is unguessable, but treat it as a password: protect the token file, and rotate it (token regenerate) if it may have leaked. To restrict the bind to the loopback interface only, set DA_MCP_HTTP_HOST=127.0.0.1 — the server prints a one-line confirmation at startup.

Remote access from another host on the LAN

Because the default DA_MCP_HTTP_HOST=0.0.0.0 already listens on all interfaces, no special launcher is needed:

DA_MCP_TRANSPORT=http npm start
# → server boots, binds 0.0.0.0:3000, prints URL with token to stderr

On the remote machine, configure your MCP client with http://<lan-ip>:3000/<token> — replace 0.0.0.0 with the host's actual LAN IP (hostname -I, ipconfig getifaddr en0, ipconfig).

Open the host firewall for inbound TCP on DA_MCP_PORT (default 3000) once per OS — this requires elevation and varies per platform:

OS	Command
Linux (firewalld)	sudo firewall-cmd --add-port=3000/tcp --permanent && sudo firewall-cmd --reload
Linux (ufw)	sudo ufw allow 3000/tcp
macOS	System Settings → Network → Firewall → allow incoming for the node binary (or turn off the application firewall)
Windows (PowerShell, admin)	New-NetFirewallRule -Direction Inbound -LocalPort 3000 -Protocol TCP -Action Allow -DisplayName "da-mcp"

OpenCode / Claude Desktop example config

stdio (default — per-client process)

{
  "mcpServers": {
    "da-mcp": {
      "command": "node",
      "args": ["/projects/da-mcp/dist/server-dispatch.js"],
      "env": {
        "DISPLAY": ":0",
        "DA_MCP_LOG": "info"
      }
    }
  }
}

HTTP (token-protected — share one daemon across clients / hosts)

Start the server with DA_MCP_TRANSPORT=http (see HTTP section above for bind/port/token details), then grab the URL it printed at startup — or regenerate the token any time with node /projects/da-mcp/dist/server-dispatch.js token regenerate. Paste the result into the url field below (substitute the server host's LAN IP for 0.0.0.0 when configuring a remote client).

OpenCode (~/.config/opencode/opencode.json):

{
  "mcpServers": {
    "da-mcp": {
      "type": "remote",
      "url": "http://<host>:<port>/<token>"
    }
  }
}

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "da-mcp": {
      "url": "http://<host>:<port>/<token>"
    }
  }
}

Cross-platform notes

OS	Screenshot	Input	Notes
Linux X11	screenshot-desktop (X11)	xdotool	wmctrl for window list/focus (installed by install-system-deps.sh)
Linux Wayland	screenshot-desktop (XWayland portal)	ydotool (daemon)	wmctrl works via XWayland if XWayland apps are present
macOS	screenshot-desktop (CG via wrapper)	Stub in v1.0.0 — see #19; Windows SEA binary is the recommended path	First screenshot call may need Screen Recording permission (TCC)
Windows	screenshot-desktop (GDI wrapper); PowerShell BitBlt fallback	PowerShell + user32 (keybd_event, mouse_event, SetCursorPos)	The v1.0.0 single-binary release target — no native NAPI deps

Development

# Strict type-check (no emit)
npx tsc --noEmit

# All tests in mock mode (CI default)
DA_MCP_TEST_MODE=mock npx vitest run

# Single test file
npx vitest run test/unit/screenshot.test.ts

# Watch mode
npx vitest

Test inventory

• 28 unit test files + 2 e2e (e2e skip in mock mode)
• 540 tests passing / 18 skipped / 0 failed in DA_MCP_TEST_MODE=mock npm test (e2e require real X11/tesseract; input dispatcher tests cover per-OS stubs for macOS + Windows PowerShell paths). Post-tool additions: da_find_text, da_wait_for_window, da_wait_for_text, da_verify_pixels, install-tesseract CLI subcommand bring the total to 540 passing.
• Test runtime: process.env['DA_MCP_TEST_MODE'] === 'mock' short-circuits native calls; _mock.ts modules inject deterministic native modules

Conventions

• 250 LOC ceiling per file (measured as non-blank, non-comment lines: awk '!/^[[:space:]]*$/ && !/^[[:space:]]*(\/\/|#|--)/' <file> | wc -l)
• ESM imports use .js suffix even for .ts source
• All spawn* calls with shell: false
• All native errors wrapped in DaMcpError with typed code from ErrorCode union
• Public surface re-exported from src/screenshot/index.ts and src/input/index.ts — consumers import from there, not from per-operation files
• Forbidden: as any, @ts-ignore, @ts-expect-error, console.log, shell: true, auto-commits

Environment variables

• DISPLAY — X11 display (Linux only)
• WAYLAND_DISPLAY — Wayland display socket
• DA_MCP_LOG — log level (trace|debug|info|warn|error), default info
• DA_MCP_TESSERACT_BIN — path to tesseract binary, default tesseract
• DA_MCP_OCR_BACKEND — cli (default) or wasm
• DA_MCP_TEST_MODE — mock skips real native calls in tests; e2e tests skip when set
• DA_MCP_SCREENSHOT_BACKEND — force a screenshot backend (node-screenshots | screenshot-desktop | windows-cli); default auto-detect
• DA_MCP_TRANSPORT — stdio (default) or http; http enables the opt-in HTTP transport
• DA_MCP_PORT — HTTP port when DA_MCP_TRANSPORT=http (default 3000)
• DA_MCP_HTTP_HOST — HTTP bind address (default 0.0.0.0 — LAN-reachable, token-gated); supports IPv4, IPv6, hostname
• DA_MCP_TOKEN_PATH — override the auth token storage path

License

MIT