Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 22 additions & 11 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,14 +74,26 @@ The MCP server returns a flat element list with `nodeId`, `parentId` and `depth`

Annotations send text only. Each note includes the user request, element label, test ID when present, React component and owner names when available, and nearby text. React debug creation stacks resolve through one bounded request to the matching Metro server's `/symbolicate` endpoint. Source maps supply an app file, line, column and function so the agent can open the element's JSX directly. This is the element creation site, not necessarily the component definition. Library frames and unresolved bundle locations never become edit locations. Missing source maps leave selection working and mark the source location unavailable. Coordinates remain a fallback. The paused image stays local for editing; use Screenshot separately to attach an image.

## Native device requests

When a mobile task has ambiguous targets, `mobile_choose_devices` asks the user
through Codex's native inline request form. The server lists this tool only for
clients that advertise the `openai/elicitation` form extension; other hosts ask in
chat. The agent discovers suitable devices
first, then supplies their IDs, platform and kind, a task-specific question, and
optional operation details. The form shows canonical device names, platform/type,
## Devices card and device requests

`mobile_show_devices` is the one user-facing device tool. Without `selectionMode`
it lists every discovered simulator, emulator and physical device in an inline
card, grouped as running, physical, then stopped by runtime. The user can boot or
stop simulators and emulators from the card and send selected devices to chat. A
backend that cannot list devices (no Android SDK, older Xcode) leaves the others
listed with a note. The tool's text result is a short summary so the model does
not repeat the list; hosts without app cards receive the list as text instead.

When a mobile task has ambiguous targets, the agent sets `selectionMode` with a
task-specific `message`. Hosts that advertise the `openai/elicitation` form
extension (Codex) get the native inline request form described below, and the card
then shows only the outcome. Hosts that render MCP app cards get the same card with
only selectable candidates and a Use button that sends the choice to chat; this
call returns immediately and tells the model to wait for that reply. Hosts with
neither return an error so the agent asks in chat. Candidate `devices` are
optional: omitted, the tool discovers them itself, and the form offers at most 20,
running devices first. The agent may instead supply discovered IDs, platform and
kind, plus optional operation details. The form shows canonical device names, platform/type,
runtime, optional verified app labels, and platform phone illustrations. Use
`selectionMode: "multiple"` to let the user choose several devices. No device is
preselected.
Expand All @@ -93,9 +105,8 @@ apps, open streams or start recordings. Stopped simulators/AVDs can be offered f
build tasks; profiling candidates should already be running. Physical iOS uses
the hardware UDID and requires a connected paired device.

This local MCP connection uses the OpenAI Extensions SDK's native form elicitation.
The host must advertise `extensions["openai/elicitation"].form`; unsupported hosts
return an explicit error. Codex owns layout and button labels, so the native form
The form uses the OpenAI Extensions SDK's native form elicitation and requires
`extensions["openai/elicitation"].form`. Codex owns layout and button labels, so the native form
does not reproduce custom footer buttons from a design mockup. Thumbnails are
illustrations, not captured app screens.

Expand Down
2 changes: 1 addition & 1 deletion docs/mcp-tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
| `mobile_open_simulator` | Open the native panel, optionally targeting a platform, device kind, or discovered device ID |
| `mobile_open_workspace` | Open fullscreen with the same optional device targeting |
| `mobile_list_simulators` | Start the bundled backend if needed and list devices |
| `mobile_choose_devices` | Ask through the native inline form for one or several verified task targets |
| `mobile_show_devices` | Show devices in an inline card with Boot/Stop, or ask the user to choose task targets through the native form or the card |
| `mobile_list_ios_devices` | Discover physical iPhones and iPads with USB/Wi-Fi, pairing state, UDID, and CoreDevice ID |
| `mobile_start_baguette` | Retry or reconnect the bundled backend |
| `mobile_boot_simulator` | Boot one listed device |
Expand Down
18 changes: 15 additions & 3 deletions docs/telemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -309,6 +309,14 @@ Counters record range selection, run toggles and Ask actions. No recording IDs,
titles, selections or device measurements enter this telemetry. The comparison
MCP operation retains the existing sampled server trace and handled-error path.

Devices cards use the `devices` surface and view with shared readiness, interaction,
frame pacing and teardown coverage. `ui.devices.process` measures result validation
in milliseconds, and `ui.devices.message_ack` ends when the host acknowledges the
Use message. Gauges report the listed and selected device counts. Counters record
`ui.devices.boot`, `ui.devices.stop`, their `_failed` variants and
`ui.devices.send`; boot and stop calls also keep the shared `ui.action` /
`ui.action.result` counters and sampled MCP client spans. Device names, IDs, runtimes, questions and chat text are never sent.

`ui.annotations.tree_processing` measures local element processing in milliseconds, including React Native nodes when available. Since 0.1.66, normal inspection validates flat records here; server-side tree flattening falls within `ui.annotations.inspection`, which measures the MCP inspection round trip, including native accessibility and optional Metro work. `ui.annotations.runtime_available` counts snapshots with runtime elements. `ui.annotations.inspection_fallback` counts native-tool retries. `ui.annotations.inspection_truncated` counts snapshots that reach the collector's work or measurement limits. Inspection timing includes failed calls and retries. It uses the current simulator surface and the same bounded timing windows as other UI measurements. Tree contents and selected regions are not sent to Sentry.

Since 0.1.69, inspection timing also includes the bounded Metro source-map lookup. `ui.annotations.source_available` counts snapshots with at least one resolved source location; `ui.annotations.message_build` measures text construction for Send to chat. Source-map transport failures use the existing server error handler with a fixed message. Source paths, component names, creation stacks, note text and images are never sent to Sentry.
Expand Down Expand Up @@ -397,11 +405,15 @@ mechanisms; they do not by themselves identify the cause of an ADB stall.

Unhandled JavaScript errors and rejected promises, React render errors, and handled MCP tool failures produce issues. Expected stopped-device errors and cancelled operations are excluded. Sentry traces 10% of ordinary tool actions, continuing the UI trace through the MCP bridge. Frame reads, polling, discovery and pointer input are excluded from trace sampling. The SDK does not record MCP arguments or results.

Native device requests retain the ordinary sampled MCP trace. `device_picker.prepare`
Device requests retain the ordinary sampled MCP trace. `device_picker.prepare`
measures candidate discovery/validation in milliseconds, excluding time spent waiting
for the user. `device_picker.result` counts accept, cancel, decline, unsupported and
failed outcomes; `device_picker.selected` records only the number of selected devices.
Attributes contain only the selection mode and outcome. Unexpected handled failures
failed outcomes, plus `rendered` when the card is returned; `device_picker.selected`
records only the number of selected devices. Attributes contain only the selection
mode (`none` when browsing), outcome and a fixed `mode` (`form`, `card`, `browse`).
Since the picker became `mobile_show_devices`, `prepare` also covers discovery of
every device when candidates are omitted, so browse durations are not comparable
with earlier validation-only samples; filter by `mode`. Unexpected handled failures
use fixed messages. Device IDs/names, app labels, questions, operation details and
thumbnails are never sent to Sentry. The form is rendered by the host, so plugin UI
readiness/render timing cannot measure that surface.
Expand Down
5 changes: 2 additions & 3 deletions scripts/smoke-claude-package.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -41,10 +41,9 @@ try {
transport.stderr?.on("data", chunk => { diagnostics = (diagnostics + chunk).slice(-5000); });
try {
await client.connect(transport);
assert.match(client.getInstructions(), /otherwise in chat/);
assert.match(client.getInstructions(), /in chat when it is unsupported/);
const { tools } = await client.listTools();
assert.equal(tools.some(tool => tool.name === "mobile_choose_devices"), forms);
for (const name of ["mobile_open_workspace", "mobile_open_simulator", "mobile_list_simulators", "mobile_list_android_devices"]) {
for (const name of ["mobile_show_devices", "mobile_open_workspace", "mobile_open_simulator", "mobile_list_simulators", "mobile_list_android_devices"]) {
assert.ok(tools.some(tool => tool.name === name), name);
}
const workspace = tools.find(tool => tool.name === "mobile_open_workspace");
Expand Down
25 changes: 16 additions & 9 deletions skills/mobile-dev/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ device tools.

1. Open the panel beside the chat with `mobile_open_simulator` by default. For an existing app, open it when starting device work; for a new app, open it before the first device launch. Reuse an open Mobile Dev panel. Use `mobile_open_workspace` when the user asks for fullscreen. Use device-list tools without opening the panel when the user requests a tool-only workflow or the host cannot show panels. A request limited to planning, docs, code review, or compilation does not need a panel or a device launch.
Both opening tools accept optional `platform` (`ios` or `android`), `kind` (`simulator`, `emulator`, or `physical`), and `deviceId`. Pass only intent established by the user's request or app project: for example, `{ platform: "android", kind: "emulator" }` for Android emulator work, or `{ platform: "ios", kind: "physical" }` for an explicitly requested physical iPhone. Omit undecided fields for open-ended work. Pass `deviceId` only after discovering the exact device; discovery can precede opening when needed to identify a named target. A missing matching device stays unselected instead of connecting to another target. Opening selects a stopped simulator or AVD without booting it. Follow later manual selections.
2. Use the task and app project to choose the platform. Follow the user's device choice; otherwise reuse a compatible device shared by the panel, then a suitable running simulator or emulator. If no suitable device runs, choose a compatible installed simulator or AVD. Pick among equivalent devices yourself for routine app development. When the intended target is ambiguous (for example, the app is open on several devices for a performance run), use `mobile_choose_devices` to ask through the native request form. Read device IDs from tool results or panel context; never invent them.
2. Use the task and app project to choose the platform. Follow the user's device choice; otherwise reuse a compatible device shared by the panel, then a suitable running simulator or emulator. If no suitable device runs, choose a compatible installed simulator or AVD. Pick among equivalent devices yourself for routine app development. When the intended target is ambiguous (for example, the app is open on several devices for a performance run), use `mobile_show_devices` with a `selectionMode` to ask the user. Read device IDs from tool results or panel context; never invent them.
3. Boot a chosen stopped iOS simulator with `mobile_boot_simulator`, or an Android AVD with `mobile_boot_android_emulator`. A mobile app development request permits choosing and booting a suitable installed simulator without a separate device-choice question. Reuse running devices without rebooting them. Follow the Android section below for device discovery and serials. Opening the panel alone does not boot a device.
4. Build the app with the app project's own tools and the chosen device ID; the project's run command may also install and launch it. To restart an already installed app, use `mobile_launch_app` instead of rebuilding. Follow the project's framework skills where available. Before starting an app dev server, check for an existing server from that project and reuse it. Keep the same device for the panel, build target, and agent control; read the panel's shared IDs before acting and follow later user selections. Leave the panel and app available while continuing app work. Do not launch the app when the user asks only for compilation.
5. Agent Device is temporarily disabled in this plugin while iterating on inline performance charts. Use the available Mobile Dev tools; do not search for Agent Device tools or start its CLI separately.
Expand All @@ -43,20 +43,27 @@ These tools work on iOS simulators and Android devices with `{ platform, deviceI

The panel's settings button opens the same controls under its App tab. Its Device tab also switches to a clean status bar (9:41, full battery and signal) for screenshots and simulates LTE, 3G, Edge, or offline networks. Those device settings are panel-only, so ask the user to set them. Android throttling applies right away to standard app networking such as fetch and OkHttp, leaves local dev servers like Metro at full speed, and is unavailable while the device uses its own proxy. iOS throttling applies to URLSession and fetch requests in apps launched after it was turned on, so relaunch the app after the user enables it.

### Choosing a task's devices
### Showing and choosing devices

When the user's task leaves the target unclear, discover suitable candidates with
`mobile_list_simulators`, `mobile_list_ios_devices`, or `mobile_list_android_devices`,
then call `mobile_choose_devices`. Pass a task-specific `message`, optional `context`
for operation details, and `devices` containing each candidate's `platform`, `kind`
When the user asks to list or see their devices, call `mobile_show_devices` with no
arguments. It shows an interactive card; reply with at most a one-line summary and
do not repeat the list. Keep the list tools for your own lookups.

When the user's task leaves the target unclear, call `mobile_show_devices` with a
task-specific `message` and `selectionMode`. Omit `devices` to offer every selectable
device, or discover suitable candidates with `mobile_list_simulators`,
`mobile_list_ios_devices`, or `mobile_list_android_devices` and pass them. Pass
optional `context` for operation details, and `devices` containing each candidate's `platform`, `kind`
(`simulator`, `emulator`, or `physical`) and `deviceId` from its discovered `udid`.
Physical iOS uses the hardware UDID, never the CoreDevice ID. Set `appName` only
after verifying that app on the candidate; for profiling, use
`mobile_performance_sources` and offer only suitable running targets.

The default `selectionMode: "single"` asks for one device. Use `"multiple"` when
the task supports several targets or the user wants a comparison. The tool waits
for the user's answer and returns `action` and verified `devices`. After
Use `selectionMode: "single"` for one device and `"multiple"` when the task supports
several targets or the user wants a comparison. With a native request form the tool
waits for the user's answer and returns `action` and verified `devices`. Otherwise
it shows the choice in the card and returns immediately; wait for the user's chat
reply naming the devices, and do not start the pending task before it. After
`action: "accept"`, keep each returned device ID with its platform and kind while
performing the task. Selection does not boot a device, open a panel, launch an app,
or start a recording. For Android CPU targets omit `kind`; for physical iOS CPU
Expand Down
Loading
Loading