Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -453,6 +453,8 @@ On mobile, after a mutating or batch call, the daemon may take the snapshot agai

On mobile, each observation, `describe_screen`, `list_testids` and `accessibility_snapshot` takes one device snapshot (plus the settle retries above). The viewport for `visible` comes from the snapshot's root frames; the device window size is read at most once per session (and not refreshed after a rotation), only when a snapshot has no root frame. Elements that are not part of the app are removed from the tree, the refs, the test IDs and `generate_locators`: on Android, the system status bar and navigation bar windows (a full-width or full-height bar window whose ids are all `com.android.systemui:id/*`; other System UI windows such as the biometric prompt stay); on both platforms, the Expo DevLauncher "Tools" button of development builds (iOS `gearshape.fill` and its "Tools" text). Pass `{ excludeDevOverlay: false }` as the third `MobilePlatformDriver` argument to keep the DevLauncher button. Taps and waits inside `@metamask/device-mcp` still search the whole snapshot.

The mobile accessibility tree (`describe_screen`, `accessibility_snapshot`, observations) is a flat list in pre-order with no `path` (on the browser, `path` holds the `role:name` of the ancestor dialogs and headings; on mobile the ancestors are native view classes). Layout-only elements are left out (`pruneLayoutOnly` in `src/platform/mobile-hierarchy.ts`): an element with no label, no value, no identifier, and a type that is not a control or a progress indicator. Its kept descendants take its place, in order. Controls are iOS `Button`, `Link`, `TextField`, `SecureTextField`, `SearchField`, `TextView`, `TextArea`, `Switch`, `Toggle`, `Slider`, `Stepper`, `Picker`, `Cell`, `Tab`, `MenuItem` and similar AX types (exact match, without the `XCUIElementType` prefix), and Android classes ending in `Button`, `EditText`, `CheckBox`, `CheckedTextView`, `Switch`, `SeekBar`, `Spinner`, `RatingBar` and similar. The device snapshot has no `clickable` flag, so a layout-only plain React Native view (`RCTViewComponentView` on iOS, `android.view.ViewGroup` on Android) that has no kept children and is at least 24×24 (iOS points, Android pixels) stays as an unlabelled tap target, for example the iOS wallet home action buttons. Node `bounds` are `{ x, y, width, height }`, in that key order. Node refs number the kept nodes only, and ambiguity is computed over them. Only nodes with a testId, label or value have a refMap entry, so only their refs can be targeted; unlabelled tap targets, progress indicators and unnamed controls are listed with `bounds` (use `tap_coordinates`) but their refs do not resolve. On real MetaMask screens: iOS wallet home 177 → 67 nodes (183 kB with `path` → 10 kB), Android wallet home 625 → 242 nodes (649 kB → 35 kB), Android login 47 → 15 nodes. `list_testids` and `generate_locators` are not affected, and `describe_screen` still returns the full `testIds` list next to the tree.

### Session Metadata

Consumers can persist launch metadata with `writeSessionMetadata(artifactsDir, metadata)`. It writes `<artifactsDir>/<sessionId>/session.json` (for example `test-artifacts/<sessionId>/session.json` with `artifactsDir: 'test-artifacts'`) and returns the file path.
Expand Down
4 changes: 4 additions & 0 deletions SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -232,8 +232,12 @@ Each node looks like:

The `testId` and `textContent` fields appear only on nodes with short or generic names — they provide extra context from the DOM to help identify ambiguous elements. Nodes with clear names omit these fields.

On the browser, `path` lists the `role:name` of the dialogs and headings that contain the node. Mobile nodes have no `path`.

When 3+ consecutive identical nodes appear (same role, name, and path), they are collapsed into a summary like `… 3 more "maskicon" (refs e2–e4)` to reduce token waste. Individual refs still work for targeting.

On mobile (iOS, Android), the tree is a flat list in screen order. A node is a native element: `role` is its type (`Button`, `StaticText`, `android.widget.Button`, `android.view.ViewGroup`), and `bounds` is its frame (`{ x, y, width, height }`). Layout-only elements are left out: elements with no label or text, no testId, and a type that is not a control (button, link, text field, switch, slider, checkbox and similar). Their children stay in the list. Containers with a testId stay. Plain unlabelled views of at least 24×24 with nothing kept inside also stay: they are often icon buttons (for example the iOS wallet home action buttons). Only a node with a testId, label or text has a ref you can click or type into. Nodes without one (unlabelled views, spinners, unnamed controls) are listed with their `bounds` but their ref cannot be targeted: use `mm tap-coordinates <x> <y>` with the center of `bounds`.

Use the `ref` value (`e3`) for click/type/get-text/wait-for commands. On the browser, each ref resolves to exactly one element, or the action fails (strict mode) instead of acting on another element: refs use the exact, case-sensitive name, and elements that share a role and name get a unique DOM selector, so two "Send" buttons get two refs that each click their own button. If a click fails with a strict mode violation, target the element by `--testid` or `--selector` instead.

#### `mm list-testids`
Expand Down
Loading
Loading