Skip to content

feat: drop priorKnowledge from describe_screen #59

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cryptotavares merged 3 commits into from
Oct 29, 2026
Merged

feat: drop priorKnowledge from describe_screen #59

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d3bf89b
feat!: drop priorKnowledge from describe_screen
cryptotavares Oct 23, 2026
e278112
chore: tidy priorKnowledge leftovers in docs and tests
cryptotavares Oct 23, 2026
fccafb3
test: remove obsolete discovery session fixtures
cryptotavares Oct 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -393,7 +393,7 @@ The daemon routes `POST /tool/:name` requests through the registry, applies Zod
| `close_tab` | Closes a browser tab matching a given role or URL. Falls back to the extension tab if the active tab is closed. |
| `wait_for_notification` | Waits for the extension notification popup to appear within a timeout. Returns the notification page URL. |
| **Discovery** | |
| `describe_screen` | Captures a comprehensive screen snapshot: extension state, visible test IDs, trimmed a11y tree with refs, optional screenshot, and prior knowledge from historical sessions. |
| `describe_screen` | Captures a comprehensive screen snapshot: extension state, visible test IDs, trimmed a11y tree with refs, and optional screenshot. |
| `accessibility_snapshot` | Captures a trimmed accessibility tree of the current page with deterministic refs (`e1`, `e2`, ...). Supports scoping to a root CSS selector. |
| `list_testids` | Collects all visible `data-testid` attributes from the current page with text previews and visibility status. |
| **State** | |
Expand Down Expand Up @@ -456,7 +456,6 @@ The `KnowledgeStore` provides cross-session learning by recording every tool exe

- **Step Recording**: Each tool invocation captures the tool name, input, outcome, observation (extension state, visible test IDs, a11y nodes), and timing.
- **Session Metadata**: Sessions are tagged with goals, flow tags, and free-form tags for filtering.
- **Prior Knowledge**: Before tool execution, the store can generate context from historical sessions — similar steps, suggested actions, and patterns to avoid — based on the current screen state.
- **Search**: Token-based search with synonym expansion across sessions, scored by relevance to screen, URL, test IDs, and a11y nodes.
- **Sensitive Data Handling**: Input text for password fields and other sensitive inputs is automatically redacted. Recorded `clipboard` and `device_clipboard` text and all CDP `params` are replaced with `[REDACTED]`, including within `run_steps`. Echoes in the same step's error messages/details, tool targets, observations, and artifact paths are also scrubbed. Raw input strings of at least eight characters match substrings; shorter strings match whole values only. JSON-escaped echoes use the original raw string length to select the matching policy. Structural step metadata and diagnostic error codes are preserved. Tools and HTTP responses retain the original values.

Expand Down Expand Up @@ -596,7 +595,7 @@ mm describe-screen
| `mm click <ref> [--timeout <ms>] [--selector <css>] [--testid <id>] [--within <scope>]` | Clicks an element by its accessibility reference (e.g., `e3`). The ref comes from a prior `describe-screen` call. Waits for the element to be visible before clicking. The `--timeout` covers the entire operation (visibility wait + click action). Default: 15s. Use `--within` to scope the target inside a parent element (`testid:<id>`, `selector:<css>`, or a bare a11y ref). |
| `mm type <ref> <text> [--timeout <ms>] [--selector <css>] [--testid <id>] [--within <scope>]` | Types text into an input element identified by its accessibility reference. Clears the field first, then sets the new value (uses Playwright's `fill()`). Accepts `--timeout` for the total time budget (default: 15s). Use `--within` to scope the target inside a parent element. |
| `mm get-text <ref> [--timeout <ms>] [--selector <css>] [--testid <id>] [--within <scope>]` | Reads the text content of an element. Returns the inner text, target descriptor, and character length. Accepts `--timeout` for the total time budget (default: 15s). Useful for asserting visible values without screenshots. |
| `mm describe-screen` | Captures the full screen state: extension info, visible test IDs, a trimmed accessibility tree with deterministic refs (`e1`, `e2`, ...), and prior knowledge from historical sessions. This is the primary command for understanding what's on screen before interacting. |
| `mm describe-screen` | Captures the full screen state: extension info, visible test IDs, and a trimmed accessibility tree with deterministic refs (`e1`, `e2`, ...). This is the primary command for understanding what's on screen before interacting. |
| `mm screenshot [--name <name>]` | Takes a full-page screenshot of the current page. Saves to the artifacts directory. Use `--name` to set a descriptive filename. |
| `mm wait-for <ref> [--timeout <ms>] [--selector <css>] [--testid <id>] [--within <scope>]` | Blocks until an element identified by its accessibility reference becomes visible, or the timeout expires. Default timeout is 15 seconds. Use `--within` to scope the target inside a parent element. |
| `mm wait-for-notification [--timeout <ms>]` | Waits for the extension notification popup to appear within a timeout. Returns the notification page URL. |
Expand Down
34 changes: 5 additions & 29 deletions SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ mm cleanup --shutdown # 5. Clean up when done
**Critical rules:**

- **Always `describe-screen` before interacting.** Refs like `e1`, `e2` are ephemeral — they change after every action.
- **Always `describe-screen` after interacting** — OR use inline `observations` from mutating tool responses. Mutating tools (click, type, navigate, etc.) return an `observations` object with fresh `state`, `testIds`, and `a11y` refs. You can use these refs directly for the next interaction without calling `describe-screen`. Call `describe-screen` when you need `priorKnowledge` or screenshots.
- **Always `describe-screen` after interacting** — OR use inline `observations` from mutating tool responses. Mutating tools (click, type, navigate, etc.) return an `observations` object with fresh `state`, `testIds`, and `a11y` refs. You can use these refs directly for the next interaction without calling `describe-screen`. Call `describe-screen` when you need the full tree or screenshots.
- **One target per command.** Specify exactly ONE of: a11y ref (`e5`), testId, or CSS selector.
- **Errors are structured.** Check the `error.code` field to decide recovery strategy (see Error Codes below).

Expand All @@ -32,7 +32,7 @@ Tool responses include different data based on the tool's category:
| **Discovery** | describe_screen, list_testids, accessibility_snapshot, screenshot | Data is already in `result` |
| **Batch** | run_steps | Controlled by `includeObservations` param |

**Observation Compaction:** Mutating tool observations are **compacted** before returning: option runs of 3 or more under a combobox or listbox are replaced with a single summary node (e.g., `"55 options (refs e2–e56)"`). The `describe-screen` tool always returns the **full, unfiltered** a11y tree — use it when you need the complete option list or `priorKnowledge`.
**Observation Compaction:** Mutating tool observations are **compacted** before returning: option runs of 3 or more under a combobox or listbox are replaced with a single summary node (e.g., `"55 options (refs e2–e56)"`). The `describe-screen` tool always returns the **full, unfiltered** a11y tree — use it when you need the complete option list.

**Diff-Based Observations:** After the first mutating tool call sets a baseline, subsequent mutations return **diff-based** observations. The `observations.a11y.diff` field (when present) shows what changed:

Expand Down Expand Up @@ -73,7 +73,7 @@ You can use the `ref` values from `observations.a11y.nodes` for the next interac

- Use `observations.state` for quick checks (screen name, loading status, balance, etc.)
- Use `observations.a11y.nodes` with the compact refs for the next interaction
- Call `describe-screen` only when you need the full tree or `priorKnowledge`
- Call `describe-screen` only when you need the full tree

```bash
mm click e3 # mutating: response includes fresh observations
Expand All @@ -83,7 +83,6 @@ mm type e1 "0.01" # use ref from previous response

Call `describe-screen` explicitly when you need:

- `priorKnowledge` (historical actions for this screen)
- A screenshot via `includeScreenshot`
- Full context after unexpected navigation
- The complete, unfiltered a11y tree (e.g., all options in a dropdown)
Expand Down Expand Up @@ -197,7 +196,6 @@ mm serve [--background]
- **Active tab**: the currently focused tab's role and URL (if tracked)
- **Test IDs**: visible `data-testid` attributes with text previews
- **A11y tree**: interactive elements with deterministic refs (`e1`, `e2`, ...)
- **Prior knowledge**: suggested actions from past sessions on this screen

```
mm describe-screen
Expand Down Expand Up @@ -240,7 +238,7 @@ Useful when you know a `testId` value and want to verify it exists. Prefer `desc

#### `mm accessibility-snapshot`

Captures just the trimmed accessibility tree with deterministic refs. Lighter than `describe-screen` (no state, no prior knowledge, no test IDs).
Captures just the trimmed accessibility tree with deterministic refs. Lighter than `describe-screen` (no state, no test IDs).

```
mm accessibility-snapshot [--root <selector>]
Expand Down Expand Up @@ -678,21 +676,11 @@ Every interaction command (`click`, `type`, `get-text`, `wait-for`) needs a targ
| Method | Format | Stability | When to use |
| ---------------- | ------------------- | ------------------------------- | ---------------------------------------------------- |
| **a11y ref** | `e1`, `e2`, ... | Ephemeral (per describe-screen) | Default — use refs from the latest `describe-screen` |
| **testId** | `data-testid` value | Stable across sessions | When you know the testId from prior knowledge |
| **testId** | `data-testid` value | Stable across sessions | When you know the testId from earlier output |
| **CSS selector** | Any CSS selector | Fragile | Last resort fallback |

**Prefer a11y refs.** They come directly from the accessibility tree and map to ARIA selectors, making them the most reliable for the current screen state.

## Prior Knowledge

When you call `describe-screen`, the response may include a `priorKnowledge` section with:

- **`similarSteps`**: Past tool invocations on the same screen with confidence scores
- **`suggestedNextActions`**: Ranked actions based on historical success (e.g., "click confirm button")
- **`avoid`**: Targets that frequently fail on this screen — skip these

Use prior knowledge to guide your actions, but always verify against the current a11y tree.

## Error Codes

When a command fails, the response includes `error.code`. Use this to decide what to do:
Expand Down Expand Up @@ -827,18 +815,6 @@ mm --project ../metamask-extension launch
mm --project ../metamask-extension describe-screen
```

### Using Prior Knowledge

```bash
mm launch --state default --goal "Test send flow" --flow-tags send
mm describe-screen
# Response includes priorKnowledge.suggestedNextActions:
# [{ action: "click", preferredTarget: { type: "testId", value: "send-button" }, confidence: 0.85 }]
# Use the suggestion but verify the target exists in the current a11y tree
mm click e3
mm cleanup --shutdown
```

## Project-Specific Commands

<!-- Consumer repos extend this section -->
Expand Down
Loading
Loading