Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -439,7 +439,7 @@ The daemon routes `POST /tool/:name` requests through the registry, applies Zod

### Accessibility References

The core uses Chrome DevTools Protocol accessibility APIs to build a deterministic reference map of interactive and meaningful accessibility nodes. Each element gets a short ref like `e1`, `e2`, etc., mapped to a Playwright-compatible selector.
The core uses Chrome DevTools Protocol accessibility APIs to build a deterministic reference map of interactive and meaningful accessibility nodes. Each element gets a short ref like `e1`, `e2`, etc., mapped to a Playwright-compatible selector. Each browser ref resolves to exactly one element, or the action fails (strict mode) instead of acting on another element. A ref uses `role=<role>[name="<name>" s]`, which matches the exact, case-sensitive accessible name. When several elements of a snapshot share a role and name, each of them gets a selector built from its DOM node and verified unique the way Playwright matches CSS (unique `data-testid`, else unique `id`, else a path anchored at the nearest ancestor with a unique `data-testid` or `id`; elements in open shadow roots get `css=<host> >> css=<inside>`). When no such selector can be built, the ref keeps the plain role selector, so an action on it fails in strict mode; refs to elements in closed shadow roots always fail. Generic elements without a DOM selector use `text="<name>"`, which also fails when ambiguous. Role and text refs from a scoped `accessibility_snapshot` are prefixed with the root selector.

Agents call `describe_screen` to get the current reference map, then use refs for interaction:

Expand Down
4 changes: 2 additions & 2 deletions SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -232,7 +232,7 @@ The `testId` and `textContent` fields appear only on nodes with short or generic

When 3+ consecutive identical nodes appear (same role, name, and path), they are collapsed into a summary like `… 3 more "maskicon" (refs e2–e4)` to reduce token waste. Individual refs still work for targeting.

Use the `ref` value (`e3`) for click/type/get-text/wait-for commands.
Use the `ref` value (`e3`) for click/type/get-text/wait-for commands. On the browser, each ref resolves to exactly one element, or the action fails (strict mode) instead of acting on another element: refs use the exact, case-sensitive name, and elements that share a role and name get a unique DOM selector, so two "Send" buttons get two refs that each click their own button. If a click fails with a strict mode violation, target the element by `--testid` or `--selector` instead.

#### `mm list-testids`

Expand Down Expand Up @@ -284,7 +284,7 @@ mm click --testid end-accessory --within "testid:account-list-item/0"
mm click --testid onboarding-complete-done --timeout 60000
```

Use `--within` to scope the target inside a parent element. Values use the format `testid:<id>`, `selector:<css>`, or a bare a11y ref (`e5`).
Use `--within` to scope the target inside a parent element. Values use the format `testid:<id>`, `selector:<css>`, or a bare a11y ref (`e5`). With a ref target (`mm click e7 --within ...`), the ref already identifies one element, so `--within` only checks that it is inside the parent; when it is on the page but outside the parent, the command fails at once with `Ref e7 is not inside the within element`.

If the page closes after clicking (e.g., confirmation popup), the response includes `pageClosedAfterClick: true` — this is normal, not an error.

Expand Down
4 changes: 3 additions & 1 deletion src/server/create-server.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1246,7 +1246,9 @@ describe('createServer with active session', () => {
// The first request is now running inside the queue.
await vi.waitFor(() => expect(mockSM.cleanup).toHaveBeenCalled());
const second = post('/tool/get_context');
await new Promise((resolve) => setTimeout(resolve, 60));
// Hold the first request well past the threshold below: under host
// load the second request can take tens of ms to reach the queue.
await new Promise((resolve) => setTimeout(resolve, 250));
releaseFirst();

const [firstRes, secondRes] = await Promise.all([first, second]);
Expand Down
126 changes: 126 additions & 0 deletions src/tools/utils/cdp.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import { describe, it, expect, vi } from 'vitest';
import {
ISOLATED_WORLD_NAME,
callInIsolatedWorld,
callOnNodeInIsolatedWorld,
withCdpSession,
} from './cdp.js';

Expand All @@ -14,10 +15,17 @@ type Send = ReturnType<typeof vi.fn>;
* Builds a page whose CDP sessions answer with `respond`.
*
* @param respond - Answers `Runtime.callFunctionOn`.
* @param resolveNode - Answers `DOM.resolveNode` (default: an object id).
* @returns The page, the sessions it opened and their send mocks.
*/
function createPage(
respond: (params: Record<string, unknown>, sessionIndex: number) => unknown,
resolveNode: (
params: Record<string, unknown>,
sessionIndex: number,
) => unknown = (params) => ({
object: { objectId: `obj-${String(params.backendNodeId)}` },
}),
) {
const sessions: { send: Send; detach: Send }[] = [];
let contextCounter = 0;
Expand All @@ -33,6 +41,10 @@ function createPage(
return { executionContextId: contextCounter };
case 'Runtime.callFunctionOn':
return respond(params, sessionIndex);
case 'DOM.resolveNode':
return resolveNode(params, sessionIndex);
case 'Runtime.releaseObjectGroup':
return {};
default:
throw new Error(`Unexpected ${method}`);
}
Expand Down Expand Up @@ -228,3 +240,117 @@ describe('callInIsolatedWorld', () => {
expect(await callInIsolatedWorld(page, '() => 1', [], 1_000)).toBe('quick');
});
});

describe('callOnNodeInIsolatedWorld', () => {
it('resolves the node in the isolated world and calls the function on it', async () => {
const { page, sessions } = createPage((params) => {
const fn = vm.runInNewContext(
`(${String(params.functionDeclaration)})`,
{},
);
return { result: { value: fn.call({ id: params.objectId }) } };
});

const result = await callOnNodeInIsolatedWorld<string>(
page,
42,
'function () { return __name(this.id, "x"); }',
);

expect(result).toBe('obj-42');
const { calls } = sessions[0].send.mock;
const resolve = calls.find(([method]) => method === 'DOM.resolveNode');
expect(resolve?.[1]).toStrictEqual({
backendNodeId: 42,
executionContextId: 1,
objectGroup: expect.stringMatching(/^mm-node-\d+$/u),
});
expect(calls).toContainEqual([
'Runtime.callFunctionOn',
expect.objectContaining({ objectId: 'obj-42', returnByValue: true }),
]);
expect(calls).toContainEqual([
'Runtime.releaseObjectGroup',
{ objectGroup: resolve?.[1].objectGroup },
]);
});

it('shares the session and world with callInIsolatedWorld', async () => {
const { page, newCDPSession } = createPage(() => ({
result: { value: 1 },
}));

await callInIsolatedWorld(page, '() => 1');
await callOnNodeInIsolatedWorld(page, 1, 'function () { return 1; }');

expect(newCDPSession).toHaveBeenCalledTimes(1);
});

it('rejects without retrying when the node cannot be resolved', async () => {
const { page, newCDPSession, sessions } = createPage(
() => ({ result: { value: 1 } }),
() => {
throw new Error('No node with given id found');
},
);

await expect(
callOnNodeInIsolatedWorld(page, 7, 'function () { return 1; }'),
).rejects.toThrowError('No node with given id found');
expect(newCDPSession).toHaveBeenCalledTimes(1);
expect(sessions[0].detach).not.toHaveBeenCalled();
});

it('rejects when the node has no object', async () => {
const { page, sessions } = createPage(
() => ({ result: { value: 1 } }),
() => ({ object: {} }),
);

await expect(
callOnNodeInIsolatedWorld(page, 7, 'function () { return 1; }'),
).rejects.toThrowError('No object for node 7');
expect(sessions[0].send).toHaveBeenCalledWith(
'Runtime.releaseObjectGroup',
expect.anything(),
);
});

it('starts over once on a new world after a stale context', async () => {
const { page, newCDPSession, sessions } = createPage(
() => ({ result: { value: 'fresh' } }),
(params, index) => {
if (index === 0) {
throw new Error('Cannot find context with specified id');
}
return { object: { objectId: `obj-${String(params.backendNodeId)}` } };
},
);

const result = await callOnNodeInIsolatedWorld(
page,
3,
'function () { return 1; }',
);

expect(result).toBe('fresh');
expect(newCDPSession).toHaveBeenCalledTimes(2);
expect(sessions[0].detach).toHaveBeenCalled();
expect(sessions[1].send).toHaveBeenCalledWith(
'DOM.resolveNode',
expect.objectContaining({ executionContextId: 2 }),
);
});

it('rejects without retrying when the function throws in the page', async () => {
const { page, newCDPSession } = createPage(() => ({
result: { type: 'object' },
exceptionDetails: { text: 'Uncaught' },
}));

await expect(
callOnNodeInIsolatedWorld(page, 1, 'function () { throw 1; }'),
).rejects.toThrowError('Uncaught');
expect(newCDPSession).toHaveBeenCalledTimes(1);
});
});
Loading
Loading