Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
# Changelog

## [0.1.62] - 17-07-2026

### Fixed

- **`browser_snapshot` `prune:true` on modals** — an element is now kept if it is itself visible+focusable (`checkVisibility()`), even under an `aria-hidden` ancestor. Fixes SPA modals rendered under a global `aria-hidden` wrapper returning `count:0`. Only genuinely CSS-hidden or non-focusable decorative nodes under `aria-hidden` are still elided. Default (`prune` off) behavior is unchanged.

### Added

- **`browser_screenshot` `path` option** — optional param that writes the PNG/JPEG to disk and returns `path` in `structuredContent` instead of inline base64. Refuses multi-viewport captures (`path_multi_viewport_unsupported`) and mismatched extension/mime (`path_extension_mismatch`). Omitting `path` keeps the existing inline-base64 behavior.

## [0.1.61] - 17-07-2026

### Added
Expand Down
12 changes: 11 additions & 1 deletion docs/mcp-tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -547,7 +547,7 @@ Return the indexed interactive elements of the live page, each with a `ref` to u
| --- | --- | --- | --- |
| `sessionId` | string | yes | Target session. |
| `selectors` | boolean | no | Also return a durable CSS `selector` per element (cacheable to act later without re-snapshotting). |
| `prune` | boolean | no | Drop elements hidden for accessibility (default `false`, output unchanged). When `true`, prunes any element that is `aria-hidden` (self or ancestor), `display:none` (self or ancestor), or `visibility:hidden`/`collapse` — **never** on off-screen, `opacity:0`, or off-viewport position. |
| `prune` | boolean | no | Drop only genuinely hidden or decorative elements (default `false`, output unchanged). When `true`, prunes an element iff it is **CSS-hidden** — `display:none`/`visibility:hidden`/`collapse`/`content-visibility` on itself or an ancestor, via `Element.checkVisibility()` — **or decorative**: under an `aria-hidden="true"` ancestor **and** not focusable. A **visible, focusable** element is **kept even under an `aria-hidden` ancestor**, so the controls of an **open modal** (whose SPA focus-trap marks a root/sibling wrapper `aria-hidden`) stay in the snapshot. Never prunes on off-screen, `opacity:0`, or off-viewport position. |
| `annotate` | boolean | no | Also return a Set-of-Marks JPEG: numbered badges (= each `ref`) drawn over the page, for vision models (main-frame, viewport-only). |

```json
Expand Down Expand Up @@ -748,11 +748,21 @@ Capture the live page as PNG(s) for vision. Pass `ref` for one element, `viewpor
| `colorScheme` | enum `light` \| `dark` | no | Emulate `prefers-color-scheme` and toggle `themeClass` on `<html>`, then restore. |
| `themeClass` | string | no | Class toggled on `<html>` for class-based dark themes (default `dark`). |
| `annotate` | boolean | no | Set-of-Marks JPEG: numbered badges (= each `ref`) over the viewport, for vision models. |
| `path` | string | no | Also write the captured image to disk and return its `path` in `structuredContent`. Single-image captures only. Without `path`, the image is returned base64-inline as before (unchanged). |

When `path` is set, its extension must match the output mime — `.png` for element/page/multi captures, `.jpg`/`.jpeg` for the `annotate` JPEG. Error codes:

- `path_multi_viewport_unsupported` — `path` given with `viewports.length > 1` (multiple images); use a single viewport with `path`.
- `path_extension_mismatch` — `path`'s extension does not match the output mime (e.g. `.png` for the `annotate` JPEG, or `.jpg` for a PNG capture).

```json
{ "sessionId": "s_abc123", "annotate": true, "colorScheme": "dark" }
```

```json
{ "sessionId": "s_abc123", "fullPage": true, "path": "./shots/home.png" }
```

### browser_inspect

Computed styles, box model and WCAG text-contrast (AA/AAA) for one element by `ref` — for design review (typography, color, spacing, contrast). Main-frame refs.
Expand Down
4 changes: 2 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@fusengine/browser-mcp",
"version": "0.1.61",
"version": "0.1.62",
"description": "MCP server + CLI giving AI agents a real, stealth browser (Patchright/Playwright) — per-country identity, self-healing actions, snapshots, multi-step plans, structured extraction, CDP attach.",
"license": "MIT",
"author": "Fusengine",
Expand Down Expand Up @@ -55,7 +55,7 @@
"typecheck": "tsc -p tsconfig.json --noEmit",
"lint": "biome check src tests",
"test": "bun test tests/unit",
"test:integration": "node --test --import tsx tests/integration/mcp.test.ts tests/integration/probe.test.ts tests/integration/snapshot.test.ts tests/integration/snapshot-frames.test.ts tests/integration/collect.test.ts tests/integration/collect-batch.test.ts tests/integration/selectors.test.ts tests/integration/visual-diff.test.ts tests/integration/session-state.test.ts tests/integration/pipeline.test.ts tests/integration/run.test.ts tests/integration/extract-schema.test.ts tests/integration/recovery.test.ts tests/integration/live-view.test.ts tests/integration/vault.test.ts",
"test:integration": "node --test --import tsx tests/integration/mcp.test.ts tests/integration/probe.test.ts tests/integration/snapshot.test.ts tests/integration/snapshot-prune.test.ts tests/integration/snapshot-frames.test.ts tests/integration/collect.test.ts tests/integration/collect-batch.test.ts tests/integration/selectors.test.ts tests/integration/visual-diff.test.ts tests/integration/session-state.test.ts tests/integration/pipeline.test.ts tests/integration/run.test.ts tests/integration/extract-schema.test.ts tests/integration/recovery.test.ts tests/integration/live-view.test.ts tests/integration/vault.test.ts",
"browsers": "patchright install chromium",
"mcp": "node --import tsx src/bin/mcp.ts",
"cli": "node --import tsx src/bin/cli.ts"
Expand Down
95 changes: 63 additions & 32 deletions src/extraction/snapshot-hidden.ts
Original file line number Diff line number Diff line change
@@ -1,66 +1,97 @@
/**
* "Hidden for accessibility" detection, mirroring Playwright's
* `isElementHiddenForAria` (packages/injected/src/roleUtils.ts): `aria-hidden`
* and `display:none` are STICKY down the ancestor chain (a descendant cannot
* un-hide itself); `visibility:hidden|collapse` is checked only on the element
* itself (CSS `visibility` inherits, so a descendant's `visibility:visible`
* already resolves through `getComputedStyle`, no manual override needed).
* `display:contents` never counts as `display:none`, so it does not blanket-
* hide the subtree. Deliberately NOT signals here: offscreen position
* (`offsetParent===null`, e.g. `position:fixed`), `opacity:0`, or `sr-only`
* absolute-offscreen patterns — none of those are ARIA-hidden.
* "Prunable" detection for `browser_snapshot`'s `prune:true` (the "C4" rule):
* an element is prunable iff (a) genuinely hidden — `Element.checkVisibility
* ({checkVisibilityCSS, contentVisibilityAuto})` is `false` (display:none
* self/ancestor, content-visibility:hidden, visibility:hidden/collapse) — OR
* (b) under an `aria-hidden="true"` ancestor AND NOT focusable (decorative).
* `checkVisibility()` deliberately ignores `aria-hidden` (pure CSS/box-model
* signal; Baseline Chrome 105/Firefox 106/Safari 17.4), which is exactly why a
* VISIBLE+FOCUSABLE element under an `aria-hidden` ancestor — e.g. an open
* modal `<dialog>`/`[role=dialog]` whose SPA also marks a sibling/root wrapper
* `aria-hidden` — is KEPT, not pruned. Deliberately NOT used: CDP
* `Accessibility.getFullAXTree` (Chromium-only, would break firefox/webkit).
* @module extraction/snapshot-hidden
*/

/**
* Browser-side definition injected into {@link SNAPSHOT_SCRIPT}: defines
* `isElementHiddenForAria(el)`, embedded the same way `selector.ts` embeds
* `SELECTOR_DEFS`.
* Browser-side definitions injected into {@link SNAPSHOT_SCRIPT}: defines
* `isPrunable(el)`, embedded the same way `selector.ts` embeds `SELECTOR_DEFS`.
*/
export const HIDDEN_DEFS = `
const ariaHiddenAncestor = (el) => {
const hasAriaHiddenAncestor = (el) => {
let node = el;
while (node && node.nodeType === 1) {
if (node.getAttribute('aria-hidden') === 'true') return true;
if (getComputedStyle(node).display === 'none') return true;
node = node.parentElement || (node.parentNode && node.parentNode.host);
}
return false;
};
const isElementHiddenForAria = (el) => {
const v = getComputedStyle(el).visibility;
if (v === 'hidden' || v === 'collapse') return true;
return ariaHiddenAncestor(el);
const isFocusable = (el) => {
if (el.closest('[inert]')) return false;
if ('disabled' in el && el.disabled) return false;
const tag = el.tagName;
const native =
(tag === 'A' && el.hasAttribute('href')) ||
tag === 'BUTTON' || tag === 'SELECT' || tag === 'TEXTAREA' || tag === 'IFRAME' ||
(tag === 'INPUT' && el.type !== 'hidden') ||
el.isContentEditable;
const ti = el.getAttribute('tabindex');
const tabindexOk = ti !== null && !Number.isNaN(parseInt(ti, 10)) && parseInt(ti, 10) >= 0;
if (!native && !tabindexOk) return false;
return el.tabIndex >= 0;
};
const isPrunable = (el) => {
if (!el.checkVisibility({ checkVisibilityCSS: true, contentVisibilityAuto: true })) return true;
return hasAriaHiddenAncestor(el) && !isFocusable(el);
};`;

/**
* Minimal ancestor-chain shape mirroring the DOM API surface used by
* {@link HIDDEN_DEFS} above, so the same decision logic is unit-testable
* without a real DOM (same intent as `selector.ts`'s `isStableToken`).
* Minimal ancestor-chain shape mirroring the DOM surface used by {@link
* HIDDEN_DEFS} above, so the same decision logic is unit-testable without a
* real DOM (same intent as `selector.ts`'s `isStableToken`). `focusable` is a
* precomputed stand-in for the browser-side `isFocusable(el)` result — the
* mirror tests the C4 combination logic, not DOM focusability itself.
*/
export interface AriaHiddenNode {
export interface PrunableNode {
/** This node's own `aria-hidden` attribute value, or `null`/absent. */
ariaHidden?: string | null;
/** This node's own computed `display`. */
display?: string;
/** This node's own (already-inherited) computed `visibility`. */
visibility?: string;
/** Precomputed focusability of this exact node (native tag/tabindex/disabled/inert). */
focusable?: boolean;
/** Parent in the ancestor chain (host element, for a shadow-root parent). */
parent?: AriaHiddenNode | null;
parent?: PrunableNode | null;
}

/**
* Node-testable mirror of the browser-side `isElementHiddenForAria`: true if
* the node's own resolved `visibility` is hidden/collapse, OR if `aria-hidden`
* or `display:none` appears on the node or any ancestor.
*/
export function isHiddenForAriaMirror(node: AriaHiddenNode): boolean {
/** True if `checkVisibility({checkVisibilityCSS:true})` would resolve to hidden. */
function isCssHiddenMirror(node: PrunableNode): boolean {
if (node.visibility === "hidden" || node.visibility === "collapse") return true;
let cur: AriaHiddenNode | null | undefined = node;
let cur: PrunableNode | null | undefined = node;
while (cur) {
if (cur.ariaHidden === "true") return true;
if (cur.display === "none") return true;
cur = cur.parent;
}
return false;
}

/** True if `aria-hidden="true"` appears on the node or any ancestor (sticky). */
function hasAriaHiddenAncestorMirror(node: PrunableNode): boolean {
let cur: PrunableNode | null | undefined = node;
while (cur) {
if (cur.ariaHidden === "true") return true;
cur = cur.parent;
}
return false;
}

/**
* Node-testable mirror of the browser-side `isPrunable`: true if the node is
* CSS-hidden (display:none anywhere up the chain, or own visibility:hidden/
* collapse), OR it is under an `aria-hidden` ancestor AND not focusable.
*/
export function isPrunableMirror(node: PrunableNode): boolean {
if (isCssHiddenMirror(node)) return true;
return hasAriaHiddenAncestorMirror(node) && node.focusable !== true;
}
2 changes: 1 addition & 1 deletion src/extraction/snapshot-walk.ts
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ export const SNAPSHOT_SCRIPT = `(arg) => {
options: el.tagName === 'SELECT' ? [...el.options].slice(0, 12).map((o) => o.label || o.value) : undefined,
ariaExpanded: el.getAttribute('aria-expanded'), ariaControls: el.getAttribute('aria-controls'),
visible: r.width > 0 && r.height > 0, obscured: obscured(el, r),
ariaHidden: isElementHiddenForAria(el),
prunable: isPrunable(el),
selector: wantSel ? genSelector(el) : undefined,
box: {x: Math.round(r.x), y: Math.round(r.y), width: Math.round(r.width), height: Math.round(r.height)}
};
Expand Down
33 changes: 21 additions & 12 deletions src/extraction/snapshot.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,28 +18,37 @@ export { REF_ATTRIBUTE } from "./snapshot-walk.js";
const MAX_ELEMENTS = 400;

/**
* Browser-returned element plus the internal `ariaHidden` scratch flag (see
* `snapshot-hidden.ts`). Never exposed on the final {@link InteractiveElement}
* output — used only to decide pruning, then stripped in {@link captureSnapshot}.
* Browser-returned element plus the internal `prunable` scratch flag (the C4
* decision from `snapshot-hidden.ts`). Never exposed on the final {@link
* InteractiveElement} output — used only to decide pruning, then stripped in
* {@link captureSnapshot}.
*/
type RawElement = InteractiveElement & { ariaHidden?: boolean };
type RawElement = InteractiveElement & { prunable?: boolean };

/**
* Whether a raw element survives pruning: kept unless `prune` is on AND the
* element was flagged hidden-for-accessibility. Exported for unit testing.
* element was flagged prunable (genuinely hidden, or decorative under an
* `aria-hidden` ancestor and not focusable — see `snapshot-hidden.ts`).
* A visible+focusable element under an `aria-hidden` ancestor (e.g. inside an
* open modal dialog) is NOT prunable and is always kept. Exported for unit
* testing.
*/
export function shouldKeep(ariaHidden: boolean | undefined, prune: boolean): boolean {
return !(prune && ariaHidden === true);
export function shouldKeep(prunable: boolean | undefined, prune: boolean): boolean {
return !(prune && prunable === true);
}

/**
* Capture the indexed interactive snapshot across all frames, tagging each
* element with a (frame-local) ref attribute and exposing a frame-scoped `ref`.
* Detached frames and frames that reject evaluation (e.g. mid-navigation) are
* skipped rather than aborting the whole snapshot. When `prune` is `true`,
* elements hidden for accessibility (`aria-hidden`, `display:none`, ancestor-
* hidden, or `visibility:hidden`/`collapse`) are dropped; default `false`
* keeps the output identical to the pre-pruning behavior.
* elements that are genuinely hidden (`Element.checkVisibility()` false —
* `display:none`, `content-visibility:hidden`, `visibility:hidden`/`collapse`)
* OR decorative under an `aria-hidden` ancestor (present but NOT focusable)
* are dropped. A visible AND focusable element under an `aria-hidden`
* ancestor — e.g. an open modal `<dialog>`/`[role=dialog]` whose SPA also
* marks a sibling/root wrapper `aria-hidden` — is always kept. Default
* `false` keeps the output identical to the pre-pruning behavior.
*/
export async function captureSnapshot(
page: Page,
Expand All @@ -61,8 +70,8 @@ export async function captureSnapshot(
}
for (const raw of local) {
if (all.length >= MAX_ELEMENTS) break;
const { ariaHidden, ...el } = raw;
if (!shouldKeep(ariaHidden, prune)) continue;
const { prunable, ...el } = raw;
if (!shouldKeep(prunable, prune)) continue;
el.ref = f === 0 ? String(el.index) : `${f}:${el.index}`;
if (f > 0) el.frame = f;
el.index = global++;
Expand Down
32 changes: 26 additions & 6 deletions src/server/tools/screenshot-result.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
* `structuredContent` payload conforming to `screenshotOutputShape`, required
* once the tool declares an `outputSchema` — the SDK (^1.29) throws `McpError`
* at runtime if a non-error `CallToolResult` lacks matching `structuredContent`.
* Disk-persistence (`writeScreenshotOrError`) lives in `screenshot-write.ts`.
* @module server/tools/screenshot-result
*/
import type { CallToolResult } from "@modelcontextprotocol/sdk/types.js";
Expand All @@ -16,6 +17,7 @@ export const screenshotOutputShape = {
notes: z.array(z.string()).optional(),
url: z.string().optional(),
marks: z.number().optional(),
path: z.string().optional(),
};

type Structured = z.infer<z.ZodObject<typeof screenshotOutputShape>>;
Expand All @@ -31,36 +33,53 @@ function single(base64: string, mimeType: string, note: string, structured: Stru
};
}

/** Single-element capture (`ref`). */
export function elementScreenshotResult(base64: string, ref: number | string): CallToolResult {
/** Single-element capture (`ref`). `path`, when given, has already been written by the caller. */
export function elementScreenshotResult(base64: string, ref: number | string, path?: string): CallToolResult {
const note = `element ref=${ref}`;
return single(base64, "image/png", note, {
kind: "element",
count: 1,
mimeType: "image/png",
notes: [note],
...(path ? { path } : {}),
});
}

/** Full-page / default single-viewport capture. */
export function pageScreenshotResult(base64: string, url: string): CallToolResult {
export function pageScreenshotResult(base64: string, url: string, path?: string): CallToolResult {
const note = `screenshot of ${url}`;
return single(base64, "image/png", note, { kind: "page", count: 1, mimeType: "image/png", url, notes: [note] });
return single(base64, "image/png", note, {
kind: "page",
count: 1,
mimeType: "image/png",
url,
notes: [note],
...(path ? { path } : {}),
});
}

/** Annotated capture (`annotate: true`). Kept as JPEG to match `annotatedScreenshot`. */
export function annotatedScreenshotResult(base64: string, url: string, marks: number): CallToolResult {
export function annotatedScreenshotResult(
base64: string,
url: string,
marks: number,
path?: string,
): CallToolResult {
return single(base64, "image/jpeg", JSON.stringify({ url, marks }), {
kind: "annotated",
count: 1,
mimeType: "image/jpeg",
url,
marks,
...(path ? { path } : {}),
});
}

/** Multi-viewport capture: several images, each preceded by a label note. */
export function multiScreenshotResult(items: Array<{ base64: string; note: string }>): CallToolResult {
export function multiScreenshotResult(
items: Array<{ base64: string; note: string }>,
path?: string,
): CallToolResult {
const content = items.flatMap((it) => [
{ type: "text" as const, text: it.note },
{ type: "image" as const, data: it.base64, mimeType: "image/png" as const },
Expand All @@ -72,6 +91,7 @@ export function multiScreenshotResult(items: Array<{ base64: string; note: strin
count: items.length,
mimeType: "image/png",
notes: items.map((it) => it.note),
...(path ? { path } : {}),
},
};
}
Loading