Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,21 @@
# Changelog

## [0.1.61] - 17-07-2026

### Added

- **Structured tool output** — every one of the 50 MCP tools now declares a zod `outputSchema`, and `structuredContent` is emitted on all return paths, including `browser_screenshot`'s 4 branches (new `screenshot-result.ts` helper). Downstream clients can now consume typed results instead of parsing text.
- **`browser_snapshot` `prune` option** — opt-in (default `false`, behavior unchanged) param that elides hidden elements (`aria-hidden`, `display:none`, `visibility:hidden`) from the accessibility tree, shrinking snapshots on dense pages.

### Changed

- MCP text `content` is now compact (non-indented) JSON — `structuredContent` unchanged — cutting tokens on every tool response with no semantic difference.
- Dependencies bumped to latest: `typescript` 6 → 7, `patchright`/`playwright` 1.60 → 1.61.1, `defuddle` 0.18 → 0.19, `@biomejs/biome` 2.4 → 2.5.4, plus patch bumps for `impit`/`libphonenumber-js`/`linkedom`/`tsx`. `@types/node` → `^22` and `engines.node` → `">=22"` (Node 20 is EOL 2026-04-30).

### Verified

- New unit coverage: batched `outputSchema` parsing across tools, `prune`, and all 4 `browser_screenshot` branches. New integration coverage: `browser_snapshot`/`browser_act` `outputSchema` + `prune` on real Chromium.

## [0.1.60] - 01-07-2026

### Added
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Shadow DOM + iframes), multi-step plans, structured extraction, visual diff, and
guardrails** for payments and bookings. It drives real Chromium, so it reads **Next.js / SPA**
pages after hydration — not just static HTML.

> 50 MCP tools · stealth + rotating proxies · HTTP fast-path (single, batch & crawl) · full-site content + screenshot snapshots · structured per-card product extraction · form fill + file upload · encrypted credential vault + TOTP (fill-by-reference, origin-bound, never exposed to the LLM) · hover + drag&drop · PDF export · cookies / permissions · network mocking · clipboard · virtualized-list scraping + autoscroll · tabs / dialogs / downloads · console + network logs · MCP screenshot resources · `FUSE_CAPS` tool-group filtering · named auth profiles (cookies + localStorage + IndexedDB, saved at login) · `blockResources` · HAR record/replay · pixel visual-diff · human handoff + live view.
> 50 MCP tools · stealth + rotating proxies · HTTP fast-path (single, batch & crawl) · full-site content + screenshot snapshots · structured per-card product extraction · form fill + file upload · encrypted credential vault + TOTP (fill-by-reference, origin-bound, never exposed to the LLM) · hover + drag&drop · PDF export · cookies / permissions · network mocking · clipboard · virtualized-list scraping + autoscroll · tabs / dialogs / downloads · console + network logs · MCP screenshot resources · `FUSE_CAPS` tool-group filtering · named auth profiles (cookies + localStorage + IndexedDB, saved at login) · `blockResources` · HAR record/replay · pixel visual-diff · human handoff + live view · typed structured output (`outputSchema` + `structuredContent`).

## Install

Expand Down
152 changes: 103 additions & 49 deletions bun.lock

Large diffs are not rendered by default.

9 changes: 9 additions & 0 deletions docs/mcp-tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,10 @@ Every field is optional unless **Required** says `yes`. Defaults shown below com

The shared identity/profile options (the `agentOptionShape`) are listed once under [`browser_open`](#browser_open); tools that accept them say so and link back.

## Return format (structured output)

Every tool declares an `outputSchema` (zod) and returns a **typed `structuredContent`** validated against it, alongside the human-readable `content`. Clients that support MCP structured output read `structuredContent` directly (typed, no parsing); clients that don't fall back to the `content` block. The `content` text is **compact JSON** (non-indented) — the pretty-print whitespace only inflated the token cost, and it stays semantically identical to `structuredContent`. Image tools (`browser_screenshot`, and any tool called with `annotate:true`) return the image in `content` **plus** the structured metadata in `structuredContent`. Error results set `isError` and, when a machine-readable code applies, expose `structuredContent: { code, message }`.

## Capability groups (`FUSE_CAPS`)

By default all 50 tools are registered. Set the `FUSE_CAPS` env var (comma-separated group names) to expose fewer tools — a lighter context for the LLM client:
Expand Down Expand Up @@ -543,12 +547,17 @@ Return the indexed interactive elements of the live page, each with a `ref` to u
| --- | --- | --- | --- |
| `sessionId` | string | yes | Target session. |
| `selectors` | boolean | no | Also return a durable CSS `selector` per element (cacheable to act later without re-snapshotting). |
| `prune` | boolean | no | Drop elements hidden for accessibility (default `false`, output unchanged). When `true`, prunes any element that is `aria-hidden` (self or ancestor), `display:none` (self or ancestor), or `visibility:hidden`/`collapse` — **never** on off-screen, `opacity:0`, or off-viewport position. |
| `annotate` | boolean | no | Also return a Set-of-Marks JPEG: numbered badges (= each `ref`) drawn over the page, for vision models (main-frame, viewport-only). |

```json
{ "sessionId": "s_abc123", "annotate": true }
```

```json
{ "sessionId": "s_abc123", "prune": true }
```

### browser_act

Execute click/fill/select/pick/upload/hover/drag on an element by `ref` (from `browser_snapshot`) or by `target` text. Returns a diff of what changed on the page (added/removed/text/url).
Expand Down
24 changes: 12 additions & 12 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "@fusengine/browser-mcp",
"version": "0.1.60",
"version": "0.1.61",
"description": "MCP server + CLI giving AI agents a real, stealth browser (Patchright/Playwright) — per-country identity, self-healing actions, snapshots, multi-step plans, structured extraction, CDP attach.",
"license": "MIT",
"author": "Fusengine",
Expand Down Expand Up @@ -29,7 +29,7 @@
"access": "public"
},
"engines": {
"node": ">=20"
"node": ">=22"
},
"bin": {
"fuse-browser": "dist/bin/cli.js",
Expand Down Expand Up @@ -66,20 +66,20 @@
],
"dependencies": {
"@modelcontextprotocol/sdk": "^1.29.0",
"defuddle": "^0.18.1",
"defuddle": "^0.19.1",
"fast-png": "^8.0.0",
"impit": "^0.14.1",
"libphonenumber-js": "^1.13.5",
"linkedom": "^0.18.12",
"patchright": "^1.60.1",
"impit": "^0.14.3",
"libphonenumber-js": "^1.13.9",
"linkedom": "^0.18.13",
"patchright": "^1.61.1",
"pixelmatch": "^7.2.0",
"playwright": "^1.60.0",
"playwright": "^1.61.1",
"zod": "^4.4.3"
},
"devDependencies": {
"@biomejs/biome": "^2.4.16",
"@types/node": "^25.9.1",
"tsx": "^4.22.4",
"typescript": "^6.0.3"
"@biomejs/biome": "^2.5.4",
"@types/node": "^22.10.5",
"tsx": "^4.23.1",
"typescript": "^7.0.2"
}
}
66 changes: 66 additions & 0 deletions src/extraction/snapshot-hidden.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
/**
* "Hidden for accessibility" detection, mirroring Playwright's
* `isElementHiddenForAria` (packages/injected/src/roleUtils.ts): `aria-hidden`
* and `display:none` are STICKY down the ancestor chain (a descendant cannot
* un-hide itself); `visibility:hidden|collapse` is checked only on the element
* itself (CSS `visibility` inherits, so a descendant's `visibility:visible`
* already resolves through `getComputedStyle`, no manual override needed).
* `display:contents` never counts as `display:none`, so it does not blanket-
* hide the subtree. Deliberately NOT signals here: offscreen position
* (`offsetParent===null`, e.g. `position:fixed`), `opacity:0`, or `sr-only`
* absolute-offscreen patterns — none of those are ARIA-hidden.
* @module extraction/snapshot-hidden
*/

/**
* Browser-side definition injected into {@link SNAPSHOT_SCRIPT}: defines
* `isElementHiddenForAria(el)`, embedded the same way `selector.ts` embeds
* `SELECTOR_DEFS`.
*/
export const HIDDEN_DEFS = `
const ariaHiddenAncestor = (el) => {
let node = el;
while (node && node.nodeType === 1) {
if (node.getAttribute('aria-hidden') === 'true') return true;
if (getComputedStyle(node).display === 'none') return true;
node = node.parentElement || (node.parentNode && node.parentNode.host);
}
return false;
};
const isElementHiddenForAria = (el) => {
const v = getComputedStyle(el).visibility;
if (v === 'hidden' || v === 'collapse') return true;
return ariaHiddenAncestor(el);
};`;

/**
* Minimal ancestor-chain shape mirroring the DOM API surface used by
* {@link HIDDEN_DEFS} above, so the same decision logic is unit-testable
* without a real DOM (same intent as `selector.ts`'s `isStableToken`).
*/
export interface AriaHiddenNode {
/** This node's own `aria-hidden` attribute value, or `null`/absent. */
ariaHidden?: string | null;
/** This node's own computed `display`. */
display?: string;
/** This node's own (already-inherited) computed `visibility`. */
visibility?: string;
/** Parent in the ancestor chain (host element, for a shadow-root parent). */
parent?: AriaHiddenNode | null;
}

/**
* Node-testable mirror of the browser-side `isElementHiddenForAria`: true if
* the node's own resolved `visibility` is hidden/collapse, OR if `aria-hidden`
* or `display:none` appears on the node or any ancestor.
*/
export function isHiddenForAriaMirror(node: AriaHiddenNode): boolean {
if (node.visibility === "hidden" || node.visibility === "collapse") return true;
let cur: AriaHiddenNode | null | undefined = node;
while (cur) {
if (cur.ariaHidden === "true") return true;
if (cur.display === "none") return true;
cur = cur.parent;
}
return false;
}
4 changes: 3 additions & 1 deletion src/extraction/snapshot-walk.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
* @module extraction/snapshot-walk
*/

import { HIDDEN_DEFS } from "./snapshot-hidden.js";
import { SELECTOR_DEFS } from "./selector.js";

/** Attribute injected on each interactive element to anchor a stable ref. */
Expand All @@ -23,7 +24,7 @@ const SELECTOR =
*/
export const SNAPSHOT_SCRIPT = `(arg) => {
const SEL = '${SELECTOR}';
const wantSel = !!(arg && arg.selectors);${SELECTOR_DEFS}
const wantSel = !!(arg && arg.selectors);${SELECTOR_DEFS}${HIDDEN_DEFS}
const obscured = (el, r) => {
const cx = r.x + r.width / 2, cy = r.y + r.height / 2;
if (r.width === 0 || cx < 0 || cy < 0 || cx > innerWidth || cy > innerHeight) return false;
Expand All @@ -48,6 +49,7 @@ export const SNAPSHOT_SCRIPT = `(arg) => {
options: el.tagName === 'SELECT' ? [...el.options].slice(0, 12).map((o) => o.label || o.value) : undefined,
ariaExpanded: el.getAttribute('aria-expanded'), ariaControls: el.getAttribute('aria-controls'),
visible: r.width > 0 && r.height > 0, obscured: obscured(el, r),
ariaHidden: isElementHiddenForAria(el),
selector: wantSel ? genSelector(el) : undefined,
box: {x: Math.round(r.x), y: Math.round(r.y), width: Math.round(r.width), height: Math.round(r.height)}
};
Expand Down
34 changes: 29 additions & 5 deletions src/extraction/snapshot.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,28 +17,52 @@ export { REF_ATTRIBUTE } from "./snapshot-walk.js";
/** Soft cap on total elements across all frames, to bound output size. */
const MAX_ELEMENTS = 400;

/**
* Browser-returned element plus the internal `ariaHidden` scratch flag (see
* `snapshot-hidden.ts`). Never exposed on the final {@link InteractiveElement}
* output — used only to decide pruning, then stripped in {@link captureSnapshot}.
*/
type RawElement = InteractiveElement & { ariaHidden?: boolean };

/**
* Whether a raw element survives pruning: kept unless `prune` is on AND the
* element was flagged hidden-for-accessibility. Exported for unit testing.
*/
export function shouldKeep(ariaHidden: boolean | undefined, prune: boolean): boolean {
return !(prune && ariaHidden === true);
}

/**
* Capture the indexed interactive snapshot across all frames, tagging each
* element with a (frame-local) ref attribute and exposing a frame-scoped `ref`.
* Detached frames and frames that reject evaluation (e.g. mid-navigation) are
* skipped rather than aborting the whole snapshot.
* skipped rather than aborting the whole snapshot. When `prune` is `true`,
* elements hidden for accessibility (`aria-hidden`, `display:none`, ancestor-
* hidden, or `visibility:hidden`/`collapse`) are dropped; default `false`
* keeps the output identical to the pre-pruning behavior.
*/
export async function captureSnapshot(page: Page, selectors = false): Promise<InteractiveElement[]> {
export async function captureSnapshot(
page: Page,
selectors = false,
prune = false,
): Promise<InteractiveElement[]> {
const frames = page.frames();
const all: InteractiveElement[] = [];
const arg = { selectors };
let global = 0;
for (let f = 0; f < frames.length && all.length < MAX_ELEMENTS; f++) {
const frame = frames[f];
if (!frame || frame.isDetached()) continue;
let local: InteractiveElement[];
let local: RawElement[];
try {
local = await evalScriptArg<InteractiveElement[], typeof arg>(frame, SNAPSHOT_SCRIPT, arg);
local = await evalScriptArg<RawElement[], typeof arg>(frame, SNAPSHOT_SCRIPT, arg);
} catch {
continue;
}
for (const el of local) {
for (const raw of local) {
if (all.length >= MAX_ELEMENTS) break;
const { ariaHidden, ...el } = raw;
if (!shouldKeep(ariaHidden, prune)) continue;
el.ref = f === 0 ? String(el.index) : `${f}:${el.index}`;
if (f > 0) el.frame = f;
el.index = global++;
Expand Down
2 changes: 1 addition & 1 deletion src/server/resources.ts
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ function registerRunsIndex(server: McpServer): void {
},
async (uri) => ({
contents: [
{ uri: uri.href, mimeType: "application/json", text: JSON.stringify(listRuns(), null, 2) },
{ uri: uri.href, mimeType: "application/json", text: JSON.stringify(listRuns()) },
],
}),
);
Expand Down
11 changes: 8 additions & 3 deletions src/server/result.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,15 @@
*/
import type { CallToolResult } from "@modelcontextprotocol/sdk/types.js";

/** Text + structured JSON result. */
/**
* Text + structured JSON result. The text block carries compact (non-indented)
* JSON: the pretty-print whitespace only inflated the wire/token cost with no
* gain for the consuming model, which reads the same data from
* `structuredContent` anyway. The two stay semantically equivalent (MCP spec).
*/
export function jsonResult(payload: Record<string, unknown>): CallToolResult {
return {
content: [{ type: "text", text: JSON.stringify(payload, null, 2) }],
content: [{ type: "text", text: JSON.stringify(payload) }],
structuredContent: payload,
};
}
Expand All @@ -21,7 +26,7 @@ export function imageJsonResult(
return {
content: [
{ type: "image", data: base64, mimeType },
{ type: "text", text: JSON.stringify(payload, null, 2) },
{ type: "text", text: JSON.stringify(payload) },
],
structuredContent: payload,
};
Expand Down
21 changes: 20 additions & 1 deletion src/server/tools/act.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,25 @@ import { withSession } from "./with-session.js";
type Shape = Record<string, z.ZodTypeAny>;
type Build = (a: Record<string, unknown>) => ActionInput;

/**
* `ActionResult` (interfaces/types.ts) has an index signature — model extra
* keys permissively. Shared with wait.ts (`waitForCondition` also returns an
* `ActionResult`, tagged `wait_for` with extra `condition`/`value` fields).
*/
export const actionResultShape = z
.object({
type: z.string(),
ok: z.boolean(),
target: z.string().optional(),
strategy: z.string().optional(),
error: z.string().optional(),
ms: z.number().optional(),
})
.catchall(z.unknown());

/** Every `actTool` handler returns `{ result, url }` via `jsonResult`. */
const actOutputShape = { result: actionResultShape, url: z.string() };

function actTool(
server: McpServer,
sessions: SessionManager,
Expand All @@ -25,7 +44,7 @@ function actTool(
inputSchema: Shape,
build: Build,
): void {
server.registerTool(name, { title: name, description, inputSchema }, async (args) => {
server.registerTool(name, { title: name, description, inputSchema, outputSchema: actOutputShape }, async (args) => {
const a = args as Record<string, unknown>;
return withSession(sessions, String(a.sessionId), async (s) => {
const action = build(a);
Expand Down
3 changes: 3 additions & 0 deletions src/server/tools/autoscroll.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ import type { SessionManager } from "../../session/manager.js";
import { jsonResult } from "../result.js";
import { withSession } from "./with-session.js";

const autoScrollOutputShape = { rounds: z.number(), height: z.number(), url: z.string() };

/** Register `browser_autoscroll`. */
export function registerAutoScrollTool(server: McpServer, sessions: SessionManager): void {
server.registerTool(
Expand All @@ -26,6 +28,7 @@ export function registerAutoScrollTool(server: McpServer, sessions: SessionManag
minCount: z.number().int().positive().optional(),
delayMs: z.number().int().nonnegative().optional(),
},
outputSchema: autoScrollOutputShape,
},
async (args) => {
const a = args as Record<string, unknown>;
Expand Down
7 changes: 7 additions & 0 deletions src/server/tools/clipboard.ts
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,12 @@ export async function applyClipboard(
return evalScript<string>(page, READ_SCRIPT);
}

/** Merged success shape across the 2 actions (read/write). */
export const CLIPBOARD_OUTPUT_SHAPE = {
written: z.literal(true).optional(),
text: z.string().optional(),
};

/** Register `browser_clipboard`. */
export function registerClipboardTool(server: McpServer, sessions: SessionManager): void {
server.registerTool(
Expand All @@ -60,6 +66,7 @@ export function registerClipboardTool(server: McpServer, sessions: SessionManage
action: z.enum(["read", "write"]),
text: z.string().optional(),
},
outputSchema: CLIPBOARD_OUTPUT_SHAPE,
},
async (args) => {
const a = args as Record<string, unknown>;
Expand Down
Loading