From fb5af255af98fb4b5c8ad5024f8d360480ceac24 Mon Sep 17 00:00:00 2001 From: Antoine van der Lee <4329185+AvdLee@users.noreply.github.com> Date: Mon, 7 Sep 2026 20:11:11 +0200 Subject: [PATCH 1/3] Document agent protocol robustness improvements --- .../features/agentic-development/rocketsim-cli.md | 13 +++++++++---- .../features/agentic-development/rs1-protocol.md | 12 ++++++++++++ 2 files changed, 21 insertions(+), 4 deletions(-) diff --git a/docs/src/content/docs/docs/features/agentic-development/rocketsim-cli.md b/docs/src/content/docs/docs/features/agentic-development/rocketsim-cli.md index 19c6d69..25c6bbe 100644 --- a/docs/src/content/docs/docs/features/agentic-development/rocketsim-cli.md +++ b/docs/src/content/docs/docs/features/agentic-development/rocketsim-cli.md @@ -37,7 +37,7 @@ The CLI gives agents a compact workflow: That loop is fast because RocketSim is already connected to the Simulator. There is no reconnection overhead between steps, and the running app can cache and optimize work across repeated commands. -RocketSim resolves booted Simulators through the same device service used by the Mac app. Commands therefore work whether the Simulator is shown in Simulator.app or Xcode 27's Device Hub. In Device Hub, use Compact Mode and focus the device you want RocketSim to control. +RocketSim resolves booted Simulators through the same device service used by the Mac app. Commands therefore work whether the Simulator is shown in Simulator.app, Xcode 27's Device Hub, or booted headlessly with `simctl boot`. Without a focused Simulator window, RocketSim automatically targets the only booted Simulator. If several are booted, pass `--udid` to choose one. In Device Hub, use Compact Mode and focus the device you want RocketSim to control. ## Why RocketSim is fast for agents @@ -75,10 +75,12 @@ rocketsim elements [--udid ] [--agent] [--agent-mode nav|act|debug] The `--agent` flag is the recommended default for agent workflows. It returns compact rows inside the `rs/1` response: -- `nav` focuses on headings, tabs, navigation bars, and top-level controls +- `nav` focuses on headings, tabs, navigation bars, and top-level controls while omitting plain static text, images, and nested duplicate text composites - `act` includes interactive element identifiers, labels, roles, values, and state - `debug` returns the full hierarchy when an action fails or an element looks wrong +Both compact modes omit individual software-keyboard keys because keyboard visibility is already reported in the response header. They also omit elements whose frames are fully outside the device canvas. Debug and plain JSON output remain unchanged when you need the complete hierarchy. + RocketSim's element pipeline is designed for real app navigation. It can include visible controls from top bars, navigation bars, tab bars, and other chrome that agents often need to move through a flow. When web content or other complex views expose limited accessibility data, RocketSim can add recovery hints so the agent knows when to use visual context. ### Screen summary and annotated snapshots @@ -183,10 +185,11 @@ Agents can wait for screen changes or elements before continuing: ```bash rocketsim wait screen-changed rocketsim wait element --label "Continue" +rocketsim wait keyboard --state shown rocketsim wait keyboard --state hidden --timeout 1 ``` -This keeps agent flows from racing ahead before the app has finished navigating or rendering. +This keeps agent flows from racing ahead before the app has finished navigating or rendering. For keyboard waits, `shown` is an alias for `visible`; use `hidden` to wait for dismissal. When the perception backend itself is failing (rather than the predicate simply staying false), `wait` reports that backend failure as an `execution_failed` error instead of a misleading timeout, so agents can run `rocketsim doctor` and recover instead of retrying a wait that can never succeed. @@ -210,7 +213,7 @@ rocketsim interact biometric match rocketsim interact biometric nomatch ``` -`interact` is designed to work with fresh screen state. When an agent uses the Agent Skill, RocketSim can guide it toward safer command sequences and recovery paths if the screen changes between inspection and interaction. +`interact` is designed to work with fresh screen state. After dispatching an interaction, RocketSim actively refreshes snapshots during a bounded settlement window before computing the result delta. The returned `screen_changed` value therefore reflects the post-interaction screen instead of a stale cached snapshot. Use `interact activate` for an accessibility element that does not respond to coordinate taps, such as a hidden debug control. It performs an accessibility press on the resolved element instead of sending a HID tap. @@ -267,6 +270,8 @@ rocketsim interact long-press --label "Reorder" --duration 1.5 --screen latest RocketSim will first try semantic accessibility activation, which is more reliable than a coordinate tap when the visual affordance does not align perfectly with the accessibility frame. This matters for controls like toggles, list rows, and buttons where the tappable area is asymmetric. +When a selector matches several elements with the same label, RocketSim automatically chooses the only actionable match if the others are non-actionable containers around it. Genuinely ambiguous matches still return `multiple_matches`. + Coordinates are still available as a fallback when the element is visible on screen but not exposed with a stable label. ## Named swipe directions diff --git a/docs/src/content/docs/docs/features/agentic-development/rs1-protocol.md b/docs/src/content/docs/docs/features/agentic-development/rs1-protocol.md index d019c80..8f29774 100644 --- a/docs/src/content/docs/docs/features/agentic-development/rs1-protocol.md +++ b/docs/src/content/docs/docs/features/agentic-development/rs1-protocol.md @@ -103,6 +103,18 @@ When something goes wrong, the envelope switches to a typed error: Error codes such as `snapshot_changed`, `accessibility_unavailable`, and `network_extension_not_ready` are documented behaviors, and each carries context the agent can act on. That is what turns a failure into a recovery path instead of a dead end. +## Reliability and compact-output behavior + +The compact `nav` and `act` snapshot modes deliberately remove rows that do not help an agent act. Both omit software-keyboard keys—the header already reports keyboard visibility—and elements whose frames are fully outside the device canvas. `nav` also omits plain static text and images, plus nested text composites that repeat an ancestor's label. Full debug and plain JSON output remain unchanged. + +Interaction deltas are computed from actively refreshed snapshots during a bounded settlement window. This means `screen_changed` reflects the post-interaction screen instead of whichever snapshot happened to be cached when the interaction finished. + +Selector resolution also handles a common accessibility-tree ambiguity. If several matches share the same label and exactly one is actionable while the others are non-actionable containers around it, RocketSim selects the actionable element automatically. Other ambiguous selectors still return `multiple_matches`. + +Simulator selection does not require a visible Simulator window. When no window is focused, RocketSim targets the single booted Simulator, including one started headlessly with `simctl boot`. If multiple Simulators are booted, the typed error tells the agent to pass `--udid`. + +For keyboard synchronization, `wait keyboard --state shown` is accepted as an alias for `visible`. `hidden` continues to wait for keyboard dismissal. + The protocol is served by the running RocketSim Mac app, not by a standalone binary that starts from scratch on every call. The app stays connected to the Simulator, keeps screen state warm, and refreshes snapshots after each interaction. See the [RocketSim CLI documentation](/docs/features/agentic-development/rocketsim-cli) for the full command surface, including `elements --agent`, `interact`, `wait`, and batched `do` flows. ## History and adoption From d027e87cf4a6bee0162704d6210136a7c6ba22f3 Mon Sep 17 00:00:00 2001 From: Antoine van der Lee <4329185+AvdLee@users.noreply.github.com> Date: Tue, 8 Sep 2026 10:14:02 +0200 Subject: [PATCH 2/3] Document Browser Preview workflow --- .../agentic-development/agent-skill.md | 4 +- .../agentic-development/browser-preview.md | 78 +++++++++++++++++++ .../features/agentic-development/index.md | 2 + .../agentic-development/rocketsim-cli.md | 14 +++- .../agentic-development/rs1-protocol.md | 2 +- 5 files changed, 96 insertions(+), 4 deletions(-) create mode 100644 docs/src/content/docs/docs/features/agentic-development/browser-preview.md diff --git a/docs/src/content/docs/docs/features/agentic-development/agent-skill.md b/docs/src/content/docs/docs/features/agentic-development/agent-skill.md index 04988d4..7e21ea3 100644 --- a/docs/src/content/docs/docs/features/agentic-development/agent-skill.md +++ b/docs/src/content/docs/docs/features/agentic-development/agent-skill.md @@ -2,7 +2,7 @@ title: "RocketSim Agent Skill" description: "Install RocketSim's bundled Agent Skill so Cursor, Claude, Codex, Xcode, and other AI coding tools can navigate the iOS Simulator safely." sidebar: - order: 3 + order: 4 --- The RocketSim Agent Skill is the recommended way to connect AI coding tools to RocketSim. It teaches your agent how to use the version-matched `rocketsim` CLI, when to read visible elements, when to interact, how to recover after screen changes, and when to use a screenshot fallback. @@ -74,6 +74,7 @@ Once the skill is installed and RocketSim is running, your agent can: - Navigate multi-step app flows with fewer retries - Use compact screen summaries to spend fewer tokens per screen read - Capture a screenshot when visual context is needed +- Start a live [Browser Preview](/docs/features/agentic-development/browser-preview) for hands-on review and agent-ready visual feedback ## How to verify it works @@ -94,6 +95,7 @@ The same command discovers booted Simulators shown through Xcode 27's Device Hub ## Learn more - [RocketSim CLI](/docs/features/agentic-development/rocketsim-cli) — the commands agents use to inspect and interact with the Simulator +- [Browser Preview](/docs/features/agentic-development/browser-preview) — live interaction and visual feedback from your browser - [Agentic Development with RocketSim](/docs/features/agentic-development/) — scenarios, example prompts, and why RocketSim is effective for agent-driven Simulator automation - [CLI & Agent settings](/docs/settings/cli-and-agent) — installing and repairing the CLI and skill - [How we test AI agents for the iOS Simulator](/blog/testing-ai-agents-ios-simulator) — repeatable scenarios, benchmark methodology, and results diff --git a/docs/src/content/docs/docs/features/agentic-development/browser-preview.md b/docs/src/content/docs/docs/features/agentic-development/browser-preview.md new file mode 100644 index 0000000..34efe66 --- /dev/null +++ b/docs/src/content/docs/docs/features/agentic-development/browser-preview.md @@ -0,0 +1,78 @@ +--- +title: "Browser Preview" +description: "Stream and control a booted iOS Simulator from your browser, inspect its accessibility tree, and turn visual feedback into an agent-ready prompt." +sidebar: + order: 2 + badge: + text: Beta + variant: caution +--- + +Browser Preview is a **Beta** feature that serves a live, interactive view of your iOS Simulator at a local URL. You can review an app without keeping Simulator.app in front of you, interact with the screen from your browser, and turn visual feedback into a prompt for your AI coding agent. + +The preview includes the correct device bezel and lets you switch between booted Simulators. Live input supports clicks, drags, scrolling, and keyboard typing, so you can move through a flow while reviewing it. + +## Start a Browser Preview + +RocketSim must be running with at least one Simulator booted. Start the preview from Terminal: + +```bash +rocketsim preview +``` + +The command prints a token-protected URL on `127.0.0.1`. Open that URL in your browser to start reviewing the active Simulator. + +Choose a different localhost port when the default is already in use: + +```bash +rocketsim preview --port 5000 +``` + +When several Simulators are booted, target one by UDID: + +```bash +rocketsim preview --udid +``` + +You can also switch between booted Simulators from the device menu above the preview. + +## Interact from the browser + +The live screen behaves like the Simulator: + +- Click to tap +- Drag to perform a touch gesture +- Scroll with your mouse or trackpad +- Click the preview and type with your keyboard +- Use the controls below the device for Home and Lock + +The device bezel matches the selected Simulator, giving your review the same visual context as the physical device. + +## Review the accessibility tree + +The **Accessibility** panel shows the visible elements RocketSim can inspect. Select an element in the tree to highlight its frame on the captured screen and add focused feedback. + +Use this panel to verify labels, understand which visual control an accessibility element represents, and avoid vague feedback such as "the button near the bottom." + +## Inspect the screen visually + +Option-click a visible element for one-off inspection, or click the crosshair in the **Visual Feedback** panel to enter inspect mode. While inspecting: + +- Hover over the screen to highlight the element under your pointer +- Click an element to select it +- Drag a marquee around several elements to review them as a group +- Press **Esc** to leave inspect mode + +Add your feedback to the selected element or group. RocketSim keeps the screen state that the feedback belongs to, even if you continue navigating afterward. + +## Review captured screen history + +Each feedback item is grouped under the screen where you created it. Select a captured screen from the **Visual Feedback** panel to revisit that state and confirm your comments against the original UI. + +The history view pauses live interaction while you inspect the capture. Choose **Return to live** or press **Esc** to continue with the current Simulator screen. + +## Generate an agent-ready feedback prompt + +After adding feedback, click **Copy prompt**. RocketSim creates a structured prompt containing the Simulator details, captured screen titles, selected accessibility elements, their frames, and your comments. + +Paste the prompt into your AI coding agent. The element-level context gives the agent a concrete implementation target while the captured history keeps feedback from different screens organized. diff --git a/docs/src/content/docs/docs/features/agentic-development/index.md b/docs/src/content/docs/docs/features/agentic-development/index.md index 6271cb4..3a62d8a 100644 --- a/docs/src/content/docs/docs/features/agentic-development/index.md +++ b/docs/src/content/docs/docs/features/agentic-development/index.md @@ -18,6 +18,7 @@ With the RocketSim Agent Skill installed from **Settings → CLI & Agent**, your - Stay in a tight interaction loop without rebuilding context between steps - Use compact screen summaries to spend less context per UI read - Fall back to screenshots when accessibility data is not enough +- Open a live Browser Preview for hands-on testing and visual feedback ## What you can do with it @@ -102,6 +103,7 @@ For most AI coding tools, install the [RocketSim Agent Skill](/docs/features/age ## Learn more - [RocketSim CLI](/docs/features/agentic-development/rocketsim-cli) — how agents inspect and interact with the Simulator +- [Browser Preview](/docs/features/agentic-development/browser-preview) — stream, control, and visually review a Simulator from your browser - [Agent Skill](/docs/features/agentic-development/agent-skill) — how to install the recommended agent workflow - [CLI & Agent settings](/docs/settings/cli-and-agent) — how to install the CLI and skill from RocketSim - [How we test AI agents for the iOS Simulator](/blog/testing-ai-agents-ios-simulator) — repeatable scenarios, benchmark methodology, and results diff --git a/docs/src/content/docs/docs/features/agentic-development/rocketsim-cli.md b/docs/src/content/docs/docs/features/agentic-development/rocketsim-cli.md index 25c6bbe..a051328 100644 --- a/docs/src/content/docs/docs/features/agentic-development/rocketsim-cli.md +++ b/docs/src/content/docs/docs/features/agentic-development/rocketsim-cli.md @@ -2,7 +2,7 @@ title: "RocketSim CLI" description: "Install and use RocketSim's built-in CLI to inspect visible elements, automate interactions, and give agents a fast path into your running Simulator." sidebar: - order: 2 + order: 3 --- RocketSim includes a built-in CLI that lets agents inspect visible UI and interact with the Simulator through the running RocketSim Mac app. The app stays connected to the Simulator, keeps useful state warm, and exposes a compact command line surface for agents and local automation. @@ -65,6 +65,16 @@ Returns the currently focused simulator as JSON, including name, runtime, and UD rocketsim simulator focused ``` +### Browser Preview + +Starts a live, interactive Simulator preview and prints its local URL: + +```bash +rocketsim preview +``` + +Pass `--port ` to choose a localhost port or `--udid ` to target a specific booted Simulator. See [Browser Preview](/docs/features/agentic-development/browser-preview) for the interactive controls and visual feedback workflow. + ### Visible elements Returns the accessibility elements currently visible on screen. @@ -268,7 +278,7 @@ rocketsim interact tap --type Button --label "OK" --screen latest rocketsim interact long-press --label "Reorder" --duration 1.5 --screen latest ``` -RocketSim will first try semantic accessibility activation, which is more reliable than a coordinate tap when the visual affordance does not align perfectly with the accessibility frame. This matters for controls like toggles, list rows, and buttons where the tappable area is asymmetric. +Selector-based taps resolve the matching accessibility element, then send a precise HID tap to its frame. Use `interact activate` when you explicitly need an accessibility press (`AXPress`), such as for an invisible control or one that ignores coordinate hit-testing. When a selector matches several elements with the same label, RocketSim automatically chooses the only actionable match if the others are non-actionable containers around it. Genuinely ambiguous matches still return `multiple_matches`. diff --git a/docs/src/content/docs/docs/features/agentic-development/rs1-protocol.md b/docs/src/content/docs/docs/features/agentic-development/rs1-protocol.md index 8f29774..6a31289 100644 --- a/docs/src/content/docs/docs/features/agentic-development/rs1-protocol.md +++ b/docs/src/content/docs/docs/features/agentic-development/rs1-protocol.md @@ -2,7 +2,7 @@ title: "rs/1 Protocol: RocketSim's Agent Protocol for the iOS Simulator" description: "rs/1 is RocketSim's agent protocol for the iOS Simulator: one compact JSON envelope, typed errors, and screen hashes for reliable AI agent automation." sidebar: - order: 4 + order: 5 label: "rs/1 Protocol" head: - tag: script From 1bbb6e208e523bd4a5fa33e60b9e11f3e5553442 Mon Sep 17 00:00:00 2001 From: Antoine van der Lee <4329185+AvdLee@users.noreply.github.com> Date: Tue, 8 Sep 2026 10:15:22 +0200 Subject: [PATCH 3/3] Correct selector tap semantics: semantic activation first, HID fallback Selector-based single-touch taps try performSemanticTap before falling back to a HID tap; only coordinate and multi-touch taps go straight to HID. Restores the accurate description and clarifies when interact activate is the right tool. --- .../docs/docs/features/agentic-development/rocketsim-cli.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/content/docs/docs/features/agentic-development/rocketsim-cli.md b/docs/src/content/docs/docs/features/agentic-development/rocketsim-cli.md index a051328..30b0014 100644 --- a/docs/src/content/docs/docs/features/agentic-development/rocketsim-cli.md +++ b/docs/src/content/docs/docs/features/agentic-development/rocketsim-cli.md @@ -278,7 +278,7 @@ rocketsim interact tap --type Button --label "OK" --screen latest rocketsim interact long-press --label "Reorder" --duration 1.5 --screen latest ``` -Selector-based taps resolve the matching accessibility element, then send a precise HID tap to its frame. Use `interact activate` when you explicitly need an accessibility press (`AXPress`), such as for an invisible control or one that ignores coordinate hit-testing. +Selector-based taps first try semantic accessibility activation, which is more reliable than a coordinate tap when the visual affordance does not align perfectly with the accessibility frame — think toggles, list rows, and buttons with asymmetric tappable areas. When semantic activation is unavailable, RocketSim falls back to a precise HID tap at the element's center. Coordinate taps and multi-touch taps always use HID directly. Use `interact activate` when you explicitly need an accessibility press without any HID fallback, such as for a hidden debug control that ignores coordinate hit-testing. When a selector matches several elements with the same label, RocketSim automatically chooses the only actionable match if the others are non-actionable containers around it. Genuinely ambiguous matches still return `multiple_matches`.