Skip to content

Document live agent preview context #1159

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AvdLee wants to merge 1 commit into from
Closed

Document live agent preview context #1159

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion docs/src/content/docs/docs/features/agentic-development/agent-skill.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ For you as a developer, this means you only have to keep RocketSim updated to th

The current skill teaches agents to start with a compact `nav` or `act` screen read, prefer labels over coordinates, guard interactions with `--screen latest`, and combine known sequential actions with `rocketsim do`. It uses interaction deltas and concrete waits to avoid unnecessary screen reads. Debug snapshots and screenshots are fallbacks when normal accessibility output is insufficient.

Simulator-targeting commands also ensure a live, interactive browser preview is available. RocketSim includes its URL in the command's `context.preview_url`, and the skill tells the agent to open that URL in a user-visible IDE browser and share it at the start and end of the task. This lets you follow the agent's work and click through the final Simulator state without switching away from your coding tool.

The agent should run `rocketsim doctor` only when setup appears broken: the CLI cannot connect, no Simulator can be found, or perception and interactions fail unexpectedly. Routine navigation should start with `rocketsim screen` or `rocketsim elements --agent`.

## What the agent can do after setup
Expand All @@ -73,6 +75,7 @@ Once the skill is installed and RocketSim is running, your agent can:
- Press simulator hardware buttons like Home, Lock, or Siri
- Navigate multi-step app flows with fewer retries
- Use compact screen summaries to spend fewer tokens per screen read
- Follow the live Simulator and review its final state in an IDE browser
- Capture a screenshot when visual context is needed

## How to verify it works
Expand All @@ -87,7 +90,7 @@ Then open your AI coding tool and try:

> Use RocketSim to navigate through `<your_app_name>` in the Simulator

If the skill is installed, RocketSim is running, and your app is already open in the Simulator, the agent should detect RocketSim, read the visible UI, and start interacting with the app based on what is on screen.
If the skill is installed, RocketSim is running, and your app is already open in the Simulator, the agent should detect RocketSim, read the visible UI, and start interacting with the app based on what is on screen. It should also share a `Live preview:` URL and open it in the IDE browser when the coding tool provides one.

The same command discovers booted Simulators shown through Xcode 27's Device Hub. If `doctor` cannot find one, put Device Hub in Compact Mode, focus the device, and retry. For network-condition tests, the agent may need you to approve RocketSim's Network Extension once; macOS does not allow an agent to complete that approval headlessly.

Expand Down
5 changes: 5 additions & 0 deletions docs/src/content/docs/docs/features/agentic-development/rs1-protocol.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,9 @@ Every agent-facing RocketSim CLI command emits one JSON envelope. The `rs` field
{
"rs": "1",
"ok": true,
"context": {
"preview_url": "http://127.0.0.1:4995/?token=..."
},
"data": {
"mode": "act",
"screen": "a3f291bc",
Expand All @@ -83,6 +86,8 @@ Every agent-facing RocketSim CLI command emits one JSON envelope. The `rs` field
}
```

The optional top-level `context` contains metadata that applies to the command result rather than its payload. For Simulator-targeting commands, RocketSim automatically ensures a browser preview is running and returns its URL as `context.preview_url`. Agents can open that URL in a user-visible IDE browser so you can follow their work and interactively review the final state. Older clients can ignore the optional field.

The `rows` array shows the compact element rows concept: instead of a nested accessibility tree, each visible element becomes one pipe-delimited row with an ephemeral id, role, label, value, and state. The agent reads a handful of short rows, picks an element, and acts on it by id or label.

The `screen` field is the screen-state hash. When an agent passes that hash (or the `latest` sentinel) along with an interaction, RocketSim fails fast if the screen changed in between, instead of tapping the wrong element.
Expand Down
Loading