Four kinds of agent share the loop, the models and the job folders: browser use over @playwright/mcp, computer use
over Cua Driver, games and embodied text, and rails that answer one question at a
hook of a running agent. The injection guard rail fails closed: a decision error quarantines the tool result too;
its runtime policy covers uncertain verdicts and results longer than it inspects.
| agent | front | what it measures | run |
|---|---|---|---|
allrecipes |
browser | browser use against the chat model: WebVoyager task Allrecipes--0, a vegetarian lasagna with over 100 reviews, 4.5 stars or more, for 6 | s1a run allrecipes --model jev --batch on --headed |
flights |
browser | browser use: wall clock on a Google Flights search | s1a run flights --model jev --batch on --profile-out run.json |
desktop |
tool | computer use: clicks toward --goal until the window shows --expect; the Calculator on macOS or Windows is the example |
s1a run desktop --app Calculator --goal "compute 12 times 7" --expect 84 --execute --clear "All Clear" --model jev --rethink off --episodes 1 |
ticket_router |
tool | correct routes on a seeded batch of 30 labelled tickets, five queues | s1a run ticket_router --model jev --rethink off --episodes 1 |
alfworld |
tool | success on unseen household tasks in text | s1a run alfworld --model jev --rethink on --episodes 12 --stride 11 |
game2048 |
tool | score and largest tile at a move cap | s1a run game2048 --model jev --rethink on --episodes 10 |
millionaire |
tool | winnings on a 15-question quiz ladder | s1a run millionaire --model jev --rethink off --episodes 5 |
blackjack |
tool | payoff per hand (RLCard) | s1a run blackjack --model jev --rethink off --episodes 100 |
injection_guard |
rail | precision and recall on a labelled injection set | s1a run injection_guard |
Every tool agent takes --model jev|clm|laya|laya-served|cua|llm|random|rule, --rethink on|off, --episodes N, --seed S,
--max-steps and --timeout, and writes a Harbor-shaped job folder under evals/results/<agent>/. A browser agent
takes --model jev|clm|laya|laya-served|cua|omnijev|llm, --goal and the same --rethink flag.
The desktop agent adds --rethink-attempts and --rethink-timeout and stalls after 3 actions without observed
progress. The browser front takes the same three names; docs/browser-front.md decision 19 describes its branch. A rail takes
--model jev|clm|laya|laya-served, the models that answer noul.
uv run python -m evals.table evals/results aggregates every job folder per eval and model into one table.
Every run prints one JSON object on stdout and nothing else there; s1a-mcp serves the same agents over stdio
with list_agents, run_agent and decide. Flags, exit codes and the job-folder layout:
architecture.md. The extras each agent needs and the keys: CONTRIBUTING.md. The --model values and the
models behind them: architecture.md.
Desktop recovery treats a changed progress observation as progress even when the completion score stays zero.
When screenshots are enabled, progress includes the decoded image dimensions and an RGBA pixel hash; capture IDs
and PNG metadata do not affect it. The hash is cached per snapshot, and image bytes stay outside the JSON state.
Persistent no-ops and repeated states can trigger recovery; progress clears the detection window without refunding
attempts or time. The full terminal fields, including next_action, are saved separately in episode.extra.terminal.
With bounded recovery enabled, a failed refresh or plan records a short failure category and a suggested next
action. Provider exception bodies are not stored. An empty or whitespace-only plan counts as a failed plan. The empty reply
still consumes the one attempt and its active time, and the fresh observation is kept. A task the policy still
answers BLOCKED after one or more replans is a failure too, even when a partial answer was fetched: the answer is
context only and the run carries the block reason and a next action, without being reported as a failed or exhausted
recovery. Failed and cancelled chat calls still count toward the run's call total.
If a call's token usage is unknown, the reported cost stays unknown rather than becoming zero.
s1a run <agent> --help lists every flag with its default. Beyond the shared ones: flights and allrecipes take --goal,
--batch on|off, --prefetch on|off, --goal-values on|off, --rethink on|off, --rethink-attempts,
--rethink-timeout, --profile-out and --logs-dir; desktop takes --app, --app-path, --window-title,
--goal, --expect, --execute, --plan, --clear, --text, --text-target, --text-mode, --verify-file,
--pixel-target and the same three --rethink flags; ticket_router takes --dataset and --batch-size;
injection_guard takes --labelled-set. The four games take no flag of their own.
--text supplies the content for a type:* action. --text-target selects an exact field label or native
identifier. The default --text-mode insert inserts at the current selection; replace sets the whole field.
Both require a fresh readback before the input is recorded as successful. Chinese and multiline text are
supported through native field replacement. --window-title selects the document when an app has several windows.
--verify-file requires the expected window state and a file changed during this episode whose UTF-8 content
matches --text. Without --execute, the first decision is only recorded as a plan.
The macOS fixture provides a local Laya demo and a fixed-plan execution check.
--pixel-target KEY=X,Y offers named points in screenshot fractions. Local Cua-S1 4B multimodal can select
among them when controls have no accessibility elements. Each click carries its screenshot capture ID.
Install the cua-four-b extra and set CUA_S1_VARIANT=4b, CUA_S1_MODALITY=multimodal.
See the macOS visual task for setup and verification commands.
allrecipes is the first Allrecipes task of the WebVoyager task set
(He et al., 2024, Apache License 2.0, attribution in NOTICE), verbatim: find a
vegetarian lasagna with more than 100 reviews, a rating of at least 4.5 stars, suitable for 6 people, and answer in
text. The run is headed
because the site answers a headless Chromium with a bot wall. scripts/browser_showcase.sh allrecipes 2 runs both models
twice with a frame after every browser call and renders the pair GIFs from the median run of each model; the numbers
and the replay are in benchmarks.md.
The guard asks one decision about each browser_snapshot or fetch_webpage result (MCP prefixes match by suffix),
after the tool call and before the next model call reads the result. The decision sees the first 6,000 characters
(MAX_CHARS) of the result's text, JSON-serialised when the result is not a string. One decision per result is the
whole budget: there is no second pass, retry or escalation. DecisionModelRail.ticks records each outcome:
| verdict | coverage | what the next model call reads | handling |
|---|---|---|---|
| act band, p ≥ 0.7 | any | a quarantine notice alone | act |
| decision failure: endpoint down, bad key, deadline, malformed answer | any | a quarantine notice alone (the rail fails closed) | act_on_failure |
| allow band, p ≤ 0.3 | complete | the result unchanged | pass |
| uncertain band, 0.3 < p < 0.7 | complete | the result unchanged | pass_uncertain |
| allow or uncertain band | partial: the result is longer than 6,000 characters | the inspected 6,000 characters, then a notice that the rest was withheld uninspected | withhold_uninspected |
The uncertain band is the model abstaining. The guard keeps pass-through for it, so as not to quarantine every page
it is unsure about; pass_uncertain names that choice, and a stricter deployment can quarantine it with its own
act or thresholds. No text the decision did not see reaches the model while the guard is attached; a page whose
useful content sits past character 6,000 loses it.
Each tick carries p, band, ms and input_tokens (or error, error_code and failed), with handling,
coverage (complete or partial), chars and inspected_chars. Ticks never hold the tool output's text. On a
failure, the tick and the warning log name only the error's status and cause type (MODEL_CALL_FAILED (ReadTimeout)),
never its message: an endpoint's error body can quote the submitted state back. Tools outside the two guarded ones
are neither asked about nor recorded.
The labelled-set scores in benchmarks.md measure the classifier on its
records. They say nothing about the runtime handling above, which tests/test_rails.py and
tests/test_rail_in_agent.py check offline with synthetic markers.
ticket_router routes locally labelled tickets to five queues: logistics, payment, returns, account and human. The
shipped set is 30 tickets in s1a/agents/_data/ticket_router_eval.jsonl, each with a title, a description and an
order status; the label is read by the scorer alone. --dataset points the agent at a
JSONL of your own with the same fields, and --batch-size caps the tickets per episode. The rules text the model
reads is RULES in s1a/agents/ticket_router.py.