Repository navigation
[Feat] Let hosted Jev read screenshots (TypeSafe image input, opt-in) on the browser and desktop pixel-target paths #97
Copy link
Copy link
Open
Description
Why
TypeSafe's Jev 1.14 release (2026-10-14) turned on image input for the hosted API: "the Jev you all know and love (jev-1.13.0) was multimodal all along". Images ride in a top-level assets map and are referenced from state, instructions or criteria, behind the TypeSafe-Danger-Zone-Images: true header (Images docs). Both jev-1.13.0 (our pinned default since #93) and jev-1.14.0 accept them. The same model is also on Vercel AI Gateway with images in the state.
In system1-agents the hosted Jev path is text-only today:
JevModelnever setssupports_images, so it keeps the defaultFalsefrombase.py.- The browser front only captures a screenshot when the model says it reads images (
decision_model.py), so--model jevnever sees the page, while--model omnijev(in-process) and, in draft, [Feat] Use native JEV-VL for browser decisions #78 (served JEV-VL) and [Feat] Add a served OneJev multimodal decision backend (System One + media) #88 (served OneJev) do. - The desktop
--pixel-targetpath refuses every model except Cua-S1 4B multimodal (agents/desktop.py).
So the one backend the README benchmarks most cannot be compared with and without a screenshot, and users without a GPU have no screenshot-reading option except the in-process models.
Proposed scope (one PR, off by default)
- Opt-in setting (e.g.
S1A_JEV_IMAGES=1) on the direct TypeSafe path only. When set,JevModel.supports_imagesis true; each image inObservation.imagesgoes intoassetswith a fresh UUID reference placed instate(once, not per question, as the docs recommend), and the request carriesTypeSafe-Danger-Zone-Images: true. The OpenRouter path stays text-only and refuses the setting with a config error unless it is shown to forwardassetsand the header. - Image preparation per the documented limits: PNG/JPEG still, each side 280 to 1568 px (downscale the longest side to 1568; pad or reject below 280), request body under 32 MiB. Base64 JSON is fine for one screenshot; multipart is optional.
- Enable
--pixel-targetfor--model jevwhen images are on, with the same one-current-screenshot rule as Cua-S1. - Provenance: step records say whether an image was sent, its sent size, and
usage.input_tokenswhen returned (image tokens are billed as input tokens, so cost per step changes). - One measured comparison:
--model jevtext-only vs--model jevwith screenshots on the README's WebVoyager Allrecipes task (and the desktop visual fixture from [RFC] Local Cua-S1 4B and screenshot targets for #14 and #15 #30/[Feat] Add local Cua-S1 4B and screenshot target selection #32 if a Mac is available): success, wall time, cost, committed underdocs/results/with the model id.
Acceptance criteria
- With the setting unset, requests, records and tests are unchanged.
- Offline tests with a fake transport: assets map and header present only when images are on; every asset is referenced; oversize and undersize screenshots are resized or rejected before sending; the proxy path refuses the setting.
- Docs (
docs/decision-models.md,docs/configuration.md,docs/capabilities.md) label this "experimental (TypeSafe danger zone)" and do not claim a quality result without the committed run. - The comparison is reported honestly, including if the screenshot does not help or costs more.
Out of scope
- The Box question type ([Feat] Find desktop click targets with Jev's Box question instead of hand-written --pixel-target coordinates #98).
- Long-context header (
TypeSafe-Danger-Zone-Long-Context). - Changing the default Jev version ([Deps] Pin Jev on the direct TypeSafe path (it sends jev-latest) and re-run benchmarks on jev-1.14.0 #93 owns that).
References
- Jev 1.14 release post: https://typesafe.ai/blog/jev-1-14
- TypeSafe Images docs: https://docs.typesafe.ai/go/images
- MarkTechPost coverage (limits, pricing): https://www.marktechpost.com/2026/10/14/typesafe-ai-jev-1-14-images-box-long-context/
- Related: [Deps] Pin Jev on the direct TypeSafe path (it sends jev-latest) and re-run benchmarks on jev-1.14.0 #93 (Jev pin), [Feat] Use native JEV-VL for browser decisions #78 / [Feat] Add a served OneJev multimodal decision backend (System One + media) #88 (served image backends), [RFC] Local Cua-S1 4B and screenshot targets for #14 and #15 #30 / [Feat] Add local Cua-S1 4B and screenshot target selection #32 (desktop pixel targets), [Feat] Confidence-gated deferral: hand low-confidence browser steps to the chat model and measure the trade-off #94 (confidence-gated deferral)
Metadata
Metadata
Assignees
Labels
No labels
Activity