Skip to content

[Feat] Let hosted Jev read screenshots (TypeSafe image input, opt-in) on the browser and desktop pixel-target paths #97

Open

Description

@Birfy

Why

TypeSafe's Jev 1.14 release (2026-10-14) turned on image input for the hosted API: "the Jev you all know and love (jev-1.13.0) was multimodal all along". Images ride in a top-level assets map and are referenced from state, instructions or criteria, behind the TypeSafe-Danger-Zone-Images: true header (Images docs). Both jev-1.13.0 (our pinned default since #93) and jev-1.14.0 accept them. The same model is also on Vercel AI Gateway with images in the state.

In system1-agents the hosted Jev path is text-only today:

So the one backend the README benchmarks most cannot be compared with and without a screenshot, and users without a GPU have no screenshot-reading option except the in-process models.

Proposed scope (one PR, off by default)

  1. Opt-in setting (e.g. S1A_JEV_IMAGES=1) on the direct TypeSafe path only. When set, JevModel.supports_images is true; each image in Observation.images goes into assets with a fresh UUID reference placed in state (once, not per question, as the docs recommend), and the request carries TypeSafe-Danger-Zone-Images: true. The OpenRouter path stays text-only and refuses the setting with a config error unless it is shown to forward assets and the header.
  2. Image preparation per the documented limits: PNG/JPEG still, each side 280 to 1568 px (downscale the longest side to 1568; pad or reject below 280), request body under 32 MiB. Base64 JSON is fine for one screenshot; multipart is optional.
  3. Enable --pixel-target for --model jev when images are on, with the same one-current-screenshot rule as Cua-S1.
  4. Provenance: step records say whether an image was sent, its sent size, and usage.input_tokens when returned (image tokens are billed as input tokens, so cost per step changes).
  5. One measured comparison: --model jev text-only vs --model jev with screenshots on the README's WebVoyager Allrecipes task (and the desktop visual fixture from [RFC] Local Cua-S1 4B and screenshot targets for #14 and #15 #30/[Feat] Add local Cua-S1 4B and screenshot target selection #32 if a Mac is available): success, wall time, cost, committed under docs/results/ with the model id.

Acceptance criteria

  • With the setting unset, requests, records and tests are unchanged.
  • Offline tests with a fake transport: assets map and header present only when images are on; every asset is referenced; oversize and undersize screenshots are resized or rejected before sending; the proxy path refuses the setting.
  • Docs (docs/decision-models.md, docs/configuration.md, docs/capabilities.md) label this "experimental (TypeSafe danger zone)" and do not claim a quality result without the committed run.
  • The comparison is reported honestly, including if the screenshot does not help or costs more.

Out of scope

References

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions