Skip to content

[Feat] Served OneJev multimodal decision backend (--model onejev) - #101

Merged
hsliuustc0106 merged 3 commits into
mainfrom
issue-88-onejev-served
Oct 30, 2026
Merged

hsliuustc0106 merged 3 commits into
mainfrom
issue-88-onejev-served

Conversation

@Birfy

@Birfy Birfy commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator

Required human contributor answers

Per the repo owner's (Danyang, @Birfy) standing instruction, these are filled from the issue (#88) instead of waiting for separate human answers.

  1. What outcome are you trying to achieve?

    Answer: Agents can call a served OneJev multimodal System One model over HTTP (--model onejev) for screenshot + text choice/noul, without cloning OmniJev into the process ([Feat] Add a served OneJev multimodal decision backend (System One + media) #88).

  2. What is the highest-leverage variable for that outcome?

    Answer: Whether the HTTP body matches OneJev/qev's documented TypeSafe-compatible /v1/systemone plus media / <image:N> convention, with clean refusal when the observation has no image.

  3. How will this work move that variable rather than create busywork?

    Answer: It adds a dedicated served adapter (like laya-served / CLM) that posts observation + questions, records provenance (url, request_id, served_by), and is covered by fake-HTTP contract tests so CI does not need a GPU.

  4. Who is the customer or user, and what do they actually need?

    Answer: People running browser agents who want OneJev vision decisions against a local qev/flide server without the in-process OmniJev path.

After answering the four questions: is a PR needed at all to achieve this outcome?

Human decision and rationale (or explicit source reference): Yes; #88 proposes this adapter as the scope of one PR.

For a bug fix: link actual observed failure evidence.

Evidence: Not a bug fix: a new served backend (#88).

Why

system1-agents has in-process --model omnijev but no served OneJev/OmniJev worker client. OmniJev org's OneJev speaks TypeSafe /v1/systemone with a media field. Closes #88.

How

Start with s1a/decision_models/onejev.py, then factory.py / browse.py and tests/test_decision_models_onejev.py.

  • --model onejev → OneJevModel.from_env(): ONEJEV_URL (required), optional ONEJEV_MODEL (default OmniJev/OneJev1.6-4B), ONEJEV_API_KEY, ONEJEV_TIMEOUT_S (default 30).
  • Body: state with <image:N> / screen, media as PNG/JPEG/WebP data URLs, questions mapped via OmniJev's short prompt (string instructions, flattened element rows).
  • No image → MODEL_CALL_FAILED (OmniJev policy). Transport errors / malformed answers mapped the same way as other served clients.
  • raw keeps served_by, url, request_id, server_timing, sent_images digests (never image bytes).
  • Registered in DECISION_MODEL_NAMES and browser front only (not tool/rails/decide), same shape as omnijev / jev-vl-served.

What

  • New env vars ONEJEV_URL, ONEJEV_MODEL, ONEJEV_API_KEY, ONEJEV_TIMEOUT_S in docs/configuration.md.
  • Docs: docs/decision-models.md (backend row + optional smoke recipe), docs/capabilities.md (registration vs unverified hardware), docs/agents.md, docs/architecture.md.
  • NOTICE / THIRD_PARTY_LICENSES for OneJev/qev (Apache-2.0; weights not vendored).
  • CHANGELOG Unreleased entry.

Verification

  • uv run ruff format --check . && uv run ruff check . && uv run ty check: all clean
  • uv run pytest -q and scripts/smoke.sh: 1339 passed, 132 skipped; smoke OK
  • CHANGELOG.md and the docs say what the code does now

Offline tests cover happy path (media + marker), missing image, transport error (after retry), malformed answers, HTTP 500, from_env / build_model, and browser registration.

Demo / evidence

  • Video link, or exemption/blocker with follow-up: exempt for this PR — acceptance criteria are CI-complete with a stub server; live GPU / GGUF smoke is documented only and not gated. Hardware validation remains unmarked in the capability matrix until a pinned run lands.
  • Application/task → agent actions → independently checked result: not recorded (out of scope per [Feat] Add a served OneJev multimodal decision backend (System One + media) #88).
  • System1-Agents / System1-Omni commits, checkpoint and worker/frontend provenance: Agents branch issue-88-onejev-served @ b0a341b; no System1-Omni involved (talks to upstream OneJev qev/flide).

HTTP client for OmniJev/OneJev via qev/flide TypeSafe-compatible
/v1/systemone plus media and <image:N> state markers. Registers in
build_model and the browser front; contract tests use a fake HTTP
server (no GPU). Closes #88.

@Birfy Birfy left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review (Birfy self-check — comment only, not an approve)

Overall: Looks good and ready to land once CI is green, with one should-fix on timeout/deadline semantics. Scope matches #88: served OneJev over TypeSafe-compatible /v1/systemone + media, browser + build_model only, stub-server contracts, docs/NOTICE/CHANGELOG attribution without claiming live GPU validation.

What is solid

  • Wire shape matches upstream OneJev docs: state.screen / image_N → <image:N>, media: [{type, data: data-URL}], string instructions + flattened criteria via omnijev_question(..., prompt="short"). Happy-path test pins this.
  • Error policy: no image → MODEL_CALL_FAILED (same wording/status as in-process omnijev); empty/unsupported media types fail closed; transport / malformed answers / HTTP 500 mapped to MODEL_CALL_FAILED; missing ONEJEV_URL / bad timeout → MODEL_SERVICE_CONFIG_ERROR.
  • Provenance: raw keeps url, request_id, served_by, server_timing, sent_images digests and never the image bytes (covered by test).
  • Registration: DECISION_MODEL_NAMES + browser front only; tool/rails/decide/probe untouched — same shape as omnijev / jev-vl-served. Registration + browse slot routing tests look right.
  • Docs / legal: configuration, decision-models (incl. smoke recipe), capabilities (registration vs unverified hardware), NOTICE + THIRD_PARTY_LICENSES (Apache-2.0, weights not vendored, PyPI qev gotcha) are consistent with the code.
  • Default model id OmniJev/OneJev1.6-4B matches current OneJev 1.6 naming (not the older OneJev-4B from the issue text).

Should-fix before merge

  1. ONEJEV_TIMEOUT_S does not match the documented “retry included” deadline.
    docs/configuration.md / the decision-models row say the timeout is a per-decision deadline that includes the one refused-connection / 502 / 504 retry. OneJevClient.decide instead gives each attempt a full timeout_s (asyncio.wait_for(..., self._timeout_s)), so a retry can take ~2×. clock is accepted and stored but never used — looks like a half-port of ServedLayaClient’s shared-deadline pattern. Prefer: one monotonic deadline for the decision (and optionally share it with /health), subtract remaining on the retry, and either use clock or drop the unused parameter. Worth a contract test that a first transport blip then success still finishes under the configured budget (or at least that the second attempt’s wait_for uses remaining time).

Nits (non-blocking)

  • No 503 + Retry-After path (Laya-served has it). Fine if intentional for qev/flide; just don’t document parity with laya-served retries beyond 502/504.
  • Explicit warm() fails hard on non-200 /health, while _warm_once softens — fine, but a one-line comment on that split would help the next reader.
  • Optional: assert one successful retry after a single ConnectError or 502 (today only “two refuses → fail” is covered).
  • docs/architecture.md browser model lists still omit jev-vl-served / clm / laya-served in a couple of places (pre-existing); adding onejev beside omnijev is consistent with that incomplete list, but a follow-up tidy would help.

Tests

Contract suite covers happy path (media + marker), provenance, browser element flattening, missing image, transport after retry, malformed answers, HTTP 500, mapping helpers, from_env / build_model, and browser registration. Good enough for CI without a GPU.

CI

Spot-checked at review time: benchmarks already green; core / full still pending — merge only after those pass.

Not approving (own PR). Happy to re-check after the deadline fix lands on the branch.

@hsliuustc0106 hsliuustc0106 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

Adds served --model onejev (HTTP /v1/systemone + media) with factory/browser registration, contract tests, and docs/NOTICE — scope matches #88. Overall risk is medium (new multimodal HTTP decision path, opt-in via ONEJEV_URL). CI is green on head b0a341b and the branch is mergeable, but documented timeout semantics do not match the code, so this is not ready to approve/merge yet.

Findings

[major] ONEJEV_TIMEOUT_S is not a shared decision deadline

  • Where: s1a/decision_models/onejev.py (OneJevClient.decide, clock); docs/configuration.md / docs/decision-models.md
  • Evidence: Docs say the timeout is a per-decision deadline with the one refused-connection / 502 / 504 retry included. decide instead wraps each attempt in asyncio.wait_for(..., self._timeout_s), so a retry can run ~2×. clock is stored and never read (half-port of ServedLayaClient). Same gap as landed clm.py, but new public surface should not copy it while advertising Laya-style inclusive wording.
  • Why it matters: Operators who set ONEJEV_TIMEOUT_S to bound step latency get a weaker guarantee than documented; async/timeouts are a first-class review focus here.
  • Suggested fix: One monotonic deadline per decision (use clock or drop it); on retry use remaining = deadline - clock() for both httpx timeout and wait_for. Optionally share the budget with /health. Add a contract test that a first transport blip then success stays under the configured budget (or that attempt 2’s wait uses remaining).

Questions

None.

No other blockers found: wire shape, no-image refusal, provenance digests, registration (browser + build_model only), and stub-server tests look sound. CI on this head: core/full/benchmarks green.

Comment thread s1a/decision_models/onejev.py Outdated
started = time.perf_counter()
post = self._client.post(f"{self.url}/v1/systemone", json=body, headers=headers, timeout=self._timeout_s)
try:
response = await asyncio.wait_for(post, self._timeout_s)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[major] Per-attempt timeout vs documented shared deadline

docs/configuration.md / the decision-models row say ONEJEV_TIMEOUT_S includes the one refused-connection / 502 / 504 retry. This wait_for(..., self._timeout_s) (and the matching httpx timeout=) restarts the full budget on every attempt, so a retry can take ~2×. clock is accepted above and unused — prefer ServedLayaClient-style remaining budget (or drop clock and rewrite the docs to per-attempt). Add a contract test that a first transport blip then success finishes under the configured budget.

[Fix] Share OneJev's HTTP deadline across retries

@hsliuustc0106 hsliuustc0106 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

Re-checked tip 8a3d9ac after #105 landed the shared-deadline fix. Prior major finding is resolved: OneJevClient.decide now uses one monotonic deadline via clock, passes remaining budget to httpx/wait_for on retry, and contract tests cover remaining budget, exhausted budget, and stalled-retry cancel. Docs (ONEJEV_TIMEOUT_S inclusive wording) match the code. Scope still matches #88; CI green on this head (core/full/benchmarks); branch mergeable. No remaining blockers — approving.

Findings

None.

Light impact

  • Impact: new onejev path + factory/browse registration, docs/NOTICE; first-hop callers are build_model / browser front only.
  • Risk: medium (new opt-in multimodal HTTP decision path).
  • Missing tests: none obvious for the deadline fix.

@hsliuustc0106
hsliuustc0106 merged commit 113ef26 into main Oct 30, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feat] Add a served OneJev multimodal decision backend (System One + media)

2 participants