fix(webv2): stop retrying an enqueue that keeps failing with a 500

[kevin9327]

Summary

When enqueue_batch answers 5xx, webv2 treats the error as transient and re-queues the run with no attempt limit (features/queue/runtime.ts). A 500 caused by the request itself (e.g. the graph-validation crash in #9852) never succeeds, so the run was resubmitted forever, filling the server log, and the loop came back after every reload because the run is persisted.

With this change, a run whose submission gets 3 consecutive 5xx responses is marked failed with the server's error, the same way a 4xx is handled:

• Before: POST enqueue_batch → 500, retried every ~1–5 s with no end.
• After: three 500s in a row, then the run shows as failed: "The server failed this run 3 times in a row: …".

502, 503 and 504 still retry with no limit, since they usually mean the server is restarting or behind a proxy, and a lost/ambiguous response (a non-ApiError) still goes through reconciliation as before. The counter is per run and resets as soon as a submission succeeds or fails with a different kind of error.

Related Issues / Discussions

Closes #9854. This covers the unbounded-retry part. It doesn't change Queue panel visibility of runs that are still being retried; that's UI work that can follow separately. The server-side trigger is #9852 / #9853.

QA Instructions

• pnpm vitest run src/features/queue/runtime.test.ts: 66 passed. Two new tests:
  • repeated ApiError(500) → the run is failed after 3 enqueue attempts and isn't submitted again (the first test fails on main);
  • three ApiError(503) then success → still submitted, never failed.
• pnpm test: 620 files / 10175 tests passed.
• format:check, lint:oxc and lint:tsc pass. The architecture vitest suites (src/architecture) pass. The architecture:check inventory scripts need Node ≥ 22 (--experimental-strip-types), so I couldn't run them on the Node 20 here; this change doesn't touch imports or ownership.

Review

Remaining limitation: the retry count lives in memory, so after a page reload a run that is still pending gets up to 3 more attempts before it fails. Once it's marked failed, that state is persisted and the retries stop.

Compatibility / Rollout

No persisted-state or API changes.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Meaningful regression coverage added / updated where needed; obsolete tests/code removed
• Persisted-state and API changes include required migrations / compatibility validation (none)
• Relevant performance/efficiency opportunities considered; material claims have evidence
• Material review findings resolved and relevant checks rerun
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-14T22:38:40.775348Z	6723f38	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[JPPhoto]

@kevin9327 Thanks for the submission. Outside of the core cause that's being addressed in another PR, are there other cases that this is an issue?

[kevin9327]

Yes, I found one more, and #9853 doesn't fix it.

_get_type_tree_root_types calls issubclass(e[1], e[0]) on every pair of a collector's resolved item types. _resolve_item_input_types replaces any type that has an origin with its get_args(), so a Literal output resolves to its values. One example is max_seq_len: Literal[256, 512] on the FLUX loader (flux_model_loader.py#L38), and the FLUX.2 loaders have the same field. If that output feeds a Collect item, the type set becomes {256, 512}, and issubclass(512, 256) raises TypeError: issubclass() arg 1 must be a class.

Here's what I checked on main at 861210e33f:

• graph: flux_model_loader.max_seq_len → collect.item, then collect.collection → integer_collection.collection.
• Batch.model_validate({"graph": …}) raises that raw TypeError from validate_graph → validate_self. It isn't a ValidationError, so FastAPI never turns it into a 422.
• the same body POSTed to a FastAPI route that takes batch: Batch = Body() comes back as 500.
• with fix(graph): reject mismatched collector items for union collection inputs #9853's diff applied, I get the same TypeError.

On the client, that 500 goes through the generic 5xx branch (runtime.ts#L1389-L1404), so the run is re-queued with no limit. I reproduced this at the API level. I haven't checked whether the workflow editor lets you draw that edge.

More generally, any other unhandled exception inside session_queue.enqueue_batch also becomes a 500, because of the catch-all in session_queue.py#L269-L270. So each new validation bug like these will loop the same way until someone fixes it on the server. That's the reason for this PR: it caps the client side for bugs like these. I'm happy to send the Literal fix as a separate PR if you want it.