[vybe]

Summary

• The turn walks the whole pool before it fails. SUB-003 used to remediate once per turn (one switch or one API-key fallback, one re-issue) and three surfaces then handed the retry back to the person. Now each refused attempt asks advance_subscription_walk for the next credential — another subscription, then the platform API key — and the same execution_id is re-issued with it as a request-scoped auth_override the agent applies to that one spawn. Nothing in the DB or the container moves until a trial serves; the serving subscription is then committed once (assign + hot-reload + one notification), with a compare-and-set on the assignment so two workers never both commit.
• Part 2 — the API key is a rung, not a reassignment. fallback_to_api_key no longer clears the assignment, flips use_platform_api_key or restarts the container. The key is tried by override only, skipped while recently refused (auth/billing ⇒ 2h, a 429 ⇒ 5 min), never offered to an agent whose use_platform_api_key is explicitly off, and a key refusal is one more exhausted candidate. "Switched to the platform API key" / "Returned to subscription" are transition notifications; the assignment is preserved throughout.
• Honest ending. Only when every credential refused does the Workspace / /chat show the usage-limit message, naming the earliest reset across everything tried; the attempts trail is appended to the FAILED row (Tried: A (rate limit) -> B (auth) -> platform API key (billing); every candidate refused.), carried on TaskExecutionResult.subscription_switch.attempts and the auto_retry audit rows, and one operator alert per agent per hour names what was tried. The "send that again" copy remains only when the turn's time budget ran out before a re-issue.

Bounds: the turn's remaining wall clock (#2789 — still the only ceiling) over a floor of settle + HTTP slack + 30 s, plus the walk's own rung cap (len(subscriptions) + 2, "already tried ⇒ exhausted"). Attribution is exact (the walk knows what it sent). Interactive triggers may take any untried skip-listed subscription as a last resort; autonomous ones only one whose last failure is ≥ 30 min old. Old agent images (no X-Trinity-Auth-Override echo) degrade to the #792 commit-and-retry per rung, capped at one restart-based rung and with no key rung.

Design note: the plan review (two independent reviewers) converged on replacing a container-level hot-reload + Redis "fallback active" marker with the per-spawn override — a marker would drift across a plain docker restart or a Redis outage and misattribute key refusals; the override is stateless and exact.

Changes

Agent server (docker/base-image/agent_server/): models.AuthOverride (SecretStr fields) on ParallelTaskRequest / ChatRequest; execution_env.build_execution_env(…, drop=) + auth_override_env_layers; both spawn sites apply it; routers/chat.py stamps X-Trinity-Auth-Override on success and on the HTTPException, stages the value for redaction, drops it for non-Claude runtimes; AgentRuntime.supports_auth_override.

Backend: services/subscription_auto_switch.py — SubscriptionWalk / Rung / advance_subscription_walk / commit_subscription_walk / apply_rung_legacy / api_key_rung / refusal + notice markers; selector gains exclude_ids + last_resort_min_age_seconds; earliest_known_reset drops past instants. task_execution_service._call_agent_with_retries and chat_execution_service._walk_chat_dispatch are the two dispatch loops. client_portal/service.py AUTH/BILLING ladder. db.set_execution_subscription (SUB-004 re-attribution, one-column UPDATE, no schema change) + CAS kwarg on assign_subscription_to_agent. failure_classifier.is_container_fault keeps a container-level 503 out of the walk. agent_runtime_state registers two EXEMPT dedupe keyspaces.

Docs: requirements/security.md §20.4, architecture/{execution,agent-runtime,backend}.md, feature-flows/subscription-auto-switch.md §#3470, docs/requirements/SUB-003…md history, learnings.md (two entries).

Test Plan

• tests/unit/test_3470_subscription_walk.py — 51 tests: state machine, last-resort rung, key rung, commit/CAS/notifications, legacy mode, four end-to-end turns through the real execute_task + real walk, /chat re-issue, portal ladder, agent env layers / header / redaction / SecretStr, review-hardening pins.
• test_792 / test_2789 re-seamed on advance_subscription_walk; test_2638's one-remediation test became "pre-dispatch switch feeds the walk and the origin is never retried"; pingpong / concurrency / auto-switch picks an alternative subscription without checking headroom — it can move agents onto a nearly-exhausted one #2409 / Default model claude-opus-5-5 fails on cached Claude Code 2.1.278, and the rejection is misclassified as auth (seat rotation + API-key fallback) #3012 / bug(execution): a chat turn continues whatever session was persisted last — after a long scheduled run it inherits that run's context, pays its compaction, and the execution record says nothing #2958 / env / reload-token suites green unmodified in contract.
• Full tests/unit tier green locally (two pre-existing test_2889 collection failures are a local No module named 'tests' cwd issue, unrelated).
• Manual: two subscriptions at their wall + a third with headroom → a Workspace message completes on the third with one "auto-switched" notification; with no headroom anywhere → one Failed card naming the reset, one operator alert.

Not covered (stated)

Pull pilots: a Workspace turn on a piloted agent is claimed from the durable queue and the pull sink has no re-delivery — follow-up issue filed. Multi-worker commits are CAS-guarded; trials on different workers remain independent by design.

Fixes #3470

🤖 Generated with Claude Code

[trinity-ability]

merge-train: batch validated on train/20261009-2324-b (#3504)