ai-providers: shared max-tokens retry, openai_responses without openai_compatible internals, dead budget field removed #860

Goal

The last duplicates in ai-providers are gone: one implementation of the "retry once with a larger max_tokens" stream, the Responses provider no longer reaches into another provider's internals, no dead fields, and the per-run stream_options cache lives with the provider registry instead of a process-global static.

Context

• Max-tokens retry stream: anthropic_attempt_stream (anthropic.rs:851–926, with RetryState :740) and retry_stream / retry_stream_with_overload (openai_compatible.rs:880–1022, RetryState impl :854). 45 % identical; five #[allow(unused_assignments)] sit in exactly these loops (anthropic.rs:893, openai_compatible.rs:929,966,976,992).
• openai_responses.rs:36–39 imports openai_compatible_default_max_tokens, openai_compatible_model_max_output_tokens, retry_stream_with_overload, OpenAiStreamState, RawEvent, RetryKind, RetryState from openai_compatible, so that module is both a provider and a library. to_raw_stream exists in both anthropic.rs:139 and openai_compatible.rs:194.
• Dead field: budget: Arc<ProviderBudgetGuard> with #[allow(dead_code)] in OpenAiCompatibleProvider (openai_compatible.rs:253–254, comment :245) and OpenAiResponsesProvider (openai_responses.rs:67).
• Global mutable state: no_stream_options_urls() (openai_compatible.rs:543, OnceLock<Mutex<HashSet<String>>>), used at :573 and :728. A RateLimitRegistry already exists (rate_limit_budget.rs, built in crates/app-logic/src/ai_provider_factory.rs).
• Test helpers are duplicated: test_budget()/budget() appears seven times across the provider test modules; src/test_support.rs already exists for the crate.
• Specs: 0065 (max-tokens retry, ADR 0056), AI providers: classify, log and retry overload and in-stream errors #599 (overload retry), AI providers: report token usage (and provider-reported cost) for every completed request #609 (stream_options).

Requirements

• New module crates/ai-providers/src/common/max_tokens_retry.rs with one retry-stream implementation used by both the Anthropic and the OpenAI-compatible/Responses providers; the provider-specific parts (which stop reasons trigger it, how the body is rebuilt) are parameters or trait methods.
• Shared items that openai_responses.rs needs (RawEvent, RetryKind, RetryState, to_raw_stream, the OpenAI max-token tables if both OpenAI providers use them) move to common/ (or to a neutral openai_common.rs); openai_responses.rs imports nothing from crate::openai_compatible.
• Remove the unused budget field from both OpenAI providers (and the constructor parameter if it becomes unused; adapt ai_provider_factory.rs), or use it for real if a test shows it is needed — no #[allow(dead_code)] remains on it.
• The stream_options rejection cache becomes state owned by the provider registry / factory and is passed into the OpenAI-compatible provider; no process-global static remains. Behaviour per app run stays the same (a rejected endpoint+model key skips the option for later requests).
• One shared test_budget() (and other duplicated test fixtures) in src/test_support.rs.

Acceptance Criteria

• grep -rn "use crate::openai_compatible" crates/ai-providers/src/openai_responses.rs returns nothing.
• grep -rn "allow(unused_assignments)\|allow(dead_code)" crates/ai-providers/src returns no hit in the stream loops or on a budget field.
• grep -rn "OnceLock" crates/ai-providers/src finds no no_stream_options cache.
• Existing max-tokens retry tests pass (test_send_retry_doubles_max_tokens_without_override, test_context_limit_retry_logs_error_then_debug_line_with_old_and_new_budget, the truncated tool-call retry tests of all providers).
• A test shows that after a stream_options rejection the next request from a provider created by the same registry omits the option, and a provider from a fresh registry sends it again.
• The production part (tests excluded) of anthropic.rs and of openai_compatible.rs is below 1,200 lines each; report the new sizes in the PR.
• The number of tests in cargo test -p ai-providers -- --list does not decrease.
• Gate green (cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace, cargo build --workspace).

Technical Notes

• Security-relevant (incomplete tool calls are never forwarded, security invariant 5; Spec 0065); review priority elevated.
• crates/app-logic/src/ai_provider_factory.rs is the only constructor caller outside the crate; check grep -rn "OpenAiCompatibleProvider::new\|OpenAiResponsesProvider::new" across the workspace.
• Public API of the crate (lib.rs re-exports) stays source-compatible except for the removed constructor parameter.
• New code comments in English.

Spec impact: none

Related Work

• Previous steps: ai-providers: one shared formatter for command results and rejected actions (single fencing call site) #845, ai-providers: one shared HTTP send loop with 429/5xx retry for all providers #851, ai-providers: one shared SSE frame loop (FrameHandler) and response-to-event-stream helper #855