feat(openrouter): pin and sort chat providers; LoCoMo v21 on glm-5.3-flash - #359
Open
fazpu wants to merge 5 commits into
Open
feat(openrouter): pin and sort chat providers; LoCoMo v21 on glm-5.3-flash#359fazpu wants to merge 5 commits into
fazpu wants to merge 5 commits into
Conversation
Provider routing existed only for embeddings; chat completions always used OpenRouter's ordinary marketplace routing, so a run could not be confined to a named set of hosts. That matters for benchmarking: two runs on the same model but different providers are not necessarily comparable, because quantization, context window and throughput all differ per host. Adds REMEMBERSTACK_OPENROUTER_CHAT_PROVIDER_ONLY (comma-separated or JSON list of provider slugs, e.g. z-ai,novita,deepinfra,gmicloud), mirroring the existing embedding parser. allow_fallbacks stays ON, which is not a loophole: 'only' already bounds the candidate pool, so failover moves between the listed providers and can never leave them. A long ingestion run would otherwise dead-letter a stage on one host's 5xx or 429. The allowlist is recorded in the selfhost attestation dict, because which hosts served a run is measurement provenance, not incidental configuration. Unset preserves today's behaviour exactly, so existing deployments are untouched. Tests cover the allowlist payload, the unset case, non-leakage into the embeddings path, and both env shapes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018eazkrM7j7scrwQUcUWbbS
Rolls the pinned protocol v19 -> v20 and moves every chat seat from openai/gpt-5.6-luna to z-ai/glm-5.3-flash, with chat completions restricted to the operator-named providers z-ai, novita, deepinfra and gmicloud. Embeddings stay on qwen/qwen3-embedding-8b via nebius. The model swap could not be a config change. The protocol ASSERTS its ingest bindings -- runner.py checks readiness.model_bindings == the expected map by exact dict equality -- so a deployment on a different model fails readiness and refuses to run. Rolling the protocol is the mechanism the repo already uses for this (v17 -> v18 -> v19), and the registry holds one current protocol at a time. The answer agent and judge move too, on operator instruction. That is worth stating plainly: the judge IS the scorer, so v20 numbers are NOT comparable with v19/v18 runs, and a score change cannot be attributed to the memory system rather than the grader. The protocol fingerprint changes accordingly (the golden is rolled to 76ac6811...), which is the intended signal. Also fixes a defect from the preceding commit: adding openrouter_chat_provider_only to the selfhost attestation dict broke the exact-equality readiness gate for every deployment. The key is now expected by the protocol and replayed by the shard runner, so a restored store reproduces the routing it was built with. Reasoning effort stays high for the ingest seats and none for answer/judge, exactly as v19 had it, so this run changes the model and the provider set and nothing else. 321 tests pass; ruff and pyright clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018eazkrM7j7scrwQUcUWbbS
main merged #348 ('keep complete direct answers') and took the v20 protocol name for it, with answer/judge still on gpt-5.6-luna -- a second session had already run conv-26 under that v20 on the benchmark host. Two protocols under one name would make those runs indistinguishable, so this rolls to v21 and keeps main's answer-prompt fix underneath. That is the third naming collision with parallel work this session, after the migration revision id and the resolver generation. v21 = main's v20 answer-completeness fix + every chat seat on z-ai/glm-5.3-flash + chat completions pinned to z-ai, novita, deepinfra, gmicloud. Fingerprint ba5ab840. 321 tests pass; ruff and pyright clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018eazkrM7j7scrwQUcUWbbS
A conv-48 ingestion dead-lettered 28 adjudicate_observations items, every one a 429 from the SAME provider: provider_name: DeepInfra provider_error_code: engine_overloaded limit_source: upstream_provider_shared_pool is_byok: false Two things made that concentrate on one host. OpenRouter's default routing weights price, and DeepInfra was both the cheapest and the slowest of the four allowed providers (measured: $0.0001003 at 30.8 tok/s, against Z.AI's $0.0001581 at 43.9) -- so it was selected first on every call and on all three engine retries. And allow_fallbacks did not rescue it, despite the preceding commit arguing it would. The docs define it as allowing backups when the primary is 'unavailable' and never enumerate the codes; empirically a provider-returned 429 arrives as 'Provider returned error' and is not re-routed. That assumption is now corrected in the code comments and covered by a test. Adds REMEMBERSTACK_OPENROUTER_CHAT_PROVIDER_SORT (price|throughput|latency) and sets throughput for the benchmark protocol. Sorting beats the obvious fix of dropping DeepInfra from the allowlist: a denylist hard-codes today's congested host, while sorting moves load off whichever host is slow at the time, and it preserves the operator's stated provider set instead of silently narrowing it. It is a bias, not a guarantee -- if every allowed provider is busy at once, calls still 429. Concurrency is part of the same decision: the identical allowlist ran 1,842 calls with one failure at 1 worker per stage and collapsed at extract 8 / normalize 6 / adjudicate 4 / embed 2. Written up in plan/analysis/openrouter_provider_routing_overload.md. 323 tests pass; ruff and pyright clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018eazkrM7j7scrwQUcUWbbS
The preceding commit blamed price-first routing: DeepInfra was the cheapest AND slowest of the four allowed providers, so the story went that it won selection on every call and every retry. That fit every observation and was wrong. It predicted that sort=throughput would fix the problem. After deploying the sort and replaying all 28 dead letters, all 28 failed again on DeepInfra -- which is what falsified it. The real cause is capability, not preference. Every chat completion here sends response_format json_schema strict, and of the four allowed providers only DeepInfra advertises structured_outputs for this model: DeepInfra structured_outputs=true Z.AI false Novita false GMICloud false So provider.only bounded the pool by NAME to four while the request bounded it by CAPABILITY to one. sort had nothing to reorder and allow_fallbacks had nowhere to fall back to. Isolated one parameter at a time: max_tokens alone and reasoning alone both route to Novita; the strict schema alone pins to DeepInfra and 429s. chat_provider_sort is kept -- biasing toward throughput is still sensible when more than one host is eligible -- but its docstring and test no longer claim it solves this, and the analysis records the wrong diagnosis alongside the right one so the next reader sees how it was caught. Practical rule now written down: when pinning providers, intersect each endpoint's supported_parameters with what the caller actually sends. An allowlist of four that supports one is a single point of failure in disguise. 323 tests pass; ruff and pyright clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018eazkrM7j7scrwQUcUWbbS
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds provider routing for chat completions (it existed only for embeddings), rolls the LoCoMo protocol to v21 on
z-ai/glm-5.3-flash, and records what a real ingestion run taught us about pinning providers.Why
Chat completions always used OpenRouter's ordinary marketplace routing, so a benchmark could not be confined to named hosts — and two runs on the same model but different providers are not necessarily comparable, since quantization, context window and throughput all differ per host.
What's here
REMEMBERSTACK_OPENROUTER_CHAT_PROVIDER_ONLY— allowlist of provider slugs (provider.only).allow_fallbacksstays on:onlyalready bounds the pool, so failover moves between listed providers and cannot leave them.REMEMBERSTACK_OPENROUTER_CHAT_PROVIDER_SORT—price | throughput | latency(provider.sort).RS-LoCoMo-Full-v21— every chat seat onz-ai/glm-5.3-flash, providers pinned toz-ai,novita,deepinfra,gmicloud, embeddings unchanged onqwen3-embedding-8bvia nebius. The model swap could not be config: the protocol asserts its ingest bindings by exact dict equality, so a deployment on a different model fails readiness outright. Answer agent and judge move too, on operator instruction — so v21 numbers are not comparable with v19/v20, and a score change cannot be attributed to the memory system rather than the grader.The finding worth reading
plan/analysis/openrouter_provider_routing_overload.md.A conv-48 ingestion dead-lettered 28 items, all
engine_overloaded429s from one provider. The first diagnosis was price-first routing picking the cheapest, most-contended host. It fit every observation, and it was wrong — it predictedsort: throughputwould fix things, and after deploying that, all 28 replays failed identically.The real cause is capability, not preference. Every chat call sends a strict
json_schema, and of the four allowed providers only DeepInfra advertisesstructured_outputsfor this model:structured_outputsSo
provider.onlybounded the pool by name to four while the request bounded it by capability to one.sorthad nothing to reorder;allow_fallbackshad nowhere to go. Isolated one parameter at a time:max_tokensalone andreasoningalone route to Novita; the strict schema alone pins to DeepInfra and 429s.Two things also documented because they cost real time:
allow_fallbacksdoes not give you 429 failover. The docs say "when the primary is unavailable" without naming codes; a provider-returned 429 arrives as"Provider returned error"and is not re-routed.The wrong diagnosis is kept in the analysis next to the right one, so the next reader sees how it was caught rather than just the conclusion.
Also fixed
Adding the allowlist to the selfhost attestation dict broke the exact-equality readiness gate for every deployment. The key is now expected by the protocol and replayed by the shard runner, so a restored store reproduces the routing it was built with.
Verification
323 tests pass; ruff and pyright clean. Behaviour is unchanged when both settings are unset, so existing deployments are untouched.
🤖 Generated with Claude Code
https://claude.ai/code/session_018eazkrM7j7scrwQUcUWbbS