Bound the gateway request path: caching, backpressure and pagination

[anilguleroglu]

Summary

A second pass over the load path, this time on what a request actually touches on its way through the gateway — Model Hub, Knowledge Engine, and the work each of them queues behind the response. Every item here is a place with no ceiling rather than a wrong one: a lookup repeated per call, a fan-out sized by user data, a backlog that grows with the request rate, a list that returns a whole collection.

Stacked on #237 — it shares files with that branch (quotaGuard, ragService), so basing it on main would only manufacture conflicts. Review after #237; it retargets to main cleanly once that merges.

Changes

• Provider rows are cached (PROVIDER_RECORD_CACHE_TTL_SECONDS, default 60). The SDK client was already pooled, but the row describing it was re-read on every call — one Knowledge Engine query (embedding → vector store → reranker) spent three uncached reads before doing any work. What's cached is the row as stored: credentialsEnc stays sealed under PROVIDER_ENCRYPTION_SECRET, decryption stays per call, plaintext credentials are never cached (test asserts this). Provider create/update/delete invalidate every affected project key; TTL 0 keeps the ciphertext out of the cache backend entirely.

• Guardrail configs are cached (GUARDRAIL_CONFIG_CACHE_TTL_SECONDS, default 30). A guarded model resolved its guardrail on the request path — twice when both an input and an output guardrail are set — and went to the database each time. Guardrail edits invalidate.

• Parent-window retrieval is bounded — the sharpest one. Expansion reads a whole document per matched chunk (up to INLINE_SOURCE_MAX_CHARS = 250k inline, or a full object download beyond that), and did so for every match of every query: no cap, unbounded Promise.all, no reuse. A topK-20 query spanning 20 documents pulled megabytes of text into heap to compute windows it had computed a second earlier. Now capped (RAG_PARENT_SOURCE_MAX_DOCUMENTS, 25), bounded in flight (…_CONCURRENCY, 4), and cached (…_CACHE_TTL_SECONDS, 60; only sources under …_CACHE_MAX_CHARS are cached, so the fix doesn't trade read amplification for a memory problem). When the cap truncates it logs — the remaining matches keep their chunk text rather than silently returning narrower context.

• Background work is queued, not started on arrival. Every usage write, audit row and trace payload became a live promise in an unbounded Set: the backlog grew with the request rate, competed with foreground queries for the same 10-connection pool, and held trace bodies (up to TRACING_MAX_BODY_SIZE_MB) in heap until it drained. Now a bounded number of runners (ASYNC_TASK_MAX_CONCURRENT, 32) drains a queue. Shedding is opt-in per call site: telemetry (trace ingest, token-last-used) is droppable and sheds oldest-first past ASYNC_TASK_MAX_QUEUED; usage, billing and audit writes are always kept — losing them corrupts records the product bills and answers audits from.

• Rate-limit counters are written as one batch. A guarded request touches a counter per metric per window — up to 25 — and issuing them separately let one caller hold every pool connection while its own limits were checked. New incrementRateLimits / incrementCounters on both contracts: MongoDB does one bulkWrite + one read-back, Redis one pipeline, the in-process providers keep looping (no round trip there to save). Behaviour note: the batch now fails as a unit where individual counters used to fail one at a time — "enforcement unavailable" is the same answer either way, and a partially-applied set would leave the windows disagreeing about how much of the request was counted.

• Pagination on the list endpoints that grow with usage. Knowledge Engine documents (client + dashboard), agents, tools and prompts returned whole collections; they now take limit/offset through a shared helper (default 100, max 500) and return pagination metadata. Where a count query filters identically to its list it reports total — countRagDocuments was realigned to match listRagDocuments for exactly this reason, since a total derived from different filters is worse than none. Elsewhere it reads one row past the window (takePage), which is cheaper and cannot drift.

Deliberately not paged: /client/v1/models serves the OpenAI-compatible discovery list, which SDK clients expect whole. Batches already had a bounded default (50, max 500). The remaining list endpoints (projects, members, license, providers, budgets, automations) are admin-managed and bounded by configuration rather than usage — say the word and I'll extend the same helper to them.

⚠️ Behaviour change: the four paged endpoints now default to 100 rows where they previously returned everything. pagination.hasMore/total make the truncation visible, and both are documented in openapi.yaml.

Validation

• npm run lint — 0 errors (164 pre-existing warnings)
• npm run test — 3694 passed, 0 failures. 5 test files fail to start because mongodb-memory-server cannot download its binary in this sandbox (403/502 from fastdl.mongodb.org); identical on a clean checkout, and the [sqlite] halves of the same parity suites pass, which is what exercises the new SQLite pagination.
• npm run build
• npm run docs:build
• npm run test:endpoints — 217 uncovered / 137 new, byte-identical to the base branch.

31 new tests: the provider and guardrail caches (including that nothing decrypted is cached), the background queue's ceiling, shedding order and never-shed guarantee, the batched counters (one call, correct rejection, fail-closed, provider routing), the parent-window cap, and the pagination helpers. The shared test setup now clears the core cache between cases — request-path caches outlive a single test in one process, and one case's fixtures must not answer the next one's reads.

Release Notes

• Docs updated when behavior changed — .env.example (11 new settings), openapi.yaml (pagination params on four endpoints)
• Security-sensitive changes reviewed — the provider cache stores only the sealed credential blob, never plaintext; a decrypt failure drops the cached row before retrying so a rotation cannot be masked by a stale entry
• License or policy files updated if repo-facing behavior changed — not applicable

---

Generated by Claude Code