Bound provider calls, cache quota policies, and unblock queue consumers

[anilguleroglu]

Summary

Five load-path fixes from a capacity review of the console and its enterprise overlay. Each one addresses a place where the system had no ceiling under load rather than a wrong one: a provider call that could hang forever, a hot path that re-read the same two documents on every request, connections that were dropped without being closed, queue consumers pinned at one job, and document ingestion that occupied an HTTP worker for its whole run.

Behaviour is unchanged for existing clients. The one API-visible addition — deferred ingest — is opt-in, because the current 201-with-indexed-document contract is what callers read chunkCount from and query immediately afterwards.

Changes

• Provider calls honour GATEWAY_REQUEST_TIMEOUT_MS. The setting was defined and documented but never read, and Fastify's own requestTimeout/connectionTimeout are unset — so a stalled provider held its socket, its request context and (when streaming) its generation indefinitely. withResilience now applies the budget per attempt and hands the operation an AbortSignal, so a timeout aborts the upstream call rather than abandoning it. The chat paths pass that signal into LangChain; the streaming path chains it into the existing client-disconnect controller. timeoutMs: 0 opts out. All 29 withResilience call sites inherit this.

• Quota resolution caches the tenant license + policy set (QUOTA_POLICY_CACHE_TTL_SECONDS, default 30, 0 disables). /client/v1/chat/completions resolves limits three times before the model is reached, each one an uncached findTenantById plus listQuotaPolicies, against a ten-connection pool. Only the inputs are cached — scope/domain filtering still runs per request, so a cache hit can never hand one caller another's limits (covered by a test). Policy create/update/delete invalidates the entry; the TTL is the backstop.

• Evicted provider runtimes release their connections. Eviction was a bare pool.delete(), which dropped the reference while the pg pool or Elasticsearch client kept every socket open — an unbounded FD leak across tenant×provider combinations. Runtimes opt in structurally via close() (implemented for pgvector and the three Elasticsearch drivers). The close is deferred by PROVIDER_RUNTIME_DISPOSE_GRACE_SECONDS (default 300) so a request still holding the runtime is never cut off, and runs immediately on shutdown.

• Agent / MCP / browser / crawler consumers take a concurrency setting. All four were fixed at 1, so a cross-node job head-of-line blocked everything behind it until the queue's 60s invoke timeout failed the ones still waiting. Defaults: agent 4, MCP 4, browser 4, crawler 2 — crawler lower because a single job already fans out to as many as sixteen fetches and may drive a Chromium instance.

• Knowledge Engine ingest can run off the request path. Chunking, embedding and vector upsert ran inside the HTTP call with no timeout to bound them. Clients opt in with async: true and get 202 with a pending document that a queue consumer indexes. The source is committed before the job is published, and a boot sweep re-publishes pending documents, so a restart on the memory queue driver costs the indexing, never the content. indexPendingRagDocument returns early for an already-indexed document, so a duplicate delivery cannot double-embed.

Scope note: on the file-ingest path, file→text conversion still runs in-request — the bytes have to become text before the document can be persisted. Only chunking and embedding are deferred. Worth a follow-up.

New settings are documented in .env.example; the async flag and the 202 response are in openapi.yaml.

Validation

• npm run lint — 0 errors (164 pre-existing warnings)
• npm run test — 3663 passed, 0 failures. 5 test files fail to start because mongodb-memory-server cannot download its binary in this sandbox (403/502 from fastdl.mongodb.org); confirmed identical on a clean checkout of main.
• npm run build
• npm run docs:build
• npm run test:endpoints — 217 uncovered / 137 new, byte-identical to main; this change adds no uncovered routes.

Adds 28 tests: the timeout (firing, abort propagation, per-attempt budget, opt-out), the policy cache (including that filtering stays per-caller on a hit), deferred disposal (grace period, prefix invalidate, throwing close()), and both halves of the split ingest.

Release Notes

• Docs updated when behavior changed — .env.example, openapi.yaml
• Security-sensitive changes reviewed — the quota cache stores license limits and policies keyed by tenant and project, and per-caller scope filtering deliberately stays outside the cache
• License or policy files updated if repo-facing behavior changed — not applicable

---

Generated by Claude Code