diff --git a/TRACKER.md b/TRACKER.md
index 103dff4..1a3e5cb 100644
--- a/TRACKER.md
+++ b/TRACKER.md
@@ -14,12 +14,14 @@
| | |
|---|---|
| **Last updated** | 2026-06-08 |
-| **Current phase** | Phase 7 — Pilot, Harden, GA (**0 / 10 steps**) — Phase 6 ✅ complete |
-| **Overall** | **74 / 84 steps** — Phases 0–6 complete |
-| **Next action** | **Step 7.1 — Load testing**: Locust suite; 1000 QPS sustained; p99 < 500 ms; chaos under load. |
+| **Current phase** | Phase 7 — Pilot, Harden, GA (**1 / 10 steps**) — Phases 0–6 ✅ complete |
+| **Overall** | **75 / 84 steps** — Phases 0–6 complete |
+| **Next action** | **Step 7.2 — Chaos engineering**: Chaos Monkey / LitmusChaos; kill each backend; verify the fallback chain holds (builds on the 7.1 in-process chaos harness). |
**Recently shipped**
+- **7.1** ✅ Load + chaos testing — a **chaos-under-load** CI gate (`eval/gateway_chaos_v0/`): drives the in-process gateway under concurrent load while injecting backend faults (`FaultSpec` + `Chaos{Vector,Keyword,Graph}RetrievalBackend` SPI wrappers behind real breakers) and asserts **graceful degradation** — no 5xx, 100% success, the failing backend's breaker opens (validating the Phase-4 breakers + fallback, builds no new resilience); `tests/perf/test_chaos_under_load.py` (timing-independent, `perf`-marked) + `task chaos-test`. An extended **Locust v1 suite** (weighted read/write mix + varied queries + a ramp `LoadTestShape`) + documented **acceptance targets** (≥ 1000 RPS sustained, e2e p99 < 500 ms) as a cluster runbook — [#164](https://github.com/officialCodeWork/AgentContextOS/pull/164)
+
- **6.10** ✅ Compliance posture (Phase 6 capstone) — new `rag-compliance` package: `RetentionEnforcer` drives tenant-scoped `purge_*` (data retention + GDPR right-to-erasure) over the Feedback/Provenance stores (non-abstract `purge_before`/`purge_tenant` SPI capability with `dry_run` so a preview counts-without-deleting; audit chain never purged in place — its retention is the 6.6b WORM export); `POST /v1/compliance/erase` (tenant-scoped, dry-run default + two-flag delete); per-tenant `data_region` enforced at ingest (`ResidencyViolationError` → 403); `GET /v1/status/compliance` posture + `compliance_posture`/`residency_ok`; PII-free `compliance.*` events; `cfg.compliance` + `tenants[].data_region`/`retention_days`; `ragctl compliance report`/`demo`; SOC 2 + GDPR control-mapping docs backed by the live posture — [#162](https://github.com/officialCodeWork/AgentContextOS/pull/162)
- **6.9** ✅ Air-gapped install bundle — `ragctl airgap build`/`inspect`/`verify`/`install`: one signed `.tar.gz` of all runtime images (`docker save`) + the packaged Helm chart + `rag.yaml` + a standalone installer, for networks with no registry / internet. Integrity = a standard `SHA256SUMS` (verifiable with `sha256sum -c`, no cosign/network) pinned as `manifest.content_hash` + an optional cosign signature over it (the 6.6b content-hash + optional-signature pattern); the same `SHA256SUMS` drives the Python verifier and the standalone `install.{sh,ps1}` (need only docker+helm). Typed/tested logic in `ragctl.airgap` (pure core + stubbable docker/helm/cosign seam; `--dry-run` = verifiable bundle minus blobs); digest-pinned manifest-driven image set (`infra/airgap/images.txt`); `task airgap:*`; `release-airgap.yml` on tags — [#161](https://github.com/officialCodeWork/AgentContextOS/pull/161)
- **6.8** ✅ SSO / SCIM — `rag-sso`: `FederatedAuth` (an `Auth` SPI backend over per-tenant `OidcProvider` / `SamlProvider`) federates a bearer token / SAML assertion → `Principal` at the existing `authenticate` seam (group claims → `acl_labels`); dependency-free defaults (stdlib HS256 JWT + `defusedxml` SAML), asymmetric OIDC / XML-DSig behind `[oidc]` / `[saml]` extras; algorithm-allowlist downgrade defense; SCIM 2.0 `ScimService` over the new tenant-scoped `ScimStore` SPI driving `/scim/v2/{Users,Groups}` (per-tenant bearer token); per-tenant IdP on `tenants[].sso`; `cfg.sso` / `cfg.scim`; `GET /v1/status/sso`; PII-free `sso.*` / `scim.*` events; `ragctl sso` / `scim` — [#159](https://github.com/officialCodeWork/AgentContextOS/pull/159)
@@ -65,8 +67,8 @@
| 4 | Reliability | 6 | **6** | 0 |
| 5 | Eval & Observability | 7 | **7** | 0 |
| 6 | Governance & Tenancy | 10 | **10** | 0 |
-| 7 | Pilot, Harden, GA | 10 | 0 | 10 |
-| **Total** | | **84** | **74** | **10** |
+| 7 | Pilot, Harden, GA | 10 | **1** | 9 |
+| **Total** | | **84** | **75** | **9** |
---
@@ -797,7 +799,7 @@ New ground — the only prior crypto was HMAC signing. The V1 plan calls for en
| Step | Title | Status | Planned deliverables |
|------|-------|:------:|----------------------|
-| 7.1 | Load testing | ⏳ | Locust suite; 1000 QPS sustained; p99 < 500 ms; chaos under load |
+| 7.1 | Load testing | ✅ | [#164](https://github.com/officialCodeWork/AgentContextOS/pull/164) — chaos-under-load CI gate (`eval/gateway_chaos_v0` fault backends + harness; `tests/perf/test_chaos_under_load.py`; graceful degradation: no 5xx + breaker opens); Locust v1 suite (varied-query mix + ramp shape); acceptance targets + runbook |
| 7.2 | Chaos engineering | ⏳ | Chaos Monkey / LitmusChaos; kill each backend; verify the fallback chain holds |
| 7.3 | Red-team | ⏳ | Prompt injection, ACL bypass, PII egress, tenant-escape probes |
| 7.4 | Design partner onboarding | ⏳ | 2–3 design partners; feedback incorporated; case study documented |
@@ -808,6 +810,15 @@ New ground — the only prior crypto was HMAC signing. The V1 plan calls for en
| 7.9 | Billing integration | ⏳ | Stripe metered billing; usage export API; invoice generation |
| 7.10 | GA cutover | ⏳ | `main` tag `v1.0.0`; release notes; all Phase 7 exit gates passed |
+### 7.1 — Load + chaos testing ✅ [#164](https://github.com/officialCodeWork/AgentContextOS/pull/164)
+
+- Builds on Step 4.6 (ADR-0025: the in-process p99-overhead gate + Locust v0). 7.1 adds the **load + resilience** side the GA bar needs, split into a deterministic CI gate + a cluster runbook
+- **Chaos-under-load is a deterministic CI gate.** New `eval/gateway_chaos_v0/` drives the in-process gateway (httpx ASGITransport, no socket / no cluster) under concurrent load while **injecting backend faults**, and asserts the gateway **degrades gracefully** — no 5xx, 100% success, the failing backend's **circuit breaker opens**. The asserted property is *resilience*, not throughput, so it is **timing-independent** and runs in the `perf` CI job (`tests/perf/test_chaos_under_load.py`); `task chaos-test` runs the harness `--check`
+- **Reuses the Phase-4 machinery — it *validates* it.** `FaultSpec` (seeded failure probability + injected latency) + **`Chaos{Vector,Keyword,Graph}RetrievalBackend`** are SPI wrappers exactly like the rag-breaker wrappers (allowlisted in the policy-coverage linter for the same reason: they forward the already-policy-merged `retrieve_ids` / `expand`), wired behind real breakers + the fallback ladder. Under 100% vector failure the vector breaker trips and the fallback fuses the survivors; under all-backends-fail it degrades to graceful empty answers — the client never sees a 5xx
+- **The query is varied per request** (a unique nonce) in both the chaos harness and the Locust suite — identical queries hit the retrieval cache and short-circuit before retrieval, so a constant-query load test would measure the cache, not the pipeline (and the chaos backends would never be exercised)
+- **The 1000-QPS / p99 < 500 ms acceptance is a cluster runbook, not a CI gate** (hardware/backends-bound — same reasoning as ADR-0025). Extended **Locust v1 suite** (`eval/gateway_load_v0/locustfile.py`): a weighted read/write mix (`/v1/retrieve` · `/v1/query` · `/v1/feedback` · `/v1/status/metrics` · `/healthz`) + a `LoadTestShape` that ramps → holds a plateau → ramps down (env-tunable peak / hold for distributed workers). Acceptance targets + the runbook in [`guides/load-testing.md`](docs/guides/load-testing.md)
+- **Scope:** in-process chaos gate (resilience) + the Locust suite + runbook for the throughput acceptance. **Deferred:** distributed-Locust-in-CI against an ephemeral cluster, latency-based breaker tripping (timeouts), storage/LLM-layer fault injection, soak/endurance tests (cluster-level chaos with LitmusChaos is Step 7.2). ~4 chaos perf tests; all gates green (ruff, mypy --strict, RAG001, policy-coverage, schema/openapi-drift, log-schema). [ADR-0043](docs/adr/ADR-0043-load-chaos-testing.md), [guides/load-testing.md](docs/guides/load-testing.md), [reference/perf.md](docs/reference/perf.md)
+
---
## PR & Branch History
diff --git a/Taskfile.yml b/Taskfile.yml
index 67a1775..ada11f2 100644
--- a/Taskfile.yml
+++ b/Taskfile.yml
@@ -115,6 +115,11 @@ tasks:
cmds:
- "{{.PYTHON}} -m eval.gateway_load_v0.harness"
+ chaos-test:
+ desc: "Run the chaos-under-load harness — inject backend faults, assert graceful degradation"
+ cmds:
+ - "{{.PYTHON}} -m eval.gateway_chaos_v0.harness --check"
+
# ---------------------------------------------------------------------------
# Schemas
# env: sets PYTHONPATH cross-platform (Task handles Windows vs Unix syntax)
diff --git a/docs/README.md b/docs/README.md
index c15157d..954d490 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -125,6 +125,7 @@
| [logging-policy.md](guides/logging-policy.md) | RAG001 policy: structured-logging requirement, allowlist, and how to extend it |
| [ragctl-quickstart.md](guides/ragctl-quickstart.md) | Five-minute tour of the `ragctl` CLI |
| [airgap-install.md](guides/airgap-install.md) | Operator runbook (Step 6.9): build + sign an offline bundle on a connected host (`ragctl airgap build --sign`), transfer it, then verify + install on the air-gapped target with the standalone `install.sh`/`install.ps1` (`sha256sum -c SHA256SUMS` → cosign → `docker load` → `helm upgrade --install`); trimming the image set; pointing the chart at in-network backends; offline-verifying a bundle with stdlib tools |
+| [load-testing.md](guides/load-testing.md) | Load + chaos testing runbook (Step 7.1): the two layers — in-process CI gates (p99 ≤ 30 ms overhead + **chaos-under-load** graceful degradation) vs the Locust suite on a cluster (≥ 1000 RPS sustained, e2e p99 < 500 ms); `task chaos-test` / `load-test` / `perf`; running Locust at scale (distributed workers, `LOAD_PEAK_USERS` / `LOAD_HOLD_S` env knobs, the ramp shape); the acceptance-target table; securing the run with an auth header |
| [curl-quickstart.md](guides/curl-quickstart.md) | 🥈 Curl-able RAG (Step 3.1): 5-minute walkthrough from `curl` to gateway response, including ingest, query, generate, OpenAPI |
| [grpcurl-quickstart.md](guides/grpcurl-quickstart.md) | gRPC quickstart (Step 3.2): 5-minute walkthrough using `grpcurl` against the in-process server — health check, list corpora, server-streaming query, structured errors |
| [mcp-quickstart.md](guides/mcp-quickstart.md) | MCP quickstart (Step 3.3): 5-minute walkthrough — `ragctl mcp-query`, running the stdio server, mounting `@ragplatform/mcp` in Claude Desktop, the three tools, error shape |
@@ -186,6 +187,7 @@ broken, and what to fix before committing to the next phase.
| [ADR-0034-physical-multi-tenancy.md](adr/ADR-0034-physical-multi-tenancy.md) | Decision (Step 6.2): a *dedicated* vector index/collection per tenant. `TenantConfig.dedicated_index` resolves to a `physical_index` key on `TenantSettings`, threaded onto `RequestContext.physical_index`; backends read only `ctx` (graph is backends→core, never rag-config) and namespace their base under it (`-`), lazily creating it; one instance + per-tenant derivation (no per-tenant instances, no SPI change); Noop is the CI conformance oracle (keyed by `physical_index`) for a cross-tenant probe gate that proves isolation independent of the tenant filter; Noop + Pinecone + Qdrant this step, others later |
| [ADR-0035-acl-pushdown.md](adr/ADR-0035-acl-pushdown.md) | Decision (Step 6.3): label-based ACL push-down at retrieval. `AclPolicyEngine` (a decorator like `QuotaPolicyEngine`) And-merges `any_in("acl_labels", principal.acl_labels)` into every `read_chunk` push-down at the canonical `HybridRetriever` PDP site — overlap semantics via the existing `AnyIn` predicate (zero backend/translator changes), **fail-closed** (label-less principal matches nothing; "public" = a shared label), **opt-in** via `cfg.acl.enabled`; emits `acl.egress_denied` on a request-level denial; graph edge ACLs + post-retrieval re-verification (6.4) deferred |
| [ADR-0036-acl-egress-verifier.md](adr/ADR-0036-acl-egress-verifier.md) | Decision (Step 6.4): a post-retrieval ACL re-check as a **defense-in-depth second layer** behind the 6.3 push-down. `AclEgressVerifier.verify(ctx, refs)` drops any returned `ChunkRef` whose labels don't overlap the principal's — same overlap semantics (no-op on correct results), reading `ChunkRef.acl_labels` (no re-hydration), **independent of the PDP** (consults only `ctx.principal.acl_labels`) so a push-down bug/bypass can't disable both; wired at the gateway as a `SupportsRoute` wrapper around `app.state.retrieval_router` (covers query/retrieve/corpus/OpenAI/agent); `cfg.acl.verify_egress` default on but gated by `enabled`; emits `acl.egress_violation` on a caught leak; a red-team gate proves a zero escaped-violation rate when the push-down is bypassed; backend-mislabel re-hydration + per-tenant violation metrics deferred |
+| [ADR-0043-load-chaos-testing.md](adr/ADR-0043-load-chaos-testing.md) | Decision (Step 7.1, Phase-7 hardening): split load testing into a deterministic CI gate + a cluster runbook. **Chaos-under-load is a CI gate** — drive the in-process gateway under concurrent load while injecting backend faults (`FaultSpec` + `Chaos{Vector,Keyword,Graph}RetrievalBackend`, SPI wrappers like the breaker wrappers, allowlisted in policy-coverage) and assert **graceful degradation**: no 5xx, 100% success, the relevant breaker opens. The asserted property is *resilience*, not throughput, so it's timing-independent + deterministic; it **reuses the Phase-4 breakers + fallback** (builds no new resilience — it validates them). **The 1000-QPS / p99<500ms acceptance is a cluster runbook, not CI** (hardware/backends-bound, same reasoning as ADR-0025) — shipped as a Locust v1 suite (weighted read/write mix + a `LoadTestShape` ramp + **varied queries** so retrieval is exercised, not the cache) + documented targets. Deferred: distributed-Locust-in-CI against an ephemeral cluster, latency-based breaker tripping, storage/LLM fault injection, soak tests; rejected asserting raw throughput in CI, killing a backend process (no separate process in-process — inject at the SPI boundary) |
| [ADR-0042-compliance-posture.md](adr/ADR-0042-compliance-posture.md) | Decision (Step 6.10, Phase-6 capstone): add the three compliance pieces on top of the controls the platform already ships (audit/ACL/PII/BYOK/SSO/quotas). New `rag-compliance` package (config-free, like rag-feedback/rag-drift): `RetentionEnforcer` drives tenant-scoped `purge_*`; `compliance_posture`/`residency_ok` are pure. **Retention is a capability on the existing stores, not a new SPI** — non-abstract `purge_before`/`purge_tenant` (default no-op) on Feedback/Provenance, with `dry_run` in the SPI so a preview counts-without-deleting uniformly (ProvenanceStore has no `list`). **Audit is never purged in place** (the hash chain would break) — audit retention is the 6.6b WORM export; `audit_days` is advisory. **Right-to-erasure is always-on, tenant-self-service, two-flag** — `POST /v1/compliance/erase` erases the *calling* tenant's data (scope from the principal, never the body), dry-run by default, delete needs `dry_run=false` AND `confirm=true`. **Residency = declared per tenant + enforced at ingest** (`tenants[].data_region` vs `cfg.compliance.region` → 403), opt-in, a single-deployment assertion not multi-region routing. **The SOC 2 / GDPR mapping is a doc backed by a live posture** (`GET /v1/status/compliance` reports which controls are on, so the mapping is checkable). Deferred: subject-level (vs tenant-level) erasure, an admin retention-sweep endpoint, multi-region routing, automated audit-evidence bundles; rejected purging the audit chain, a `Purgeable` SPI mixin, a static doc with no live backing |
| [ADR-0041-airgap-bundle.md](adr/ADR-0041-airgap-bundle.md) | Decision (Step 6.9): ship the platform as one signed, self-contained offline bundle (all runtime images + Helm chart + config + installer). Integrity reuses the WORM-export pattern (6.6b): a standard `SHA256SUMS` whose hash is pinned as `manifest.content_hash` is the **hard gate** — verifiable with nothing but `sha256sum`, no network/cosign — and a cosign signature **over `SHA256SUMS`** adds authenticity; the *same* `SHA256SUMS` drives the Python verifier and the standalone shell installer so they can't diverge. The shell/pwsh `install.{sh,ps1}` (shipped inside the bundle) need only docker+helm (air-gap hosts lack uv/the workspace); `ragctl airgap` holds the typed/tested build+verify logic (pure core separated from a stubbable docker/helm/cosign subprocess seam; `--dry-run` = a verifiable bundle minus image blobs, so the path is testable with no Docker). Digest-pinned manifest-driven image set (`infra/airgap/images.txt` + the chart-derived gateway image); key-based cosign is the air-gap recommendation (keyless needs Rekor + an identity policy), keyless is the connected-release path (`release-airgap.yml` on tags). Deferred: ctr/podman load, registry re-tag/push, multi-arch selection, bundling backend charts, TUF-rooted offline keyless verify; rejected `oras`/OCI (no registry to pull from in an air-gap), a second HMAC scheme (cosign already the signer), a pure-shell build (would escape mypy/tests) |
| [ADR-0040-sso-scim.md](adr/ADR-0040-sso-scim.md) | Decision (Step 6.8): enterprise identity in two surfaces. **Federation** — `FederatedAuth` *is* an `Auth` SPI backend (the `authenticate(token, tenant_id) → Principal` seam already runs at the boundary, so wiring it is the whole integration — no middleware change); group claims → `acl_labels` so Step 6.3 push-down + 6.5 PII egress govern federated users unchanged (`authorize` stays a coarse allow — federation establishes *who*, the PDP decides *what*). Dependency-free defaults (stdlib HS256 JWT with full `exp`/`nbf`/`iss`/`aud` + constant-time compare; `defusedxml` SAML validating Issuer/Conditions/Audience) with asymmetric OIDC (PyJWT, `[oidc]`) + SAML XML-DSig (signxml, `[saml]`, injected verifier → fail-closed) behind extras; **algorithm-allowlist** designs out `alg:none`/RS↔HS confusion. Per-tenant IdP on `tenants[].sso` (reuses Step 6.1 config; no provider → bearer rejected, header-identity still works). **Provisioning** — SCIM 2.0 is a separate surface with its own per-tenant bearer token (`cfg.scim.tokens`, not a user JWT), a tenant-scoped `ScimStore` SPI (`NoopScimStore`) + `ScimService`, SCIM-shaped errors, disabled→404; no new governed SPI call (linter passes). PII-free `sso.*`/`scim.*` events (hashed subject, never email/userName). Deferred: JWKS rotation, SP-initiated SAML + metadata, SCIM bulk/`/Me`/ETag, directory-backed deprovisioning, admin-console card; rejected Authlib/python3-saml (heavy lxml/xmlsec on the default install), a dedicated SSO middleware, SCIM token on `TenantConfig` |
diff --git a/docs/adr/ADR-0043-load-chaos-testing.md b/docs/adr/ADR-0043-load-chaos-testing.md
new file mode 100644
index 0000000..f68aa3c
--- /dev/null
+++ b/docs/adr/ADR-0043-load-chaos-testing.md
@@ -0,0 +1,72 @@
+# ADR-0043 — Load + chaos testing for GA
+
+**Status:** Accepted
+**Date:** 2026-06-08
+**Step:** 7.1 — Load testing (Phase 7 — Pilot, Harden, GA)
+**Related:** [ADR-0025](ADR-0025-latency-load-testing.md) (latency gate + Locust v0), [ADR-0016?](#) (circuit breakers / fallback — Phase 4), [guides/load-testing.md](../guides/load-testing.md), [reference/perf.md](../reference/perf.md)
+
+## Context
+
+Step 4.6 (ADR-0025) delivered the **latency** side: an in-process p99-overhead gate
+(≤ 30 ms, deterministic in CI) + a Locust smoke for over-the-wire runs. Step 7.1 —
+the first hardening step — adds the **load + resilience** side the GA bar needs:
+*1000 QPS sustained, p99 < 500 ms, chaos under load*. Two of those are throughput
+targets that need a real cluster; the third — **resilience under failure** — is the
+genuinely new, CI-gateable property.
+
+## Decision
+
+**1. Chaos-under-load is a deterministic, in-process CI gate.** Mirroring the
+latency gate: drive the gateway in-process (httpx ASGITransport, no socket / no
+cluster) under concurrent load while **injecting backend faults**, and assert the
+gateway *degrades gracefully* — no 5xx, 100% success, and the relevant **circuit
+breaker opens**. The asserted property is **resilience**, not raw throughput, so it
+is deterministic and runs in the `perf` CI job on all the noisy timing aside (the
+assertions are timing-independent).
+
+**2. Reuse the Phase-4 machinery — the chaos test *validates* it.** The breakers
+(4.4) + fallback ladder (4.2) already provide graceful degradation; 7.1 builds no
+new resilience. The chaos harness injects faults into chaos-wrapped backends
+(`FaultSpec` + `Chaos{Vector,Keyword,Graph}RetrievalBackend` — SPI wrappers exactly
+like the breaker wrappers, allowlisted in the policy-coverage linter for the same
+reason) behind real breakers, and proves: under 100% backend failure the breaker
+trips, the fallback fuses the survivors / returns a graceful empty answer, and the
+client never sees a 5xx.
+
+**3. The 1000-QPS / p99 acceptance is a cluster runbook, not a CI gate.** Real
+sustained throughput depends on hardware, backends, and network — it cannot be a
+deterministic CI gate (ADR-0025's reasoning). So 7.1 ships a **Locust v1 suite**
+(weighted read/write mix with *varied queries* so the retrieval path is exercised,
+not just the cache; a `LoadTestShape` that ramps to a sustained plateau) plus
+**documented acceptance targets** + a [runbook](../guides/load-testing.md) for
+validating them against a deployed cluster. The in-process harness gives the
+CI-checkable smoke; the cluster run gives the GA number.
+
+**4. Vary the load query.** Both the chaos harness and the Locust suite append a
+unique nonce per request — identical queries hit the retrieval cache and
+short-circuit before retrieval, so a constant-query load test would measure the
+cache, not the pipeline (and the chaos backends would never be exercised).
+
+## Consequences
+
+- New `eval/gateway_chaos_v0/` (fault backends + harness + sample report);
+ `tests/perf/test_chaos_under_load.py` (the graceful-degradation gate); an extended
+ `eval/gateway_load_v0/locustfile.py` (v1 suite + ramp shape); `task chaos-test`;
+ the load-testing runbook + acceptance targets.
+- The GA latency/throughput SLOs (≤ 250 ms SaaS / < 500 ms dev p99, ≥ 1000 RPS) are
+ documented + Locust-runnable; resilience is CI-gated.
+- **Deferred:** a distributed-Locust harness in CI against an ephemeral cluster
+ (k6/Locust-operator), latency-based breaker tripping (timeouts, not just errors),
+ fault injection at the *storage* / LLM layer, and soak/endurance tests.
+
+## Alternatives considered
+
+- **Assert raw throughput (1000 QPS) in CI.** Rejected (same as ADR-0025) — it's
+ hardware/backends-bound and would be flaky; CI asserts the *resilience* invariant
+ and the deterministic p99-overhead, and the throughput is a cluster runbook.
+- **A real cluster + distributed Locust in CI.** Too heavy / slow / flaky for the PR
+ gate; left as a deferred ephemeral-environment job. The in-process chaos gate
+ catches resilience regressions deterministically today.
+- **Kill a backend process for chaos.** The in-process design has no separate
+ process to kill; injecting `BackendError` / latency at the SPI boundary exercises
+ the exact same breaker + fallback paths, deterministically.
diff --git a/docs/guides/load-testing.md b/docs/guides/load-testing.md
new file mode 100644
index 0000000..6bc0835
--- /dev/null
+++ b/docs/guides/load-testing.md
@@ -0,0 +1,94 @@
+# Guide: load + chaos testing (Step 7.1)
+
+How to validate the GA performance bar — **1000 QPS sustained, p99 < 500 ms,
+graceful under failure** — using the in-process gates (CI) and the Locust suite +
+chaos harness (cluster).
+
+## The two layers
+
+| Layer | What it proves | Where it runs |
+|-------|----------------|---------------|
+| **In-process gates** (CI) | gateway overhead p99 ≤ 30 ms; **graceful degradation under chaos** | `tests/perf/` — deterministic, no infra |
+| **Locust suite** (cluster) | sustained throughput ≥ 1000 RPS; end-to-end p99 < 500 ms | a deployed gateway + load generator |
+
+The CI gates catch *regressions* deterministically; the cluster run produces the
+*GA acceptance number* (which is hardware/backends-bound and can't be a CI gate —
+see [ADR-0043](../adr/ADR-0043-load-chaos-testing.md)).
+
+## 1. Chaos under load (CI-gated, run anywhere)
+
+```bash
+task chaos-test # ragctl-free: python -m eval.gateway_chaos_v0.harness --check
+# or, targeted:
+uv run python -m eval.gateway_chaos_v0.harness --vector-failure-rate 1.0
+uv run python -m eval.gateway_chaos_v0.harness --all-fail
+uv run python -m eval.gateway_chaos_v0.harness --latency-ms 5
+```
+
+It drives the in-process gateway under concurrent load while injecting backend
+faults, and asserts **graceful degradation**: the circuit breaker opens on the
+failing backend, the fallback ladder keeps serving, and **no request 5xx-es**.
+`pytest -m perf tests/perf/test_chaos_under_load.py` runs it as the gate.
+
+The report (`eval/gateway_chaos_v0/report.md`) shows, per backend: calls, injected
+failures, and the resulting breaker state — e.g. 100% vector failure → the vector
+breaker is `open` while success stays at 100%.
+
+## 2. In-process load + latency profile (CI-gated)
+
+```bash
+task load-test # writes eval/gateway_load_v0/report.*
+uv run python -m eval.gateway_load_v0.harness --check # exit 1 if over the overhead budget
+task perf # the p99 ≤ 30 ms overhead gate
+```
+
+This measures the gateway's *own* per-request overhead (noop backends), the
+deterministic signal. End-to-end latency under real backends is the Locust job.
+
+## 3. Locust at scale → the 1000-QPS acceptance (cluster)
+
+Locust is an operator tool, not a project dependency:
+
+```bash
+uv pip install locust
+# Start a gateway (or point --host at your deployment):
+uv run uvicorn rag_gateway.app:app --port 8000
+# Drive load with the built-in ramp shape:
+locust -f eval/gateway_load_v0/locustfile.py --host http://localhost:8000 --headless
+```
+
+The suite runs a weighted mix (`/v1/retrieve` 5 · `/v1/query` 4 · `/v1/feedback` 1
+· `/v1/status/metrics` 1 · `/healthz` 1) with **varied queries** (a per-request
+nonce so the retrieval path is exercised rather than served from cache), and a
+`LoadTestShape` that ramps → holds a plateau → ramps down.
+
+**Reaching 1000 RPS.** One box rarely sustains 1000 RPS through a full pipeline.
+Scale via env + distributed workers:
+
+```bash
+LOAD_PEAK_USERS=800 LOAD_HOLD_S=600 \
+ locust -f eval/gateway_load_v0/locustfile.py --host https://gw.internal \
+ --headless --master # + N --worker processes
+```
+
+Env knobs: `LOAD_PEAK_USERS`, `LOAD_SPAWN_RATE`, `LOAD_RAMP_S`, `LOAD_HOLD_S`.
+
+### Acceptance targets
+
+| Metric | Target | Notes |
+|--------|--------|-------|
+| Sustained throughput | **≥ 1000 RPS** | on the plateau, error rate ~0% |
+| End-to-end p99 | **< 500 ms** (dev SLO; ≤ 250 ms SaaS) | dominated by backend latency |
+| Error rate under steady load | ~0% | 5xx ≈ 0 |
+| Under chaos (a backend down) | **no 5xx**, breaker opens | the chaos gate |
+
+Read these off the Locust UI / `--csv` output on the plateau (ignore ramp-up).
+
+## Securing the run
+
+The default gateway is creds-free (anonymous dev tenant). For a secured deployment,
+add an `Authorization` header in the Locust user's `on_start` and point `--host` at
+the real gateway.
+
+See [reference/perf.md](../reference/perf.md) and
+[ADR-0043](../adr/ADR-0043-load-chaos-testing.md).
diff --git a/docs/reference/perf.md b/docs/reference/perf.md
index c2ee91f..5832254 100644
--- a/docs/reference/perf.md
+++ b/docs/reference/perf.md
@@ -133,3 +133,29 @@ The overhead number's source of truth is `rag_gateway.perf.DEFAULT_GATEWAY_OVERH
so retrieval returns chunks and rerank/pack run (the default empty corpus short-circuits them).
* **Real-backend budgets:** a backend implementation consults `tests/contract/budgets.py`
(`spi_budget(spi, method)`) from its own perf harness.
+
+## Chaos under load (Step 7.1)
+
+`eval/gateway_chaos_v0/` adds the **resilience** gate alongside the latency one: it
+drives the in-process gateway under concurrent load while injecting backend faults
+and asserts **graceful degradation** — the circuit breaker opens on the failing
+backend, the fallback ladder keeps serving, and no request 5xx-es.
+
+```bash
+task chaos-test # python -m eval.gateway_chaos_v0.harness --check
+uv run python -m eval.gateway_chaos_v0.harness --all-fail # worst case → graceful empty answers
+pytest -m perf tests/perf/test_chaos_under_load.py # the gate
+```
+
+- `FaultSpec` (failure probability + injected latency, seeded) + `Chaos{Vector,
+ Keyword,Graph}RetrievalBackend` — SPI wrappers (like the breaker wrappers) that
+ inject faults on the read path, wired behind real breakers in the harness.
+- The asserted property is **timing-independent** (no 5xx, 100% success, breaker
+ opens), so it's a deterministic CI gate; the p99 budget is the harness `--check`.
+- The harness reuses the Phase-4 breakers + fallback — it *validates* them, it
+ builds no new resilience.
+
+The **1000-QPS / p99 < 500 ms** acceptance is a *cluster* run (the Locust v1 suite
+in `eval/gateway_load_v0/locustfile.py`), documented in
+[guides/load-testing.md](../guides/load-testing.md) — not a CI gate. See
+[ADR-0043](../adr/ADR-0043-load-chaos-testing.md).
diff --git a/eval/gateway_chaos_v0/__init__.py b/eval/gateway_chaos_v0/__init__.py
new file mode 100644
index 0000000..9411990
--- /dev/null
+++ b/eval/gateway_chaos_v0/__init__.py
@@ -0,0 +1,8 @@
+"""Step 7.1 — chaos-under-load testing for the gateway.
+
+Drives the in-process gateway under concurrent load while injecting backend
+faults (failures + latency), and asserts the Phase-4 resilience machinery
+(circuit breakers + fallback ladder) keeps the gateway serving: graceful
+degradation instead of a 5xx storm. The deterministic, dependency-free
+counterpart to a chaos run against a deployed cluster.
+"""
diff --git a/eval/gateway_chaos_v0/faults.py b/eval/gateway_chaos_v0/faults.py
new file mode 100644
index 0000000..556b577
--- /dev/null
+++ b/eval/gateway_chaos_v0/faults.py
@@ -0,0 +1,152 @@
+"""Fault-injecting retrieval backends for chaos-under-load testing (Step 7.1).
+
+Each wrapper *is-a* retrieval-backend SPI (like the Step 4.4 breaker wrappers), so
+it drops into a ``HybridRetriever`` wherever the real backend goes. It injects a
+configurable **fault profile** on the hot retrieval call — a failure probability
+(raises :class:`~rag_core.errors.BackendError`) and/or added latency — so a load
+run can verify the gateway degrades gracefully (the breaker opens, the fallback
+ladder fuses the survivors / returns a graceful empty answer) instead of 5xx-ing.
+
+Faults are seeded so a run is reproducible; the per-call counters let the harness
+assert how many calls actually failed.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import random
+from typing import Any, Literal
+
+from rag_core.errors import BackendError
+from rag_core.filter import FilterExpr
+from rag_core.spi.graph_store import GraphRetrievalBackend
+from rag_core.spi.keyword_store import KeywordRetrievalBackend
+from rag_core.spi.vector_store import VectorRetrievalBackend
+from rag_core.types import Chunk, ChunkRef, CorpusId, NeighborResult, RequestContext
+
+__all__ = [
+ "FaultSpec",
+ "ChaosVectorRetrievalBackend",
+ "ChaosKeywordRetrievalBackend",
+ "ChaosGraphRetrievalBackend",
+]
+
+
+class FaultSpec:
+ """A backend fault profile: a per-call failure probability + injected latency.
+
+ ``failure_rate`` is the probability a retrieval call raises ``BackendError``;
+ ``latency_ms`` is added (via ``asyncio.sleep``) to every retrieval call. The
+ RNG is seeded so a run is reproducible. ``calls`` / ``failures`` count what
+ actually happened, so the harness + tests can assert the breaker tripped.
+ """
+
+ def __init__(
+ self, *, failure_rate: float = 0.0, latency_ms: float = 0.0, seed: int = 1234
+ ) -> None:
+ if not 0.0 <= failure_rate <= 1.0:
+ raise ValueError("failure_rate must be in [0, 1]")
+ self.failure_rate = failure_rate
+ self.latency_ms = latency_ms
+ self._rng = random.Random(seed) # noqa: S311 - chaos fault injection, not crypto
+ self.calls = 0
+ self.failures = 0
+
+ async def inject(self, label: str) -> None:
+ """Apply the fault for one retrieval call (sleep, then maybe raise)."""
+ self.calls += 1
+ if self.latency_ms > 0:
+ await asyncio.sleep(self.latency_ms / 1000.0)
+ if self.failure_rate > 0 and self._rng.random() < self.failure_rate:
+ self.failures += 1
+ raise BackendError(f"chaos: injected {label} backend failure", backend=label)
+
+
+class ChaosVectorRetrievalBackend(VectorRetrievalBackend):
+ """A vector backend whose ``retrieve_ids`` is fault-injected (Step 7.1)."""
+
+ def __init__(self, inner: VectorRetrievalBackend, fault: FaultSpec) -> None:
+ self._inner = inner
+ self._fault = fault
+
+ async def retrieve_ids(
+ self,
+ ctx: RequestContext,
+ vector: list[float],
+ top_k: int,
+ corpus_ids: list[CorpusId],
+ filters: FilterExpr | None = None,
+ ) -> list[ChunkRef]:
+ await self._fault.inject("vector")
+ return await self._inner.retrieve_ids(ctx, vector, top_k, corpus_ids, filters)
+
+ async def hydrate(self, ctx: RequestContext, chunk_refs: list[ChunkRef]) -> list[ChunkRef]:
+ return await self._inner.hydrate(ctx, chunk_refs)
+
+ async def health(self) -> bool:
+ return await self._inner.health()
+
+
+class ChaosKeywordRetrievalBackend(KeywordRetrievalBackend):
+ """A keyword backend whose ``retrieve_ids`` is fault-injected (Step 7.1)."""
+
+ def __init__(self, inner: KeywordRetrievalBackend, fault: FaultSpec) -> None:
+ self._inner = inner
+ self._fault = fault
+
+ async def retrieve_ids(
+ self,
+ ctx: RequestContext,
+ query_text: str,
+ top_k: int,
+ corpus_ids: list[CorpusId],
+ filters: FilterExpr | None = None,
+ ) -> list[ChunkRef]:
+ await self._fault.inject("keyword")
+ return await self._inner.retrieve_ids(ctx, query_text, top_k, corpus_ids, filters)
+
+ async def hydrate(self, ctx: RequestContext, chunk_refs: list[ChunkRef]) -> list[Chunk]:
+ return await self._inner.hydrate(ctx, chunk_refs)
+
+ async def health(self) -> bool:
+ return await self._inner.health()
+
+
+class ChaosGraphRetrievalBackend(GraphRetrievalBackend):
+ """A graph backend whose ``expand`` is fault-injected (Step 7.1)."""
+
+ def __init__(self, inner: GraphRetrievalBackend, fault: FaultSpec) -> None:
+ self._inner = inner
+ self._fault = fault
+
+ async def query(
+ self, ctx: RequestContext, statement: str, parameters: dict[str, Any]
+ ) -> list[dict[str, Any]]:
+ return await self._inner.query(ctx, statement, parameters)
+
+ async def expand(
+ self,
+ ctx: RequestContext,
+ seed_ids: list[str],
+ *,
+ hops: int = 1,
+ rel_types: list[str] | None = None,
+ direction: Literal["out", "in", "both"] = "out",
+ node_filter: FilterExpr | None = None,
+ edge_filter: FilterExpr | None = None,
+ limit: int | None = None,
+ ) -> list[NeighborResult]:
+ await self._fault.inject("graph")
+ return await self._inner.expand(
+ ctx,
+ seed_ids,
+ hops=hops,
+ rel_types=rel_types,
+ direction=direction,
+ node_filter=node_filter,
+ edge_filter=edge_filter,
+ limit=limit,
+ )
+
+ async def health(self) -> bool:
+ return await self._inner.health()
diff --git a/eval/gateway_chaos_v0/harness.py b/eval/gateway_chaos_v0/harness.py
new file mode 100644
index 0000000..723e46a
--- /dev/null
+++ b/eval/gateway_chaos_v0/harness.py
@@ -0,0 +1,340 @@
+"""Step 7.1 — chaos-under-load harness.
+
+Drives the in-process gateway under concurrent load while injecting backend faults,
+and verifies the Phase-4 resilience machinery keeps it serving: the circuit
+breaker opens on the failing backend and the fallback ladder fuses the survivors
+(or returns a graceful empty answer) — so requests stay **2xx** instead of 5xx.
+
+This is the deterministic, dependency-free chaos test (no sockets, no cluster), so
+its verdict is reproducible in CI. For chaos against a *deployed* cluster, see the
+runbook in ``docs/guides/load-testing.md``.
+
+Run::
+
+ uv run python -m eval.gateway_chaos_v0.harness
+ uv run python -m eval.gateway_chaos_v0.harness --vector-failure-rate 1.0
+ uv run python -m eval.gateway_chaos_v0.harness --all-fail --check
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import sys
+import time
+from collections import Counter
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Any
+
+import httpx
+from fastapi import FastAPI
+from rag_breaker import (
+ GRAPH_BACKEND,
+ KEYWORD_BACKEND,
+ VECTOR_BACKEND,
+ BreakerGraphRetrievalBackend,
+ BreakerKeywordRetrievalBackend,
+ BreakerRegistry,
+ BreakerVectorRetrievalBackend,
+)
+from rag_core.spi.noop import NoopEmbedder, NoopGraphStore, NoopKeywordStore, NoopVectorStore
+from rag_gateway.app import build_app
+from rag_retrieval import HybridRetriever, RetrievalRouter
+
+from eval.gateway_chaos_v0.faults import (
+ ChaosGraphRetrievalBackend,
+ ChaosKeywordRetrievalBackend,
+ ChaosVectorRetrievalBackend,
+ FaultSpec,
+)
+
+HERE = Path(__file__).parent
+REPORT_JSON = HERE / "report.json"
+REPORT_MD = HERE / "report.md"
+
+# Acceptance: under injected backend chaos the gateway must *degrade gracefully* —
+# the breaker + fallback keep it serving, so success stays at (or above) this floor
+# and the per-request p99 stays under budget. The noop pipeline is fast; the
+# budget leaves generous headroom for the breaker/fallback bookkeeping under load.
+DEFAULT_SUCCESS_FLOOR = 0.99
+DEFAULT_P99_BUDGET_MS = 150.0
+
+
+def build_chaos_app(
+ *,
+ vector_fault: FaultSpec,
+ keyword_fault: FaultSpec,
+ graph_fault: FaultSpec,
+) -> tuple[FastAPI, BreakerRegistry, dict[str, FaultSpec]]:
+ """Build a gateway whose retrieval backends are chaos-wrapped behind breakers."""
+ registry = BreakerRegistry()
+ vector = BreakerVectorRetrievalBackend(
+ ChaosVectorRetrievalBackend(NoopVectorStore(), vector_fault),
+ registry.get(VECTOR_BACKEND),
+ )
+ keyword = BreakerKeywordRetrievalBackend(
+ ChaosKeywordRetrievalBackend(NoopKeywordStore(), keyword_fault),
+ registry.get(KEYWORD_BACKEND),
+ )
+ graph = BreakerGraphRetrievalBackend(
+ ChaosGraphRetrievalBackend(NoopGraphStore(), graph_fault),
+ registry.get(GRAPH_BACKEND),
+ )
+ hybrid = HybridRetriever(vector_backend=vector, keyword_backend=keyword, graph_backend=graph)
+ router = RetrievalRouter(hybrid=hybrid, embedder=NoopEmbedder(dimension=8))
+ app = build_app(retrieval_router=router, breaker_registry=registry, fallback_enabled=True)
+ faults = {"vector": vector_fault, "keyword": keyword_fault, "graph": graph_fault}
+ return app, registry, faults
+
+
+@dataclass
+class ChaosResult:
+ """Outcome of a chaos-under-load run."""
+
+ requests: int
+ concurrency: int
+ status_counts: dict[str, int]
+ success_rate: float
+ server_errors: int
+ p50_ms: float
+ p95_ms: float
+ p99_ms: float
+ max_ms: float
+ breaker_states: dict[str, str]
+ fault_calls: dict[str, int]
+ fault_failures: dict[str, int]
+ success_floor: float
+ p99_budget_ms: float
+ profile: list[Any] = field(default_factory=list)
+
+ @property
+ def graceful(self) -> bool:
+ """True iff the gateway degraded gracefully: no 5xx, success ≥ floor, p99 ≤ budget."""
+ return (
+ self.server_errors == 0
+ and self.success_rate >= self.success_floor
+ and self.p99_ms <= self.p99_budget_ms
+ )
+
+ def as_dict(self) -> dict[str, Any]:
+ return {
+ "requests": self.requests,
+ "concurrency": self.concurrency,
+ "status_counts": self.status_counts,
+ "success_rate": round(self.success_rate, 4),
+ "server_errors": self.server_errors,
+ "latency_ms": {
+ "p50": round(self.p50_ms, 3),
+ "p95": round(self.p95_ms, 3),
+ "p99": round(self.p99_ms, 3),
+ "max": round(self.max_ms, 3),
+ },
+ "breaker_states": self.breaker_states,
+ "fault_calls": self.fault_calls,
+ "fault_failures": self.fault_failures,
+ "success_floor": self.success_floor,
+ "p99_budget_ms": self.p99_budget_ms,
+ "graceful": self.graceful,
+ }
+
+
+def _percentile(sorted_values: list[float], q: float) -> float:
+ n = len(sorted_values)
+ if n == 0:
+ return 0.0
+ if n == 1:
+ return sorted_values[0]
+ rank = max(1, min(n, int(q * n + 0.999999)))
+ return sorted_values[rank - 1]
+
+
+def _body(idx: int) -> dict[str, Any]:
+ # Vary the query per request: identical queries hit the retrieval cache and
+ # short-circuit before retrieval, so the chaos backends would never be exercised.
+ return {
+ "tenant_id": "chaos-probe",
+ "principal_id": "chaos-probe",
+ "query": f"chaos under load probe {idx}",
+ "rerank": True,
+ "pack": True,
+ "generate": False,
+ }
+
+
+async def _fire(client: httpx.AsyncClient, idx: int) -> tuple[int, float]:
+ """Fire one /v1/query; return (status_code, latency_ms) — never raises on 5xx."""
+ t0 = time.perf_counter()
+ try:
+ resp = await client.post("/v1/query", json=_body(idx))
+ status = resp.status_code
+ except Exception: # noqa: BLE001 - a transport error counts as a server error
+ status = 599
+ return status, (time.perf_counter() - t0) * 1000.0
+
+
+async def _run(
+ app: FastAPI,
+ registry: BreakerRegistry,
+ names: list[str],
+ *,
+ requests: int,
+ concurrency: int,
+ warmup: int,
+) -> tuple[dict[str, int], list[float], dict[str, str]]:
+ transport = httpx.ASGITransport(app=app)
+ statuses: Counter[int] = Counter()
+ latencies: list[float] = []
+ async with httpx.AsyncClient(transport=transport, base_url="http://gateway") as client:
+ for i in range(warmup):
+ await _fire(client, -1 - i)
+ sem = asyncio.Semaphore(concurrency)
+
+ async def _task(idx: int) -> None:
+ async with sem:
+ status, elapsed = await _fire(client, idx)
+ statuses[status] += 1
+ latencies.append(elapsed)
+
+ await asyncio.gather(*[_task(i) for i in range(requests)])
+ states = {name: (await registry.get(name).snapshot()).state.value for name in names}
+ return {str(k): v for k, v in sorted(statuses.items())}, latencies, states
+
+
+def run_chaos(
+ *,
+ requests: int = 600,
+ concurrency: int = 32,
+ warmup: int = 30,
+ vector_failure_rate: float = 1.0,
+ keyword_failure_rate: float = 0.0,
+ graph_failure_rate: float = 0.0,
+ latency_ms: float = 0.0,
+ success_floor: float = DEFAULT_SUCCESS_FLOOR,
+ p99_budget_ms: float = DEFAULT_P99_BUDGET_MS,
+) -> ChaosResult:
+ """Drive the chaos app under load and summarise the resilience outcome."""
+ app, registry, faults = build_chaos_app(
+ vector_fault=FaultSpec(failure_rate=vector_failure_rate, latency_ms=latency_ms, seed=1),
+ keyword_fault=FaultSpec(failure_rate=keyword_failure_rate, latency_ms=latency_ms, seed=2),
+ graph_fault=FaultSpec(failure_rate=graph_failure_rate, latency_ms=latency_ms, seed=3),
+ )
+ status_counts, latencies, breaker_states = asyncio.run(
+ _run(app, registry, list(faults), requests=requests, concurrency=concurrency, warmup=warmup)
+ )
+ latencies.sort()
+ total = sum(status_counts.values())
+ ok = sum(v for k, v in status_counts.items() if int(k) < 400)
+ server_errors = sum(v for k, v in status_counts.items() if int(k) >= 500)
+ return ChaosResult(
+ requests=requests,
+ concurrency=concurrency,
+ status_counts=status_counts,
+ success_rate=(ok / total) if total else 0.0,
+ server_errors=server_errors,
+ p50_ms=_percentile(latencies, 0.50),
+ p95_ms=_percentile(latencies, 0.95),
+ p99_ms=_percentile(latencies, 0.99),
+ max_ms=latencies[-1] if latencies else 0.0,
+ breaker_states=breaker_states,
+ fault_calls={name: f.calls for name, f in faults.items()},
+ fault_failures={name: f.failures for name, f in faults.items()},
+ success_floor=success_floor,
+ p99_budget_ms=p99_budget_ms,
+ )
+
+
+def _write_report_md(path: Path, report: ChaosResult) -> None:
+ verdict = "PASS" if report.graceful else "FAIL"
+ lines = [
+ "# Gateway chaos-under-load — harness report",
+ "",
+ "Generated by `python -m eval.gateway_chaos_v0.harness` against the in-process "
+ "gateway with chaos-wrapped backends behind circuit breakers + the fallback ladder.",
+ "",
+ f"- requests: **{report.requests}** · concurrency: **{report.concurrency}**",
+ f"- success rate: **{report.success_rate:.4f}** (floor {report.success_floor}) "
+ f"· 5xx: **{report.server_errors}**",
+ f"- p99: **{report.p99_ms:.2f} ms** (budget {report.p99_budget_ms:.0f} ms) → **{verdict}**",
+ "",
+ "## Injected faults",
+ "",
+ "| backend | calls | failures | breaker |",
+ "|---------|------:|---------:|---------|",
+ ]
+ for name in report.fault_calls:
+ lines.append(
+ f"| {name} | {report.fault_calls[name]} | {report.fault_failures[name]} | "
+ f"{report.breaker_states[name]} |"
+ )
+ lines += [
+ "",
+ "## Status codes",
+ "",
+ "| status | count |",
+ "|--------|------:|",
+ ]
+ for status, count in report.status_counts.items():
+ lines.append(f"| {status} | {count} |")
+ lines += [
+ "",
+ "**Graceful degradation** = no 5xx, success ≥ floor, p99 ≤ budget — the breaker "
+ "opens on the failing backend and the fallback ladder keeps the gateway serving.",
+ "",
+ ]
+ path.write_text("\n".join(lines), encoding="utf-8")
+
+
+def main(argv: list[str] | None = None) -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--requests", type=int, default=600)
+ parser.add_argument("--concurrency", type=int, default=32)
+ parser.add_argument("--warmup", type=int, default=30)
+ parser.add_argument("--vector-failure-rate", type=float, default=1.0)
+ parser.add_argument("--keyword-failure-rate", type=float, default=0.0)
+ parser.add_argument("--graph-failure-rate", type=float, default=0.0)
+ parser.add_argument("--latency-ms", type=float, default=0.0)
+ parser.add_argument(
+ "--all-fail",
+ action="store_true",
+ help="Fail every retrieval backend (the worst case → graceful empty answers).",
+ )
+ parser.add_argument("--no-write", action="store_true")
+ parser.add_argument(
+ "--check", action="store_true", help="Exit 1 if degradation was not graceful."
+ )
+ args = parser.parse_args(argv)
+
+ if args.all_fail:
+ args.vector_failure_rate = args.keyword_failure_rate = args.graph_failure_rate = 1.0
+
+ report = run_chaos(
+ requests=args.requests,
+ concurrency=args.concurrency,
+ warmup=args.warmup,
+ vector_failure_rate=args.vector_failure_rate,
+ keyword_failure_rate=args.keyword_failure_rate,
+ graph_failure_rate=args.graph_failure_rate,
+ latency_ms=args.latency_ms,
+ )
+
+ if not args.no_write:
+ REPORT_JSON.write_text(json.dumps(report.as_dict(), indent=2) + "\n", encoding="utf-8")
+ _write_report_md(REPORT_MD, report)
+ print(f"wrote {REPORT_JSON}")
+ print(f"wrote {REPORT_MD}")
+
+ verdict = "PASS" if report.graceful else "FAIL"
+ print(
+ f"summary: requests={report.requests} success_rate={report.success_rate:.4f} "
+ f"5xx={report.server_errors} p99={report.p99_ms:.2f}ms breakers={report.breaker_states} "
+ f"-> {verdict}"
+ )
+ if args.check and not report.graceful:
+ return 1
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/eval/gateway_chaos_v0/report.json b/eval/gateway_chaos_v0/report.json
new file mode 100644
index 0000000..299a35d
--- /dev/null
+++ b/eval/gateway_chaos_v0/report.json
@@ -0,0 +1,33 @@
+{
+ "requests": 400,
+ "concurrency": 24,
+ "status_counts": {
+ "200": 400
+ },
+ "success_rate": 1.0,
+ "server_errors": 0,
+ "latency_ms": {
+ "p50": 19.348,
+ "p95": 57.985,
+ "p99": 60.365,
+ "max": 61.204
+ },
+ "breaker_states": {
+ "vector": "open",
+ "keyword": "closed",
+ "graph": "closed"
+ },
+ "fault_calls": {
+ "vector": 5,
+ "keyword": 1290,
+ "graph": 0
+ },
+ "fault_failures": {
+ "vector": 5,
+ "keyword": 0,
+ "graph": 0
+ },
+ "success_floor": 0.99,
+ "p99_budget_ms": 150.0,
+ "graceful": true
+}
diff --git a/eval/gateway_chaos_v0/report.md b/eval/gateway_chaos_v0/report.md
new file mode 100644
index 0000000..c8b03b8
--- /dev/null
+++ b/eval/gateway_chaos_v0/report.md
@@ -0,0 +1,23 @@
+# Gateway chaos-under-load — harness report
+
+Generated by `python -m eval.gateway_chaos_v0.harness` against the in-process gateway with chaos-wrapped backends behind circuit breakers + the fallback ladder.
+
+- requests: **400** · concurrency: **24**
+- success rate: **1.0000** (floor 0.99) · 5xx: **0**
+- p99: **60.36 ms** (budget 150 ms) → **PASS**
+
+## Injected faults
+
+| backend | calls | failures | breaker |
+|---------|------:|---------:|---------|
+| vector | 5 | 5 | open |
+| keyword | 1290 | 0 | closed |
+| graph | 0 | 0 | closed |
+
+## Status codes
+
+| status | count |
+|--------|------:|
+| 200 | 400 |
+
+**Graceful degradation** = no 5xx, success ≥ floor, p99 ≤ budget — the breaker opens on the failing backend and the fallback ladder keeps the gateway serving.
diff --git a/eval/gateway_load_v0/locustfile.py b/eval/gateway_load_v0/locustfile.py
index 736468f..5e7f997 100644
--- a/eval/gateway_load_v0/locustfile.py
+++ b/eval/gateway_load_v0/locustfile.py
@@ -1,9 +1,21 @@
-"""Locust load test for a *running* AgentContextOS gateway (Step 4.6).
+"""Locust load test for a *running* AgentContextOS gateway (Steps 4.6 / 7.1).
Unlike the in-process harness (``eval/gateway_load_v0/harness.py``), this drives
a real gateway over HTTP to measure end-to-end latency + throughput under many
concurrent virtual users — kernel TCP, uvicorn, and serialization included.
+Step 7.1 (Phase-7 hardening) turns the v0 smoke into a fuller suite: a weighted
+read/write task mix with **varied queries** (so the retrieval path is actually
+exercised rather than served from the cache), plus a ``LoadTestShape`` that ramps
+to a sustained plateau — the shape behind the **acceptance targets** below.
+
+Acceptance targets (validated on a deployed cluster — *not* a CI gate; the CI gates
+are the in-process p99-overhead gate + the chaos-under-load gate)::
+
+ sustained throughput ≥ 1000 RPS # scale workers/users to reach it
+ end-to-end p99 < 500 ms # dev SLO (≤ 250 ms SaaS target)
+ error rate ~ 0% # under steady load
+
Locust is an operator tool, not a project dependency, so install it ad hoc::
uv pip install locust # or: pipx run locust ...
@@ -12,53 +24,117 @@
uv run uvicorn rag_gateway.app:app --port 8000
-Then drive load. Web UI (open http://localhost:8089)::
+Drive load — Web UI (http://localhost:8089)::
locust -f eval/gateway_load_v0/locustfile.py --host http://localhost:8000
-Headless (50 users, spawn 10/s, run 30s)::
+Headless, using the built-in ramp shape::
+
+ locust -f eval/gateway_load_v0/locustfile.py --host http://localhost:8000 --headless
- locust -f eval/gateway_load_v0/locustfile.py --host http://localhost:8000 \
- --headless -u 50 -r 10 -t 30s
+Override the shape via env (peak users / spawn rate / stage seconds)::
-The default gateway runs creds-free (anonymous dev tenant), so no auth headers
-are needed; point ``--host`` at a secured deployment and add an
-``Authorization`` header in ``on_start`` for a real environment.
+ LOAD_PEAK_USERS=400 LOAD_HOLD_S=600 locust -f .../locustfile.py --host ... --headless
+
+The default gateway runs creds-free (anonymous dev tenant), so no auth headers are
+needed; point ``--host`` at a secured deployment and add an ``Authorization``
+header in ``on_start`` for a real environment. See
+[docs/guides/load-testing.md](../../docs/guides/load-testing.md).
"""
from __future__ import annotations
-from locust import HttpUser, between, task
+import os
+import random
+import uuid
+
+from locust import HttpUser, LoadTestShape, between, task
_TENANT = "loadtest"
_PRINCIPAL = "loadtest"
-_QUERY = "what is retrieval augmented generation"
-_RETRIEVE_BODY = {"tenant_id": _TENANT, "principal_id": _PRINCIPAL, "query": _QUERY}
-_QUERY_BODY = {
- "tenant_id": _TENANT,
- "principal_id": _PRINCIPAL,
- "query": _QUERY,
- "rerank": True,
- "pack": True,
- "generate": False,
-}
+# A small topic pool; each request appends a unique nonce so most requests miss the
+# retrieval cache and actually exercise the pipeline (a pure-cache load test would
+# only measure the cache).
+_TOPICS = [
+ "what is retrieval augmented generation",
+ "how does hybrid search fuse vector and keyword results",
+ "explain circuit breakers and graceful degradation",
+ "summarise the data retention policy",
+ "compare OIDC and SAML single sign-on",
+]
+
+
+def _query() -> str:
+ return f"{random.choice(_TOPICS)} #{uuid.uuid4().hex[:8]}" # noqa: S311 - load-test variety
+
+
+def _read_body(*, generate: bool = False) -> dict[str, object]:
+ return {
+ "tenant_id": _TENANT,
+ "principal_id": _PRINCIPAL,
+ "query": _query(),
+ "rerank": True,
+ "pack": True,
+ "generate": generate,
+ }
class GatewayUser(HttpUser):
- """A virtual user hitting the hot read paths in a realistic 3:2:1 mix."""
+ """A virtual user exercising the read paths + the light write/status surfaces."""
# Small think-time so a single box can still push meaningful RPS.
wait_time = between(0.0, 0.05)
- @task(3)
+ @task(5)
def retrieve(self) -> None:
- self.client.post("/v1/retrieve", json=_RETRIEVE_BODY, name="POST /v1/retrieve")
+ body = {"tenant_id": _TENANT, "principal_id": _PRINCIPAL, "query": _query()}
+ self.client.post("/v1/retrieve", json=body, name="POST /v1/retrieve")
- @task(2)
+ @task(4)
def query(self) -> None:
- self.client.post("/v1/query", json=_QUERY_BODY, name="POST /v1/query")
+ self.client.post("/v1/query", json=_read_body(), name="POST /v1/query")
+
+ @task(1)
+ def feedback(self) -> None:
+ body = {
+ "tenant_id": _TENANT,
+ "principal_id": _PRINCIPAL,
+ "request_id": uuid.uuid4().hex,
+ "signal": "thumb_up",
+ }
+ self.client.post("/v1/feedback", json=body, name="POST /v1/feedback")
+
+ @task(1)
+ def status(self) -> None:
+ self.client.get("/v1/status/metrics", name="GET /v1/status/metrics")
@task(1)
def healthz(self) -> None:
self.client.get("/healthz", name="GET /healthz")
+
+
+class RampShape(LoadTestShape):
+ """Ramp → sustained plateau → ramp-down — the acceptance load profile.
+
+ Stages (env-overridable): ramp to ``LOAD_PEAK_USERS`` over ``LOAD_RAMP_S``,
+ hold for ``LOAD_HOLD_S``, then ramp down over ``LOAD_RAMP_S``. Scale the peak
+ (and/or run distributed workers) until the plateau sustains ≥ 1000 RPS.
+ """
+
+ _peak = int(os.environ.get("LOAD_PEAK_USERS", "200"))
+ _spawn = float(os.environ.get("LOAD_SPAWN_RATE", "20"))
+ _ramp_s = int(os.environ.get("LOAD_RAMP_S", "60"))
+ _hold_s = int(os.environ.get("LOAD_HOLD_S", "300"))
+
+ def tick(self) -> tuple[int, float] | None:
+ t = self.get_run_time()
+ ramp, hold = self._ramp_s, self._hold_s
+ if t < ramp: # ramping up
+ return max(1, int(self._peak * t / ramp)), self._spawn
+ if t < ramp + hold: # plateau
+ return self._peak, self._spawn
+ if t < ramp + hold + ramp: # ramping down
+ remaining = (ramp + hold + ramp) - t
+ return max(1, int(self._peak * remaining / ramp)), self._spawn
+ return None # done
diff --git a/tests/perf/test_chaos_under_load.py b/tests/perf/test_chaos_under_load.py
new file mode 100644
index 0000000..cd5280c
--- /dev/null
+++ b/tests/perf/test_chaos_under_load.py
@@ -0,0 +1,67 @@
+"""Gateway chaos-under-load gate (Step 7.1).
+
+The gate: under injected backend faults, the gateway must **degrade gracefully** —
+the Phase-4 circuit breaker opens on the failing backend and the fallback ladder
+keeps serving, so requests stay 2xx instead of 5xx-ing.
+
+``perf``-marked (runs in the dedicated perf CI job, not the cross-OS unit sweep).
+The assertions are deliberately **timing-independent** — no 5xx, 100% success, the
+breaker opens — so they're deterministic regardless of CI-runner speed (the p99
+budget is exercised by the harness ``--check``, not asserted here).
+
+See [docs/guides/load-testing.md](../../docs/guides/load-testing.md) and
+[ADR-0043](../../docs/adr/ADR-0043-load-chaos-testing.md).
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from eval.gateway_chaos_v0.harness import run_chaos
+
+pytestmark = pytest.mark.perf
+
+_REQUESTS = 200
+_CONCURRENCY = 16
+
+
+def test_vector_failure_degrades_gracefully() -> None:
+ # 100% vector-backend failure → the breaker opens, the fallback fuses the
+ # survivors, and every request still succeeds (no 5xx).
+ result = run_chaos(requests=_REQUESTS, concurrency=_CONCURRENCY, vector_failure_rate=1.0)
+ assert result.server_errors == 0, result.status_counts
+ assert result.success_rate == 1.0
+ assert result.breaker_states["vector"] == "open"
+
+
+def test_all_backends_fail_returns_graceful_empty() -> None:
+ # Worst case — every retrieval backend fails — still degrades to graceful
+ # empty answers (200), never a 5xx storm, with at least one breaker tripped.
+ result = run_chaos(
+ requests=_REQUESTS,
+ concurrency=_CONCURRENCY,
+ vector_failure_rate=1.0,
+ keyword_failure_rate=1.0,
+ graph_failure_rate=1.0,
+ )
+ assert result.server_errors == 0, result.status_counts
+ assert result.success_rate == 1.0
+ assert "open" in result.breaker_states.values()
+
+
+def test_healthy_load_keeps_breakers_closed() -> None:
+ # No faults → no breaker trips, full success.
+ result = run_chaos(requests=_REQUESTS, concurrency=_CONCURRENCY, vector_failure_rate=0.0)
+ assert result.server_errors == 0
+ assert result.success_rate == 1.0
+ assert set(result.breaker_states.values()) == {"closed"}
+
+
+def test_latency_injection_does_not_error() -> None:
+ # Injected backend latency slows requests but must not cause errors — latency
+ # is not a failure, so the gateway absorbs it (no breaker trip on slowness).
+ result = run_chaos(
+ requests=100, concurrency=_CONCURRENCY, vector_failure_rate=0.0, latency_ms=3.0
+ )
+ assert result.server_errors == 0
+ assert result.success_rate == 1.0
diff --git a/tests/policy/coverage.py b/tests/policy/coverage.py
index ea8e786..77d6e58 100644
--- a/tests/policy/coverage.py
+++ b/tests/policy/coverage.py
@@ -127,6 +127,12 @@
# exemption for the whole file. Make the exemption explicit so a future
# edit of that corpus text cannot silently fail CI.
Path("eval/agent_loop_v0/harness.py"),
+ # Step 7.1 — chaos fault-injection backends are test doubles that *forward*
+ # the caller's already-policy-merged retrieve_ids / expand to an inner noop
+ # backend (injecting failures / latency first), exactly like the rag-breaker
+ # SPI wrappers above. They read no chunk themselves and add no policy
+ # bypass — the policed read path is the HybridRetriever that drives them.
+ Path("eval/gateway_chaos_v0/faults.py"),
# Step 2.6 — query-understanding components that delegate to LLM /
# Embedder SPIs. Query understanding is pre-retrieval text
# transformation; the upstream gateway (Phase 3) is the