officialCodeWork · officialCodeWork · Jun 8, 2026 · Jun 8, 2026
diff --git a/TRACKER.md b/TRACKER.md
@@ -14,12 +14,14 @@
 | | |
 |---|---|
 | **Last updated** | 2026-06-08 |
-| **Current phase** | Phase 7 — Pilot, Harden, GA (**0 / 10 steps**) — Phase 6 ✅ complete |
-| **Overall** | **74 / 84 steps** — Phases 0–6 complete |
-| **Next action** | **Step 7.1 — Load testing**: Locust suite; 1000 QPS sustained; p99 < 500 ms; chaos under load. |
+| **Current phase** | Phase 7 — Pilot, Harden, GA (**1 / 10 steps**) — Phases 0–6 ✅ complete |
+| **Overall** | **75 / 84 steps** — Phases 0–6 complete |
+| **Next action** | **Step 7.2 — Chaos engineering**: Chaos Monkey / LitmusChaos; kill each backend; verify the fallback chain holds (builds on the 7.1 in-process chaos harness). |
 
 **Recently shipped**
 
+- **7.1** ✅ Load + chaos testing — a **chaos-under-load** CI gate (`eval/gateway_chaos_v0/`): drives the in-process gateway under concurrent load while injecting backend faults (`FaultSpec` + `Chaos{Vector,Keyword,Graph}RetrievalBackend` SPI wrappers behind real breakers) and asserts **graceful degradation** — no 5xx, 100% success, the failing backend's breaker opens (validating the Phase-4 breakers + fallback, builds no new resilience); `tests/perf/test_chaos_under_load.py` (timing-independent, `perf`-marked) + `task chaos-test`.  An extended **Locust v1 suite** (weighted read/write mix + varied queries + a ramp `LoadTestShape`) + documented **acceptance targets** (≥ 1000 RPS sustained, e2e p99 < 500 ms) as a cluster runbook — [#164](https://github.com/officialCodeWork/AgentContextOS/pull/164)
+
 - **6.10** ✅ Compliance posture (Phase 6 capstone) — new `rag-compliance` package: `RetentionEnforcer` drives tenant-scoped `purge_*` (data retention + GDPR right-to-erasure) over the Feedback/Provenance stores (non-abstract `purge_before`/`purge_tenant` SPI capability with `dry_run` so a preview counts-without-deleting; audit chain never purged in place — its retention is the 6.6b WORM export); `POST /v1/compliance/erase` (tenant-scoped, dry-run default + two-flag delete); per-tenant `data_region` enforced at ingest (`ResidencyViolationError` → 403); `GET /v1/status/compliance` posture + `compliance_posture`/`residency_ok`; PII-free `compliance.*` events; `cfg.compliance` + `tenants[].data_region`/`retention_days`; `ragctl compliance report`/`demo`; SOC 2 + GDPR control-mapping docs backed by the live posture — [#162](https://github.com/officialCodeWork/AgentContextOS/pull/162)
 - **6.9** ✅ Air-gapped install bundle — `ragctl airgap build`/`inspect`/`verify`/`install`: one signed `.tar.gz` of all runtime images (`docker save`) + the packaged Helm chart + `rag.yaml` + a standalone installer, for networks with no registry / internet. Integrity = a standard `SHA256SUMS` (verifiable with `sha256sum -c`, no cosign/network) pinned as `manifest.content_hash` + an optional cosign signature over it (the 6.6b content-hash + optional-signature pattern); the same `SHA256SUMS` drives the Python verifier and the standalone `install.{sh,ps1}` (need only docker+helm). Typed/tested logic in `ragctl.airgap` (pure core + stubbable docker/helm/cosign seam; `--dry-run` = verifiable bundle minus blobs); digest-pinned manifest-driven image set (`infra/airgap/images.txt`); `task airgap:*`; `release-airgap.yml` on tags — [#161](https://github.com/officialCodeWork/AgentContextOS/pull/161)
 - **6.8** ✅ SSO / SCIM — `rag-sso`: `FederatedAuth` (an `Auth` SPI backend over per-tenant `OidcProvider` / `SamlProvider`) federates a bearer token / SAML assertion → `Principal` at the existing `authenticate` seam (group claims → `acl_labels`); dependency-free defaults (stdlib HS256 JWT + `defusedxml` SAML), asymmetric OIDC / XML-DSig behind `[oidc]` / `[saml]` extras; algorithm-allowlist downgrade defense; SCIM 2.0 `ScimService` over the new tenant-scoped `ScimStore` SPI driving `/scim/v2/{Users,Groups}` (per-tenant bearer token); per-tenant IdP on `tenants[].sso`; `cfg.sso` / `cfg.scim`; `GET /v1/status/sso`; PII-free `sso.*` / `scim.*` events; `ragctl sso` / `scim` — [#159](https://github.com/officialCodeWork/AgentContextOS/pull/159)
@@ -65,8 +67,8 @@
 | 4 | Reliability | 6 | **6** | 0 |
 | 5 | Eval & Observability | 7 | **7** | 0 |
 | 6 | Governance & Tenancy | 10 | **10** | 0 |
-| 7 | Pilot, Harden, GA | 10 | 0 | 10 |
-| **Total** | | **84** | **74** | **10** |
+| 7 | Pilot, Harden, GA | 10 | **1** | 9 |
+| **Total** | | **84** | **75** | **9** |
 
 ---
 
@@ -797,7 +799,7 @@ New ground — the only prior crypto was HMAC signing.  The V1 plan calls for en
 
 | Step | Title | Status | Planned deliverables |
 |------|-------|:------:|----------------------|
-| 7.1 | Load testing | ⏳ | Locust suite; 1000 QPS sustained; p99 < 500 ms; chaos under load |
+| 7.1 | Load testing | ✅ | [#164](https://github.com/officialCodeWork/AgentContextOS/pull/164) — chaos-under-load CI gate (`eval/gateway_chaos_v0` fault backends + harness; `tests/perf/test_chaos_under_load.py`; graceful degradation: no 5xx + breaker opens); Locust v1 suite (varied-query mix + ramp shape); acceptance targets + runbook |
 | 7.2 | Chaos engineering | ⏳ | Chaos Monkey / LitmusChaos; kill each backend; verify the fallback chain holds |
 | 7.3 | Red-team | ⏳ | Prompt injection, ACL bypass, PII egress, tenant-escape probes |
 | 7.4 | Design partner onboarding | ⏳ | 2–3 design partners; feedback incorporated; case study documented |
@@ -808,6 +810,15 @@ New ground — the only prior crypto was HMAC signing.  The V1 plan calls for en
 | 7.9 | Billing integration | ⏳ | Stripe metered billing; usage export API; invoice generation |
 | 7.10 | GA cutover | ⏳ | `main` tag `v1.0.0`; release notes; all Phase 7 exit gates passed |
 
+### 7.1 — Load + chaos testing ✅ [#164](https://github.com/officialCodeWork/AgentContextOS/pull/164)
+
+- Builds on Step 4.6 (ADR-0025: the in-process p99-overhead gate + Locust v0).  7.1 adds the **load + resilience** side the GA bar needs, split into a deterministic CI gate + a cluster runbook
+- **Chaos-under-load is a deterministic CI gate.**  New `eval/gateway_chaos_v0/` drives the in-process gateway (httpx ASGITransport, no socket / no cluster) under concurrent load while **injecting backend faults**, and asserts the gateway **degrades gracefully** — no 5xx, 100% success, the failing backend's **circuit breaker opens**.  The asserted property is *resilience*, not throughput, so it is **timing-independent** and runs in the `perf` CI job (`tests/perf/test_chaos_under_load.py`); `task chaos-test` runs the harness `--check`
+- **Reuses the Phase-4 machinery — it *validates* it.**  `FaultSpec` (seeded failure probability + injected latency) + **`Chaos{Vector,Keyword,Graph}RetrievalBackend`** are SPI wrappers exactly like the rag-breaker wrappers (allowlisted in the policy-coverage linter for the same reason: they forward the already-policy-merged `retrieve_ids` / `expand`), wired behind real breakers + the fallback ladder.  Under 100% vector failure the vector breaker trips and the fallback fuses the survivors; under all-backends-fail it degrades to graceful empty answers — the client never sees a 5xx
+- **The query is varied per request** (a unique nonce) in both the chaos harness and the Locust suite — identical queries hit the retrieval cache and short-circuit before retrieval, so a constant-query load test would measure the cache, not the pipeline (and the chaos backends would never be exercised)
+- **The 1000-QPS / p99 < 500 ms acceptance is a cluster runbook, not a CI gate** (hardware/backends-bound — same reasoning as ADR-0025).  Extended **Locust v1 suite** (`eval/gateway_load_v0/locustfile.py`): a weighted read/write mix (`/v1/retrieve` · `/v1/query` · `/v1/feedback` · `/v1/status/metrics` · `/healthz`) + a `LoadTestShape` that ramps → holds a plateau → ramps down (env-tunable peak / hold for distributed workers).  Acceptance targets + the runbook in [`guides/load-testing.md`](docs/guides/load-testing.md)
+- **Scope:** in-process chaos gate (resilience) + the Locust suite + runbook for the throughput acceptance.  **Deferred:** distributed-Locust-in-CI against an ephemeral cluster, latency-based breaker tripping (timeouts), storage/LLM-layer fault injection, soak/endurance tests (cluster-level chaos with LitmusChaos is Step 7.2).  ~4 chaos perf tests; all gates green (ruff, mypy --strict, RAG001, policy-coverage, schema/openapi-drift, log-schema).  [ADR-0043](docs/adr/ADR-0043-load-chaos-testing.md), [guides/load-testing.md](docs/guides/load-testing.md), [reference/perf.md](docs/reference/perf.md)
+
 ---
 
 ## PR & Branch History

diff --git a/Taskfile.yml b/Taskfile.yml
@@ -115,6 +115,11 @@ tasks:
     cmds:
       - "{{.PYTHON}} -m eval.gateway_load_v0.harness"
 
+  chaos-test:
+    desc: "Run the chaos-under-load harness — inject backend faults, assert graceful degradation"
+    cmds:
+      - "{{.PYTHON}} -m eval.gateway_chaos_v0.harness --check"
+
   # ---------------------------------------------------------------------------
   # Schemas
   # env: sets PYTHONPATH cross-platform (Task handles Windows vs Unix syntax)

diff --git a/docs/README.md b/docs/README.md
@@ -125,6 +125,7 @@
 | [logging-policy.md](guides/logging-policy.md) | RAG001 policy: structured-logging requirement, allowlist, and how to extend it |
 | [ragctl-quickstart.md](guides/ragctl-quickstart.md) | Five-minute tour of the `ragctl` CLI |
 | [airgap-install.md](guides/airgap-install.md) | Operator runbook (Step 6.9): build + sign an offline bundle on a connected host (`ragctl airgap build --sign`), transfer it, then verify + install on the air-gapped target with the standalone `install.sh`/`install.ps1` (`sha256sum -c SHA256SUMS` → cosign → `docker load` → `helm upgrade --install`); trimming the image set; pointing the chart at in-network backends; offline-verifying a bundle with stdlib tools |
+| [load-testing.md](guides/load-testing.md) | Load + chaos testing runbook (Step 7.1): the two layers — in-process CI gates (p99 ≤ 30 ms overhead + **chaos-under-load** graceful degradation) vs the Locust suite on a cluster (≥ 1000 RPS sustained, e2e p99 < 500 ms); `task chaos-test` / `load-test` / `perf`; running Locust at scale (distributed workers, `LOAD_PEAK_USERS` / `LOAD_HOLD_S` env knobs, the ramp shape); the acceptance-target table; securing the run with an auth header |
 | [curl-quickstart.md](guides/curl-quickstart.md) | 🥈 Curl-able RAG (Step 3.1): 5-minute walkthrough from `curl` to gateway response, including ingest, query, generate, OpenAPI |
 | [grpcurl-quickstart.md](guides/grpcurl-quickstart.md) | gRPC quickstart (Step 3.2): 5-minute walkthrough using `grpcurl` against the in-process server — health check, list corpora, server-streaming query, structured errors |
 | [mcp-quickstart.md](guides/mcp-quickstart.md) | MCP quickstart (Step 3.3): 5-minute walkthrough — `ragctl mcp-query`, running the stdio server, mounting `@ragplatform/mcp` in Claude Desktop, the three tools, error shape |
@@ -186,6 +187,7 @@ broken, and what to fix before committing to the next phase.
 | [ADR-0034-physical-multi-tenancy.md](adr/ADR-0034-physical-multi-tenancy.md) | Decision (Step 6.2): a *dedicated* vector index/collection per tenant. `TenantConfig.dedicated_index` resolves to a `physical_index` key on `TenantSettings`, threaded onto `RequestContext.physical_index`; backends read only `ctx` (graph is backends→core, never rag-config) and namespace their base under it (`<base>-<key>`), lazily creating it; one instance + per-tenant derivation (no per-tenant instances, no SPI change); Noop is the CI conformance oracle (keyed by `physical_index`) for a cross-tenant probe gate that proves isolation independent of the tenant filter; Noop + Pinecone + Qdrant this step, others later |
 | [ADR-0035-acl-pushdown.md](adr/ADR-0035-acl-pushdown.md) | Decision (Step 6.3): label-based ACL push-down at retrieval. `AclPolicyEngine` (a decorator like `QuotaPolicyEngine`) And-merges `any_in("acl_labels", principal.acl_labels)` into every `read_chunk` push-down at the canonical `HybridRetriever` PDP site — overlap semantics via the existing `AnyIn` predicate (zero backend/translator changes), **fail-closed** (label-less principal matches nothing; "public" = a shared label), **opt-in** via `cfg.acl.enabled`; emits `acl.egress_denied` on a request-level denial; graph edge ACLs + post-retrieval re-verification (6.4) deferred |
 | [ADR-0036-acl-egress-verifier.md](adr/ADR-0036-acl-egress-verifier.md) | Decision (Step 6.4): a post-retrieval ACL re-check as a **defense-in-depth second layer** behind the 6.3 push-down. `AclEgressVerifier.verify(ctx, refs)` drops any returned `ChunkRef` whose labels don't overlap the principal's — same overlap semantics (no-op on correct results), reading `ChunkRef.acl_labels` (no re-hydration), **independent of the PDP** (consults only `ctx.principal.acl_labels`) so a push-down bug/bypass can't disable both; wired at the gateway as a `SupportsRoute` wrapper around `app.state.retrieval_router` (covers query/retrieve/corpus/OpenAI/agent); `cfg.acl.verify_egress` default on but gated by `enabled`; emits `acl.egress_violation` on a caught leak; a red-team gate proves a zero escaped-violation rate when the push-down is bypassed; backend-mislabel re-hydration + per-tenant violation metrics deferred |
+| [ADR-0043-load-chaos-testing.md](adr/ADR-0043-load-chaos-testing.md) | Decision (Step 7.1, Phase-7 hardening): split load testing into a deterministic CI gate + a cluster runbook. **Chaos-under-load is a CI gate** — drive the in-process gateway under concurrent load while injecting backend faults (`FaultSpec` + `Chaos{Vector,Keyword,Graph}RetrievalBackend`, SPI wrappers like the breaker wrappers, allowlisted in policy-coverage) and assert **graceful degradation**: no 5xx, 100% success, the relevant breaker opens. The asserted property is *resilience*, not throughput, so it's timing-independent + deterministic; it **reuses the Phase-4 breakers + fallback** (builds no new resilience — it validates them). **The 1000-QPS / p99<500ms acceptance is a cluster runbook, not CI** (hardware/backends-bound, same reasoning as ADR-0025) — shipped as a Locust v1 suite (weighted read/write mix + a `LoadTestShape` ramp + **varied queries** so retrieval is exercised, not the cache) + documented targets. Deferred: distributed-Locust-in-CI against an ephemeral cluster, latency-based breaker tripping, storage/LLM fault injection, soak tests; rejected asserting raw throughput in CI, killing a backend process (no separate process in-process — inject at the SPI boundary) |
 | [ADR-0042-compliance-posture.md](adr/ADR-0042-compliance-posture.md) | Decision (Step 6.10, Phase-6 capstone): add the three compliance pieces on top of the controls the platform already ships (audit/ACL/PII/BYOK/SSO/quotas). New `rag-compliance` package (config-free, like rag-feedback/rag-drift): `RetentionEnforcer` drives tenant-scoped `purge_*`; `compliance_posture`/`residency_ok` are pure. **Retention is a capability on the existing stores, not a new SPI** — non-abstract `purge_before`/`purge_tenant` (default no-op) on Feedback/Provenance, with `dry_run` in the SPI so a preview counts-without-deleting uniformly (ProvenanceStore has no `list`). **Audit is never purged in place** (the hash chain would break) — audit retention is the 6.6b WORM export; `audit_days` is advisory. **Right-to-erasure is always-on, tenant-self-service, two-flag** — `POST /v1/compliance/erase` erases the *calling* tenant's data (scope from the principal, never the body), dry-run by default, delete needs `dry_run=false` AND `confirm=true`. **Residency = declared per tenant + enforced at ingest** (`tenants[].data_region` vs `cfg.compliance.region` → 403), opt-in, a single-deployment assertion not multi-region routing. **The SOC 2 / GDPR mapping is a doc backed by a live posture** (`GET /v1/status/compliance` reports which controls are on, so the mapping is checkable). Deferred: subject-level (vs tenant-level) erasure, an admin retention-sweep endpoint, multi-region routing, automated audit-evidence bundles; rejected purging the audit chain, a `Purgeable` SPI mixin, a static doc with no live backing |
 | [ADR-0041-airgap-bundle.md](adr/ADR-0041-airgap-bundle.md) | Decision (Step 6.9): ship the platform as one signed, self-contained offline bundle (all runtime images + Helm chart + config + installer). Integrity reuses the WORM-export pattern (6.6b): a standard `SHA256SUMS` whose hash is pinned as `manifest.content_hash` is the **hard gate** — verifiable with nothing but `sha256sum`, no network/cosign — and a cosign signature **over `SHA256SUMS`** adds authenticity; the *same* `SHA256SUMS` drives the Python verifier and the standalone shell installer so they can't diverge. The shell/pwsh `install.{sh,ps1}` (shipped inside the bundle) need only docker+helm (air-gap hosts lack uv/the workspace); `ragctl airgap` holds the typed/tested build+verify logic (pure core separated from a stubbable docker/helm/cosign subprocess seam; `--dry-run` = a verifiable bundle minus image blobs, so the path is testable with no Docker). Digest-pinned manifest-driven image set (`infra/airgap/images.txt` + the chart-derived gateway image); key-based cosign is the air-gap recommendation (keyless needs Rekor + an identity policy), keyless is the connected-release path (`release-airgap.yml` on tags). Deferred: ctr/podman load, registry re-tag/push, multi-arch selection, bundling backend charts, TUF-rooted offline keyless verify; rejected `oras`/OCI (no registry to pull from in an air-gap), a second HMAC scheme (cosign already the signer), a pure-shell build (would escape mypy/tests) |
 | [ADR-0040-sso-scim.md](adr/ADR-0040-sso-scim.md) | Decision (Step 6.8): enterprise identity in two surfaces. **Federation** — `FederatedAuth` *is* an `Auth` SPI backend (the `authenticate(token, tenant_id) → Principal` seam already runs at the boundary, so wiring it is the whole integration — no middleware change); group claims → `acl_labels` so Step 6.3 push-down + 6.5 PII egress govern federated users unchanged (`authorize` stays a coarse allow — federation establishes *who*, the PDP decides *what*). Dependency-free defaults (stdlib HS256 JWT with full `exp`/`nbf`/`iss`/`aud` + constant-time compare; `defusedxml` SAML validating Issuer/Conditions/Audience) with asymmetric OIDC (PyJWT, `[oidc]`) + SAML XML-DSig (signxml, `[saml]`, injected verifier → fail-closed) behind extras; **algorithm-allowlist** designs out `alg:none`/RS↔HS confusion. Per-tenant IdP on `tenants[].sso` (reuses Step 6.1 config; no provider → bearer rejected, header-identity still works). **Provisioning** — SCIM 2.0 is a separate surface with its own per-tenant bearer token (`cfg.scim.tokens`, not a user JWT), a tenant-scoped `ScimStore` SPI (`NoopScimStore`) + `ScimService`, SCIM-shaped errors, disabled→404; no new governed SPI call (linter passes). PII-free `sso.*`/`scim.*` events (hashed subject, never email/userName). Deferred: JWKS rotation, SP-initiated SAML + metadata, SCIM bulk/`/Me`/ETag, directory-backed deprovisioning, admin-console card; rejected Authlib/python3-saml (heavy lxml/xmlsec on the default install), a dedicated SSO middleware, SCIM token on `TenantConfig` |