diff --git a/TRACKER.md b/TRACKER.md index 1a3e5cb..a927b16 100644 --- a/TRACKER.md +++ b/TRACKER.md @@ -14,12 +14,13 @@ | | | |---|---| | **Last updated** | 2026-06-08 | -| **Current phase** | Phase 7 — Pilot, Harden, GA (**1 / 10 steps**) — Phases 0–6 ✅ complete | -| **Overall** | **75 / 84 steps** — Phases 0–6 complete | -| **Next action** | **Step 7.2 — Chaos engineering**: Chaos Monkey / LitmusChaos; kill each backend; verify the fallback chain holds (builds on the 7.1 in-process chaos harness). | +| **Current phase** | Phase 7 — Pilot, Harden, GA (**2 / 10 steps**) — Phases 0–6 ✅ complete | +| **Overall** | **76 / 84 steps** — Phases 0–6 complete | +| **Next action** | **Step 7.3 — Red-team**: prompt injection, ACL bypass, PII egress, and tenant-escape probes as an adversarial test suite. | **Recently shipped** +- **7.2** ✅ Chaos engineering — a deterministic in-process **kill-matrix gate** (`eval/gateway_chaos_v0/kill_matrix.py` + `tests/perf/test_chaos_kill_matrix.py`) that extends 7.1 from the three retrieval backends to the **full hot-path set** (vector/keyword/graph/embedder/retrieval_cache/reranker/llm): kill each backend in turn (100% unavailable) behind the real breakers + fallback, drive `/v1/query`, and assert **no single failure 5xx-es the gateway** (no 5xx, on-path retrieval breaker opens, expected degraded shape; a seeded keyword corpus + real `hydrate` make rerank/generate actually run). **Chaos fixed what it found** — the matrix exposed that a down **retrieval cache** or **reranker** 5xx-ed, so the gateway gained two minimal **degrade-open** guards (`gateway.cache.degraded` → miss on `/v1/query` + `/v1/retrieve`; `gateway.rerank.degraded` → retrieval-only, honouring `RerankPipeline`'s "caller decides" contract); LLM + embedder already degraded. **LitmusChaos cluster manifests** (`infra/chaos/`: gateway pod-delete + backend `pod-network-loss`/`latency` with httpProbe acceptance) as the cluster runbook + `task chaos-kill`. No `dist/`/SPI/config change. [#165](https://github.com/officialCodeWork/AgentContextOS/pull/165) - **7.1** ✅ Load + chaos testing — a **chaos-under-load** CI gate (`eval/gateway_chaos_v0/`): drives the in-process gateway under concurrent load while injecting backend faults (`FaultSpec` + `Chaos{Vector,Keyword,Graph}RetrievalBackend` SPI wrappers behind real breakers) and asserts **graceful degradation** — no 5xx, 100% success, the failing backend's breaker opens (validating the Phase-4 breakers + fallback, builds no new resilience); `tests/perf/test_chaos_under_load.py` (timing-independent, `perf`-marked) + `task chaos-test`. An extended **Locust v1 suite** (weighted read/write mix + varied queries + a ramp `LoadTestShape`) + documented **acceptance targets** (≥ 1000 RPS sustained, e2e p99 < 500 ms) as a cluster runbook — [#164](https://github.com/officialCodeWork/AgentContextOS/pull/164) - **6.10** ✅ Compliance posture (Phase 6 capstone) — new `rag-compliance` package: `RetentionEnforcer` drives tenant-scoped `purge_*` (data retention + GDPR right-to-erasure) over the Feedback/Provenance stores (non-abstract `purge_before`/`purge_tenant` SPI capability with `dry_run` so a preview counts-without-deleting; audit chain never purged in place — its retention is the 6.6b WORM export); `POST /v1/compliance/erase` (tenant-scoped, dry-run default + two-flag delete); per-tenant `data_region` enforced at ingest (`ResidencyViolationError` → 403); `GET /v1/status/compliance` posture + `compliance_posture`/`residency_ok`; PII-free `compliance.*` events; `cfg.compliance` + `tenants[].data_region`/`retention_days`; `ragctl compliance report`/`demo`; SOC 2 + GDPR control-mapping docs backed by the live posture — [#162](https://github.com/officialCodeWork/AgentContextOS/pull/162) @@ -67,8 +68,8 @@ | 4 | Reliability | 6 | **6** | 0 | | 5 | Eval & Observability | 7 | **7** | 0 | | 6 | Governance & Tenancy | 10 | **10** | 0 | -| 7 | Pilot, Harden, GA | 10 | **1** | 9 | -| **Total** | | **84** | **75** | **9** | +| 7 | Pilot, Harden, GA | 10 | **2** | 8 | +| **Total** | | **84** | **76** | **8** | --- @@ -800,7 +801,7 @@ New ground — the only prior crypto was HMAC signing. The V1 plan calls for en | Step | Title | Status | Planned deliverables | |------|-------|:------:|----------------------| | 7.1 | Load testing | ✅ | [#164](https://github.com/officialCodeWork/AgentContextOS/pull/164) — chaos-under-load CI gate (`eval/gateway_chaos_v0` fault backends + harness; `tests/perf/test_chaos_under_load.py`; graceful degradation: no 5xx + breaker opens); Locust v1 suite (varied-query mix + ramp shape); acceptance targets + runbook | -| 7.2 | Chaos engineering | ⏳ | Chaos Monkey / LitmusChaos; kill each backend; verify the fallback chain holds | +| 7.2 | Chaos engineering | ✅ | [#165](https://github.com/officialCodeWork/AgentContextOS/pull/165) — in-process **kill-matrix** gate (kill each hot-path backend → no 5xx, breaker opens, expected degraded shape); two new gateway **degrade-open** guards (retrieval cache + reranker); LitmusChaos cluster manifests (`infra/chaos/`) + `task chaos-kill` | | 7.3 | Red-team | ⏳ | Prompt injection, ACL bypass, PII egress, tenant-escape probes | | 7.4 | Design partner onboarding | ⏳ | 2–3 design partners; feedback incorporated; case study documented | | 7.5 | Documentation site | ⏳ | Docusaurus / MkDocs site; API reference generated from OpenAPI; quickstart guides | @@ -819,6 +820,15 @@ New ground — the only prior crypto was HMAC signing. The V1 plan calls for en - **The 1000-QPS / p99 < 500 ms acceptance is a cluster runbook, not a CI gate** (hardware/backends-bound — same reasoning as ADR-0025). Extended **Locust v1 suite** (`eval/gateway_load_v0/locustfile.py`): a weighted read/write mix (`/v1/retrieve` · `/v1/query` · `/v1/feedback` · `/v1/status/metrics` · `/healthz`) + a `LoadTestShape` that ramps → holds a plateau → ramps down (env-tunable peak / hold for distributed workers). Acceptance targets + the runbook in [`guides/load-testing.md`](docs/guides/load-testing.md) - **Scope:** in-process chaos gate (resilience) + the Locust suite + runbook for the throughput acceptance. **Deferred:** distributed-Locust-in-CI against an ephemeral cluster, latency-based breaker tripping (timeouts), storage/LLM-layer fault injection, soak/endurance tests (cluster-level chaos with LitmusChaos is Step 7.2). ~4 chaos perf tests; all gates green (ruff, mypy --strict, RAG001, policy-coverage, schema/openapi-drift, log-schema). [ADR-0043](docs/adr/ADR-0043-load-chaos-testing.md), [guides/load-testing.md](docs/guides/load-testing.md), [reference/perf.md](docs/reference/perf.md) +### 7.2 — Chaos engineering ✅ [#165](https://github.com/officialCodeWork/AgentContextOS/pull/165) + +- Builds on Step 7.1 (ADR-0043). 7.1 fault-injected the three *retrieval* backends under load; 7.2 closes out the deliverable — "**kill each backend; verify the fallback chain holds**" — and the LitmusChaos piece 7.1 deferred. The GA bar is a sharp invariant: **no single backend failure can 5xx the gateway** +- **The kill-matrix is a deterministic in-process CI gate.** `eval/gateway_chaos_v0/kill_matrix.py` walks the full hot-path backend set — vector / keyword / graph / embedder / retrieval_cache / reranker / llm — kills **one at a time** (100% unavailable) behind the *real* Phase-4 breakers + fallback + the gateway degrade guards, drives `/v1/query`, and asserts **no 5xx, 100% success**, the killed *on-path* retrieval breaker **opens**, and the expected degraded shape (`tests/perf/test_chaos_kill_matrix.py`, `perf`-marked + timing-independent; `task chaos-kill`). A tiny seeded keyword corpus + the store's real `hydrate` make rerank/pack/generate actually run, so every kill exercises a genuine degrade path +- **Chaos *fixes* what it finds.** Running the matrix exposed two ways to 5xx: a down **retrieval cache** (`get_best`/`put_with_query`) and a down **reranker** (`reranker.rerank`) propagated unhandled. Both gained minimal **degrade-open** guards at the call site (matching the existing LLM guard): the cache treats an outage as a *miss* (`gateway.cache.degraded`, on `/v1/query` + `/v1/retrieve`); the reranker falls back to *retrieval-only* (`gateway.rerank.degraded`, honouring `RerankPipeline`'s documented "caller decides" contract — the reranker is also the hydration stage, so the degraded response carries no citations, same shape as `rerank=false`). The LLM (answer-generation guard) + embedder (understanding's per-component catch) already degraded +- **Graph is off the default read path.** `RetrievalRouter.decide` only sets `use_graph` when the caller supplies graph seeds; a standard query never does, so killing graph cannot affect it (breaker stays closed) — the matrix records this honestly (no 5xx asserted, breaker-open *not* required) rather than forcing a synthetic graph query +- **Cluster chaos is a runbook, not a CI gate** (topology-bound). `infra/chaos/` LitmusChaos manifests: gateway `pod-delete` (PDB-protected instance loss) + `pod-network-loss`/`pod-network-latency` toward a backend host (the cluster way to "kill a backend", since backends are reached by host), each with an embedded **httpProbe** asserting the gateway stays 200 throughout +- **Scope:** kill-matrix gate + the two degrade-open hardenings + the LitmusChaos runbook. No new core/wire types, SPI methods, config, or events (`dist/` untouched; the degrade kinds are log-only like `gateway.answer.failed`; the kill wrappers are pure-raise so the policy-coverage linter needs no new rule). **Deferred:** latency-based breaker tripping, multi-backend simultaneous kills as a gate, Litmus-in-CI against an ephemeral cluster, soak chaos. 12 kill-matrix perf tests; all gates green (ruff, mypy --strict 325 files, RAG001, policy-coverage, schema/openapi-drift, log-schema). [ADR-0044](docs/adr/ADR-0044-chaos-engineering.md), [guides/chaos-engineering.md](docs/guides/chaos-engineering.md), [reference/perf.md](docs/reference/perf.md) + --- ## PR & Branch History @@ -964,6 +974,7 @@ Complete log of every PR. Routine Dependabot bumps are grouped; everything else | [#156](https://github.com/officialCodeWork/AgentContextOS/pull/156) | 2026-06-08 | feat(crypto): cfg.kms + key-manager factory + AWS KMS provider (Step 6.7b) | | [#157](https://github.com/officialCodeWork/AgentContextOS/pull/157) | 2026-06-08 | feat(crypto): GCP / Azure / Vault KMS providers (Step 6.7c) | | [#158](https://github.com/officialCodeWork/AgentContextOS/pull/158) | 2026-06-08 | feat(crypto): zero-downtime key rotation — RotatingKeyManager (Step 6.7d) | +| [#165](https://github.com/officialCodeWork/AgentContextOS/pull/165) | 2026-06-08 | test(perf): chaos kill-matrix gate + cache/rerank degrade-open + LitmusChaos (Step 7.2) | | #78–#80, #116–#118 | Open | Dependabot bumps — awaiting merge | | #81 | Closed | Dependabot bump — superseded | diff --git a/Taskfile.yml b/Taskfile.yml index ada11f2..d6c36d5 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -120,6 +120,11 @@ tasks: cmds: - "{{.PYTHON}} -m eval.gateway_chaos_v0.harness --check" + chaos-kill: + desc: "Run the chaos kill-matrix — kill each backend in turn, assert no single failure 5xx-es the gateway" + cmds: + - "{{.PYTHON}} -m eval.gateway_chaos_v0.kill_matrix --check" + # --------------------------------------------------------------------------- # Schemas # env: sets PYTHONPATH cross-platform (Task handles Windows vs Unix syntax) diff --git a/apps/gateway/src/rag_gateway/query.py b/apps/gateway/src/rag_gateway/query.py index 93b32eb..589cbd8 100644 --- a/apps/gateway/src/rag_gateway/query.py +++ b/apps/gateway/src/rag_gateway/query.py @@ -40,6 +40,7 @@ ProvenanceNotFoundError, RagError, RateLimitError, + RerankerError, ResidencyViolationError, RetrievalError, ScimConflictError, @@ -604,13 +605,28 @@ async def _handle_query( shadow_expansion: dict[str, list[str]] = {} cache_hit = False if cache is not None: - cached_refs = await cache.get_best( - ctx, - query_text=body.query, - plan_hash=plan_hash, - corpus_version=corpus_version, - scope=scope, - ) + try: + cached_refs = await cache.get_best( + ctx, + query_text=body.query, + plan_hash=plan_hash, + corpus_version=corpus_version, + scope=scope, + ) + except Exception as exc: + # The retrieval cache is a non-authoritative optimisation: a + # backend outage (e.g. Redis down) must degrade to a cache *miss*, + # never fail the request. Chaos gate: Step 7.2 kills this backend. + _log.warning( + "gateway.cache.degraded", + extra={ + "event_kind": "gateway.cache.degraded", + "stage": "get", + "tenant_id": str(ctx.tenant_id), + "error": repr(exc), + }, + ) + cached_refs = None if cached_refs is not None: cache_hit = True chunk_refs = cached_refs @@ -650,28 +666,56 @@ async def _handle_query( raise RetrievalError(f"retrieval failed: {exc!r}") from exc if cache is not None: - await cache.put_with_query( - ctx, - query_text=body.query, - plan_hash=plan_hash, - corpus_version=corpus_version, - value=chunk_refs, - ) + try: + await cache.put_with_query( + ctx, + query_text=body.query, + plan_hash=plan_hash, + corpus_version=corpus_version, + value=chunk_refs, + ) + except Exception as exc: + # A cache *write* failure is purely an optimisation loss — + # the fresh result is already in hand; never fail on it. + _log.warning( + "gateway.cache.degraded", + extra={ + "event_kind": "gateway.cache.degraded", + "stage": "put", + "tenant_id": str(ctx.tenant_id), + "error": repr(exc), + }, + ) # A/B routing (Step 5.7c) — record the served arm's outcome into the # experiment tracker, once per fresh retrieval (not on a cache hit). _record_ab_outcome(deps, variant, chunk_refs) - # Optional rerank (returns hydrated Chunks). + # Optional rerank (returns hydrated Chunks). A reranker outage is + # non-fatal: the RerankPipeline propagates RerankerError by contract and + # leaves the caller to fall back to retrieval-only. We degrade to the + # chunk-ref view (no hydrated chunks / citations this request) rather than + # 5xx — Step 7.2's chaos gate kills this backend and asserts the survival. chunks: list[Chunk] = [] if body.rerank and chunk_refs: with timings.stage("rerank"): - chunks = await deps.reranker.rerank( - ctx, - body.query, - chunk_refs, - top_k=body.top_k, - ) + try: + chunks = await deps.reranker.rerank( + ctx, + body.query, + chunk_refs, + top_k=body.top_k, + ) + except RerankerError as exc: + _log.warning( + "gateway.rerank.degraded", + extra={ + "event_kind": "gateway.rerank.degraded", + "tenant_id": str(ctx.tenant_id), + "error": repr(exc), + }, + ) + chunks = [] # Optional packing. packed: PackedContext | None = None @@ -831,13 +875,26 @@ async def _handle_retrieve( scope = _scope_for(body, variant) corpus_version = await resolve_corpus_version(ctx, deps.corpus_store, list(body.corpus_ids)) if cache is not None: - cached_refs = await cache.get_best( - ctx, - query_text=body.query, - plan_hash=plan_hash, - corpus_version=corpus_version, - scope=scope, - ) + try: + cached_refs = await cache.get_best( + ctx, + query_text=body.query, + plan_hash=plan_hash, + corpus_version=corpus_version, + scope=scope, + ) + except Exception as exc: + # Non-authoritative cache: a backend outage degrades to a miss. + _log.warning( + "gateway.cache.degraded", + extra={ + "event_kind": "gateway.cache.degraded", + "stage": "get", + "tenant_id": str(ctx.tenant_id), + "error": repr(exc), + }, + ) + cached_refs = None if cached_refs is not None: span.set_attribute("rag.gateway.cache_hit", True) total_ms = (time.monotonic() - start) * 1000.0 @@ -886,14 +943,26 @@ async def _handle_retrieve( raise RetrievalError(f"retrieval failed: {exc!r}") from exc if cache is not None: - await cache.put_with_query( - ctx, - query_text=body.query, - plan_hash=plan_hash, - corpus_version=corpus_version, - value=chunk_refs, - scope=scope, - ) + try: + await cache.put_with_query( + ctx, + query_text=body.query, + plan_hash=plan_hash, + corpus_version=corpus_version, + value=chunk_refs, + scope=scope, + ) + except Exception as exc: + # A cache write failure is an optimisation loss, never fatal. + _log.warning( + "gateway.cache.degraded", + extra={ + "event_kind": "gateway.cache.degraded", + "stage": "put", + "tenant_id": str(ctx.tenant_id), + "error": repr(exc), + }, + ) # A/B routing (Step 5.7c) — record the served arm's outcome (fresh path). _record_ab_outcome(deps, variant, chunk_refs) diff --git a/docs/README.md b/docs/README.md index 954d490..abebeaa 100644 --- a/docs/README.md +++ b/docs/README.md @@ -126,6 +126,7 @@ | [ragctl-quickstart.md](guides/ragctl-quickstart.md) | Five-minute tour of the `ragctl` CLI | | [airgap-install.md](guides/airgap-install.md) | Operator runbook (Step 6.9): build + sign an offline bundle on a connected host (`ragctl airgap build --sign`), transfer it, then verify + install on the air-gapped target with the standalone `install.sh`/`install.ps1` (`sha256sum -c SHA256SUMS` → cosign → `docker load` → `helm upgrade --install`); trimming the image set; pointing the chart at in-network backends; offline-verifying a bundle with stdlib tools | | [load-testing.md](guides/load-testing.md) | Load + chaos testing runbook (Step 7.1): the two layers — in-process CI gates (p99 ≤ 30 ms overhead + **chaos-under-load** graceful degradation) vs the Locust suite on a cluster (≥ 1000 RPS sustained, e2e p99 < 500 ms); `task chaos-test` / `load-test` / `perf`; running Locust at scale (distributed workers, `LOAD_PEAK_USERS` / `LOAD_HOLD_S` env knobs, the ramp shape); the acceptance-target table; securing the run with an auth header | +| [chaos-engineering.md](guides/chaos-engineering.md) | Chaos engineering runbook (Step 7.2): the two layers — the in-process **kill-matrix gate** (`task chaos-kill`: kill *each* hot-path backend in turn → no 5xx, breaker opens, expected degraded shape) vs **LitmusChaos** on a cluster (`infra/chaos/`: gateway pod-delete + backend network-loss/latency with httpProbe acceptance); the per-backend survival table; the two degrade-open gaps 7.2 hardened (retrieval cache + reranker); why graph is off the default read path; acceptance targets | | [curl-quickstart.md](guides/curl-quickstart.md) | 🥈 Curl-able RAG (Step 3.1): 5-minute walkthrough from `curl` to gateway response, including ingest, query, generate, OpenAPI | | [grpcurl-quickstart.md](guides/grpcurl-quickstart.md) | gRPC quickstart (Step 3.2): 5-minute walkthrough using `grpcurl` against the in-process server — health check, list corpora, server-streaming query, structured errors | | [mcp-quickstart.md](guides/mcp-quickstart.md) | MCP quickstart (Step 3.3): 5-minute walkthrough — `ragctl mcp-query`, running the stdio server, mounting `@ragplatform/mcp` in Claude Desktop, the three tools, error shape | @@ -188,6 +189,7 @@ broken, and what to fix before committing to the next phase. | [ADR-0035-acl-pushdown.md](adr/ADR-0035-acl-pushdown.md) | Decision (Step 6.3): label-based ACL push-down at retrieval. `AclPolicyEngine` (a decorator like `QuotaPolicyEngine`) And-merges `any_in("acl_labels", principal.acl_labels)` into every `read_chunk` push-down at the canonical `HybridRetriever` PDP site — overlap semantics via the existing `AnyIn` predicate (zero backend/translator changes), **fail-closed** (label-less principal matches nothing; "public" = a shared label), **opt-in** via `cfg.acl.enabled`; emits `acl.egress_denied` on a request-level denial; graph edge ACLs + post-retrieval re-verification (6.4) deferred | | [ADR-0036-acl-egress-verifier.md](adr/ADR-0036-acl-egress-verifier.md) | Decision (Step 6.4): a post-retrieval ACL re-check as a **defense-in-depth second layer** behind the 6.3 push-down. `AclEgressVerifier.verify(ctx, refs)` drops any returned `ChunkRef` whose labels don't overlap the principal's — same overlap semantics (no-op on correct results), reading `ChunkRef.acl_labels` (no re-hydration), **independent of the PDP** (consults only `ctx.principal.acl_labels`) so a push-down bug/bypass can't disable both; wired at the gateway as a `SupportsRoute` wrapper around `app.state.retrieval_router` (covers query/retrieve/corpus/OpenAI/agent); `cfg.acl.verify_egress` default on but gated by `enabled`; emits `acl.egress_violation` on a caught leak; a red-team gate proves a zero escaped-violation rate when the push-down is bypassed; backend-mislabel re-hydration + per-tenant violation metrics deferred | | [ADR-0043-load-chaos-testing.md](adr/ADR-0043-load-chaos-testing.md) | Decision (Step 7.1, Phase-7 hardening): split load testing into a deterministic CI gate + a cluster runbook. **Chaos-under-load is a CI gate** — drive the in-process gateway under concurrent load while injecting backend faults (`FaultSpec` + `Chaos{Vector,Keyword,Graph}RetrievalBackend`, SPI wrappers like the breaker wrappers, allowlisted in policy-coverage) and assert **graceful degradation**: no 5xx, 100% success, the relevant breaker opens. The asserted property is *resilience*, not throughput, so it's timing-independent + deterministic; it **reuses the Phase-4 breakers + fallback** (builds no new resilience — it validates them). **The 1000-QPS / p99<500ms acceptance is a cluster runbook, not CI** (hardware/backends-bound, same reasoning as ADR-0025) — shipped as a Locust v1 suite (weighted read/write mix + a `LoadTestShape` ramp + **varied queries** so retrieval is exercised, not the cache) + documented targets. Deferred: distributed-Locust-in-CI against an ephemeral cluster, latency-based breaker tripping, storage/LLM fault injection, soak tests; rejected asserting raw throughput in CI, killing a backend process (no separate process in-process — inject at the SPI boundary) | +| [ADR-0044-chaos-engineering.md](adr/ADR-0044-chaos-engineering.md) | Decision (Step 7.2, Phase-7 hardening): **kill each backend, verify the fallback chain holds** — a deterministic in-process **kill-matrix CI gate** extending 7.1 to the full hot-path set (vector/keyword/graph/embedder/retrieval_cache/reranker/llm); kill one at a time → no 5xx, on-path retrieval breaker opens, expected degraded shape; seeded keyword corpus + real `hydrate` so rerank/generate actually run. **Chaos *fixes* what it finds** — the matrix exposed that a down retrieval-cache or reranker 5xx-ed, so the gateway gained two **degrade-open** guards (`gateway.cache.degraded` → miss; `gateway.rerank.degraded` → retrieval-only, honouring `RerankPipeline`'s "caller decides" contract); LLM + embedder already degraded. **Graph is off the seed-less read path** → its kill is survivable by construction (no breaker-open required). **Cluster chaos is a runbook** — LitmusChaos `infra/chaos/` (gateway pod-delete + backend `pod-network-loss`/`latency` with httpProbe acceptance). No `dist/`/SPI/config change (degrade kinds aren't registered events; kill wrappers are pure-raise). Deferred: latency-based breaker tripping, multi-kill-as-gate, Litmus-in-CI, soak; rejected leaving the holes documented-only, reranker fallback inside the pipeline, a synthetic graph query, pod-kill in CI | | [ADR-0042-compliance-posture.md](adr/ADR-0042-compliance-posture.md) | Decision (Step 6.10, Phase-6 capstone): add the three compliance pieces on top of the controls the platform already ships (audit/ACL/PII/BYOK/SSO/quotas). New `rag-compliance` package (config-free, like rag-feedback/rag-drift): `RetentionEnforcer` drives tenant-scoped `purge_*`; `compliance_posture`/`residency_ok` are pure. **Retention is a capability on the existing stores, not a new SPI** — non-abstract `purge_before`/`purge_tenant` (default no-op) on Feedback/Provenance, with `dry_run` in the SPI so a preview counts-without-deleting uniformly (ProvenanceStore has no `list`). **Audit is never purged in place** (the hash chain would break) — audit retention is the 6.6b WORM export; `audit_days` is advisory. **Right-to-erasure is always-on, tenant-self-service, two-flag** — `POST /v1/compliance/erase` erases the *calling* tenant's data (scope from the principal, never the body), dry-run by default, delete needs `dry_run=false` AND `confirm=true`. **Residency = declared per tenant + enforced at ingest** (`tenants[].data_region` vs `cfg.compliance.region` → 403), opt-in, a single-deployment assertion not multi-region routing. **The SOC 2 / GDPR mapping is a doc backed by a live posture** (`GET /v1/status/compliance` reports which controls are on, so the mapping is checkable). Deferred: subject-level (vs tenant-level) erasure, an admin retention-sweep endpoint, multi-region routing, automated audit-evidence bundles; rejected purging the audit chain, a `Purgeable` SPI mixin, a static doc with no live backing | | [ADR-0041-airgap-bundle.md](adr/ADR-0041-airgap-bundle.md) | Decision (Step 6.9): ship the platform as one signed, self-contained offline bundle (all runtime images + Helm chart + config + installer). Integrity reuses the WORM-export pattern (6.6b): a standard `SHA256SUMS` whose hash is pinned as `manifest.content_hash` is the **hard gate** — verifiable with nothing but `sha256sum`, no network/cosign — and a cosign signature **over `SHA256SUMS`** adds authenticity; the *same* `SHA256SUMS` drives the Python verifier and the standalone shell installer so they can't diverge. The shell/pwsh `install.{sh,ps1}` (shipped inside the bundle) need only docker+helm (air-gap hosts lack uv/the workspace); `ragctl airgap` holds the typed/tested build+verify logic (pure core separated from a stubbable docker/helm/cosign subprocess seam; `--dry-run` = a verifiable bundle minus image blobs, so the path is testable with no Docker). Digest-pinned manifest-driven image set (`infra/airgap/images.txt` + the chart-derived gateway image); key-based cosign is the air-gap recommendation (keyless needs Rekor + an identity policy), keyless is the connected-release path (`release-airgap.yml` on tags). Deferred: ctr/podman load, registry re-tag/push, multi-arch selection, bundling backend charts, TUF-rooted offline keyless verify; rejected `oras`/OCI (no registry to pull from in an air-gap), a second HMAC scheme (cosign already the signer), a pure-shell build (would escape mypy/tests) | | [ADR-0040-sso-scim.md](adr/ADR-0040-sso-scim.md) | Decision (Step 6.8): enterprise identity in two surfaces. **Federation** — `FederatedAuth` *is* an `Auth` SPI backend (the `authenticate(token, tenant_id) → Principal` seam already runs at the boundary, so wiring it is the whole integration — no middleware change); group claims → `acl_labels` so Step 6.3 push-down + 6.5 PII egress govern federated users unchanged (`authorize` stays a coarse allow — federation establishes *who*, the PDP decides *what*). Dependency-free defaults (stdlib HS256 JWT with full `exp`/`nbf`/`iss`/`aud` + constant-time compare; `defusedxml` SAML validating Issuer/Conditions/Audience) with asymmetric OIDC (PyJWT, `[oidc]`) + SAML XML-DSig (signxml, `[saml]`, injected verifier → fail-closed) behind extras; **algorithm-allowlist** designs out `alg:none`/RS↔HS confusion. Per-tenant IdP on `tenants[].sso` (reuses Step 6.1 config; no provider → bearer rejected, header-identity still works). **Provisioning** — SCIM 2.0 is a separate surface with its own per-tenant bearer token (`cfg.scim.tokens`, not a user JWT), a tenant-scoped `ScimStore` SPI (`NoopScimStore`) + `ScimService`, SCIM-shaped errors, disabled→404; no new governed SPI call (linter passes). PII-free `sso.*`/`scim.*` events (hashed subject, never email/userName). Deferred: JWKS rotation, SP-initiated SAML + metadata, SCIM bulk/`/Me`/ETag, directory-backed deprovisioning, admin-console card; rejected Authlib/python3-saml (heavy lxml/xmlsec on the default install), a dedicated SSO middleware, SCIM token on `TenantConfig` | diff --git a/docs/adr/ADR-0044-chaos-engineering.md b/docs/adr/ADR-0044-chaos-engineering.md new file mode 100644 index 0000000..d77e404 --- /dev/null +++ b/docs/adr/ADR-0044-chaos-engineering.md @@ -0,0 +1,108 @@ +# ADR-0044 — Chaos engineering: kill each backend, verify the fallback chain + +**Status:** Accepted +**Date:** 2026-06-08 +**Step:** 7.2 — Chaos engineering (Phase 7 — Pilot, Harden, GA) +**Related:** [ADR-0043](ADR-0043-load-chaos-testing.md) (load + chaos-under-load, Step 7.1), [ADR-0025](ADR-0025-latency-load-testing.md) (latency gate), ADR-0016 (circuit breakers / fallback — Phase 4), [guides/chaos-engineering.md](../guides/chaos-engineering.md), [reference/perf.md](../reference/perf.md) + +## Context + +Step 7.1 (ADR-0043) delivered **chaos-under-load**: fault-inject the three +*retrieval* backends under concurrent load and assert graceful degradation. It +explicitly deferred to 7.2 "cluster-level chaos with LitmusChaos" and the wider +backend set. Step 7.2 — "Chaos Monkey / LitmusChaos; **kill each backend**; +verify the fallback chain holds" — closes that out. The GA bar is a sharp, +sellable property: **no single backend failure can 5xx the gateway.** + +Running the kill matrix exposed that this was *not yet true*. The gateway query +path called the **retrieval cache** (`get_best` / `put_with_query`) and the +**reranker** (`reranker.rerank`) with no error handling, so a 100 %-down cache +or reranker propagated to a 5xx. The LLM and embedder already degraded (the +answer-generation guard; the understanding pipeline's per-component catch), and +the retrieval backends already had breakers + the fallback ladder. + +## Decision + +**1. The kill-matrix is a deterministic, in-process CI gate.** Mirroring the +7.1 chaos gate, it walks the full hot-path backend set — vector, keyword, graph, +embedder, retrieval cache, reranker, LLM — kills **one at a time** (100 % +unavailable, the worst case) behind the *real* Phase-4 breakers + fallback + +the gateway degrade guards, drives `/v1/query`, and asserts **no 5xx, 100 % +success**, plus the killed *on-path* retrieval backend's **circuit breaker +opens**. The asserted property is resilience, so it is timing-independent and +runs in the `perf` CI job (`tests/perf/test_chaos_kill_matrix.py`; +`task chaos-kill`). A tiny seeded keyword corpus + the store's real `hydrate` +make rerank / pack / generate actually run, so every kill exercises a genuine +degrade path (an empty corpus would short-circuit them). + +**2. Chaos engineering *fixes* what it finds — 7.2 hardens the two real gaps.** +Where 7.1 "built no new resilience", 7.2's whole point is to find weaknesses and +close them. So the gateway gains two **degrade-open** guards (minimal, at the +call site, matching the existing LLM-guard pattern): + +- **Retrieval cache** — a non-authoritative optimisation. A `get` failure + degrades to a *cache miss*; a `put` failure is an optimisation loss. Both log + `gateway.cache.degraded` and never fail the request (`/v1/query` + `/v1/retrieve`). +- **Reranker** — the `RerankPipeline` already propagates `RerankerError` *by + contract* and leaves "the caller (the gateway) to decide whether to fall back + to retrieval-only". The gateway now catches it, logs `gateway.rerank.degraded`, + and degrades to retrieval-only (the reranker is also the hydration stage, so + the degraded response carries no hydrated citations — the same shape as + `rerank=false`). + +These are observable, honest degrades (a killed reranker → 200 with no +citations; a killed LLM → 200 with citations but no answer; a killed cache → +200, a clean miss) and are asserted directly by the gate. + +**3. The graph backend is off the default read path — its kill is survivable by +construction.** `RetrievalRouter.decide` only sets `use_graph` when the caller +supplies graph seeds; a standard `/v1/query` never does, so the graph backend +is never called and killing it cannot affect that query (its breaker stays +closed). The matrix records this honestly — graph is in the matrix (no 5xx is +asserted) but its breaker-open is *not* required — rather than forcing a +synthetic graph query just to make a breaker trip. + +**4. Cluster chaos is a runbook, not a CI gate.** Real pod-kills + network +faults depend on the cluster, CNI, and topology — they cannot be a deterministic +PR gate (ADR-0043/0025 reasoning). So 7.2 ships **LitmusChaos manifests** +(`infra/chaos/`): gateway `pod-delete` (instance loss, PDB-protected) and +`pod-network-loss` / `pod-network-latency` toward a backend host (the +cluster-level way to "kill a backend", since backends are reached by host). Each +`ChaosEngine` embeds an **httpProbe** asserting the gateway stays 200 throughout +— the cluster-level "no 5xx under chaos" — plus a [runbook](../guides/chaos-engineering.md) +and acceptance targets. + +## Consequences + +- Gateway gains two degrade-open guards (`gateway.cache.degraded` / + `gateway.rerank.degraded` — log kinds, not registered events, matching + `gateway.answer.failed`). No new core types, wire types, SPI methods, or + config — `dist/` is untouched, the policy-coverage linter needs no new entry + (the kill wrappers are pure-raise). +- New `eval/gateway_chaos_v0/kill.py` (kill wrappers) + `kill_matrix.py` + (harness) + `tests/perf/test_chaos_kill_matrix.py` (the gate) + `task chaos-kill`; + `infra/chaos/` LitmusChaos manifests + runbook. +- The GA resilience invariant — *no single backend failure 5xx-es the gateway* — + is now CI-enforced deterministically and cluster-validatable. +- **Deferred:** latency-based breaker tripping (timeouts, not just errors); + multi-backend simultaneous kills as a gate (the matrix kills one at a time); + storage/LLM-layer fault injection in the cluster runbook beyond network faults; + automated chaos in CI against an ephemeral cluster (Litmus-operator job); soak/ + endurance chaos. + +## Alternatives considered + +- **Leave the cache/reranker 5xx holes, only document them.** Rejected — a GA + *hardening* step that finds two ways to 5xx the gateway and ships them unfixed + defeats the purpose of chaos engineering. The fixes are minimal and match the + existing degrade-open pattern. +- **Put the reranker fallback inside `RerankPipeline`.** Rejected — the pipeline + *documents* that it propagates `RerankerError` and leaves the fallback policy + to the caller (so a caller that *wants* the failure surfaced still can). The + gateway is the right place for the retrieval-only degrade. +- **Force a graph-shaped query so the graph kill trips a breaker.** Rejected as + dishonest — it would manufacture a code path a real `/v1/query` never takes. + The truthful statement ("graph is off the seed-less read path") is more useful. +- **Kill backends by deleting pods in CI.** Rejected (ADR-0043) — in-process + kills at the SPI boundary exercise the exact breaker + fallback + degrade paths + deterministically; pod-kill chaos is the cluster runbook. diff --git a/docs/guides/chaos-engineering.md b/docs/guides/chaos-engineering.md new file mode 100644 index 0000000..3743ab9 --- /dev/null +++ b/docs/guides/chaos-engineering.md @@ -0,0 +1,104 @@ +# Guide: chaos engineering (Step 7.2) + +How to validate the GA resilience bar — **no single backend failure can 5xx the +gateway, and the fallback chain holds** — using the deterministic in-process +kill-matrix gate (CI) and the LitmusChaos manifests (cluster). + +## The two layers + +| Layer | What it proves | Where it runs | +|-------|----------------|---------------| +| **Kill-matrix gate** (CI) | kill *each* backend in turn → no 5xx, breaker opens, the gateway degrades gracefully | `tests/perf/test_chaos_kill_matrix.py` — deterministic, no infra | +| **LitmusChaos** (cluster) | pod-delete + backend network faults on a live deployment → no user-facing 5xx | a deployed cluster + LitmusChaos | + +The CI gate catches resilience *regressions* deterministically on every PR; the +cluster run validates the same property end-to-end on real pods + network +(topology-bound, so a runbook, not a gate — see +[ADR-0044](../adr/ADR-0044-chaos-engineering.md)). + +## What survival looks like + +The kill matrix walks the full hot-path backend set and kills one at a time +(100 % unavailable). Each kill exercises a real degrade path: + +| killed backend | survives via | degraded response | +|----------------|--------------|-------------------| +| vector | breaker opens → fallback fuses the survivors | 200, citations from the survivors | +| keyword | breaker opens → fallback → graceful empty | 200, possibly empty | +| graph | *off the default read path* (needs caller seeds) | 200, unaffected | +| embedder | understanding's per-component catch + keyword-only fallback | 200, citations + answer | +| retrieval cache | gateway cache **degrade-open** (treats the outage as a miss) | 200, clean miss — full pipeline runs | +| reranker | gateway **rerank degrade** (falls back to retrieval-only) | 200, **no** citations (rerank also hydrates) | +| llm | the answer-generation guard (answer omitted) | 200, citations kept, **no** answer | + +The reranker + retrieval-cache guards are **new in 7.2** — chaos engineering +found those two paths could 5xx and they were hardened to degrade-open (the LLM +and embedder already degraded). See [reference/perf.md](../reference/perf.md). + +## 1. Kill-matrix gate (CI, run anywhere) + +```bash +task chaos-kill # python -m eval.gateway_chaos_v0.kill_matrix --check +# or targeted: +uv run python -m eval.gateway_chaos_v0.kill_matrix --target reranker +uv run python -m eval.gateway_chaos_v0.kill_matrix --target llm +uv run python -m eval.gateway_chaos_v0.kill_matrix # writes kill_matrix.{json,md} +pytest -m perf tests/perf/test_chaos_kill_matrix.py # the gate +``` + +It builds one in-process gateway per killed backend (httpx ASGITransport, no +socket / no cluster), drives `/v1/query`, and asserts **no 5xx, 100 % success**, +the on-path retrieval breaker **opens**, and the expected degraded shape. The +report (`eval/gateway_chaos_v0/kill_matrix.md`) is a per-backend survival table. + +The matrix seeds a tiny keyword corpus and wires the store's real `hydrate`, so +rerank / pack / generate actually run — every kill hits a real degrade path +rather than being short-circuited by an empty corpus. + +## 2. LitmusChaos at the cluster (runbook) + +The cluster-level counterpart lives in [`infra/chaos/`](../../infra/chaos/). It +kills real gateway pods and severs the gateway → backend network link, asserting +the gateway stays 200 throughout via an embedded **httpProbe**. + +```bash +# Install LitmusChaos + the generic experiments, then the least-privilege RBAC: +kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-latest.yaml +kubectl apply -n rag-platform \ + -f https://hub.litmuschaos.io/api/chaos/latest?file=charts/generic/experiments.yaml +kubectl apply -f infra/chaos/rbac.yaml + +# Run an experiment + watch the verdict: +kubectl apply -f infra/chaos/chaosengine-gateway-pod-delete.yaml +kubectl get chaosresult -n rag-platform -w +``` + +| Manifest | Fault | Acceptance (httpProbe) | +|----------|-------|------------------------| +| `chaosengine-gateway-pod-delete.yaml` | SIGKILL gateway replicas | `/healthz` stays 200 (PDB + replicas survive) | +| `chaosengine-backend-network-loss.yaml` | 100 % loss → a backend host | `/v1/query` stays 200 (breaker opens, fallback serves) | +| `chaosengine-backend-network-latency.yaml` | +300 ms RTT → a backend host | `/healthz` stays 200 (latency absorbed, no false trip) | + +Point `DESTINATION_HOSTS` at one backend at a time (qdrant / redis / postgres / +elasticsearch) to validate each in isolation. Requires `replicaCount > 1` + the +chart's PodDisruptionBudget so pod-delete has survivors. Full detail in +[`infra/chaos/README.md`](../../infra/chaos/README.md). + +### Acceptance targets + +| Metric | Target | Notes | +|--------|--------|-------| +| User-facing 5xx under any single-backend kill/fault | **0** | the property the gate + httpProbe enforce | +| On-path retrieval breaker (vector/keyword) | **open** during the fault | isolation, then auto-recovers | +| Gateway availability during pod-delete | service stays up | PDB protects quorum | +| Latency injected on a backend | absorbed, **no** breaker trip | breakers key on errors, not slowness | + +## Relationship to load testing + +Chaos engineering (7.2) is the *resilience* axis; [load testing](load-testing.md) +(7.1) is the *throughput/latency* axis. The chaos-under-load harness +(`task chaos-test`) sits between them — faults under concurrent load. All three +share `eval/gateway_chaos_v0/` and the Phase-4 breakers + fallback they validate. + +See [reference/perf.md](../reference/perf.md) and +[ADR-0044](../adr/ADR-0044-chaos-engineering.md). diff --git a/docs/reference/perf.md b/docs/reference/perf.md index 5832254..f6f2ac8 100644 --- a/docs/reference/perf.md +++ b/docs/reference/perf.md @@ -159,3 +159,40 @@ The **1000-QPS / p99 < 500 ms** acceptance is a *cluster* run (the Locust v1 sui in `eval/gateway_load_v0/locustfile.py`), documented in [guides/load-testing.md](../guides/load-testing.md) — not a CI gate. See [ADR-0043](../adr/ADR-0043-load-chaos-testing.md). + +## Chaos kill-matrix (Step 7.2) + +`eval/gateway_chaos_v0/kill_matrix.py` + `tests/perf/test_chaos_kill_matrix.py` +extend the 7.1 chaos gate from the three retrieval backends to the **full +hot-path backend set**, proving the GA bar: **no single backend failure can 5xx +the gateway**. + +```bash +task chaos-kill # kill_matrix --check +uv run python -m eval.gateway_chaos_v0.kill_matrix # writes kill_matrix.{json,md} +uv run python -m eval.gateway_chaos_v0.kill_matrix --target reranker +pytest -m perf tests/perf/test_chaos_kill_matrix.py # the gate +``` + +It kills each of `vector / keyword / graph / embedder / retrieval_cache / +reranker / llm` in turn (100 % unavailable) behind the real breakers + fallback, +drives `/v1/query`, and asserts **no 5xx, 100 % success**, the on-path retrieval +breaker **opens**, and the expected degraded shape. A seeded keyword corpus + the +store's real `hydrate` make rerank/pack/generate run, so every kill exercises a +real degrade path. + +**Two gaps were hardened** (chaos found them): the gateway now degrades-open on a +retrieval-cache outage (treats it as a miss — `gateway.cache.degraded`, on +`/v1/query` + `/v1/retrieve`) and on a reranker outage (falls back to +retrieval-only — `gateway.rerank.degraded`, honouring the `RerankPipeline`'s +documented "caller decides" contract). The LLM (answer-generation guard) and +embedder (understanding's per-component catch) already degraded. + +The **graph** backend is only consulted when the caller supplies graph seeds, so +a standard query never calls it — its kill is survivable by construction (no +breaker-open required). + +Cluster-level chaos (LitmusChaos pod-delete + backend network faults) is the +runbook in [`infra/chaos/`](../../infra/chaos/) + +[guides/chaos-engineering.md](../guides/chaos-engineering.md) — not a CI gate. +See [ADR-0044](../adr/ADR-0044-chaos-engineering.md). diff --git a/eval/gateway_chaos_v0/kill.py b/eval/gateway_chaos_v0/kill.py new file mode 100644 index 0000000..8e8f84b --- /dev/null +++ b/eval/gateway_chaos_v0/kill.py @@ -0,0 +1,117 @@ +"""Backend *kill* wrappers for the Step 7.2 chaos kill-matrix. + +Each wrapper is a drop-in for one gateway backend SPI whose hot method is +**killed** — it raises on every call, simulating a backend that is completely +unavailable (the in-process equivalent of LitmusChaos deleting that backend's +pod). Where Step 7.1's :class:`~eval.gateway_chaos_v0.faults.FaultSpec` injects +*probabilistic* faults under load, a kill is the deterministic worst case: one +backend at 100 % unavailability, the rest healthy. + +The kill-matrix harness builds one gateway per killed backend and asserts the +gateway still serves (no 5xx). Survival comes from existing machinery the kill +*validates* — the Phase-4 circuit breakers + fallback ladder for the retrieval +backends, the Step 2.6 understanding pipeline's per-component catch for the +embedder, and the Step 7.2 gateway degrade-open guards for the retrieval cache, +reranker, and LLM. + +These wrappers are **pure-raise** — they never forward to an inner backend, so +they hold no governed call site and need no PolicyEngine consultation (the +policed read path is the ``HybridRetriever`` that drives the live backends). +""" + +from __future__ import annotations + +from rag_cache import TieredRetrievalCache +from rag_core.errors import BackendError, RerankerError +from rag_core.spi.llm import LLMMessage, LLMResponse +from rag_core.spi.noop import NoopEmbedder, NoopLLM, NoopReranker +from rag_core.types import Chunk, ChunkId, ChunkRef, Embedding, RequestContext + +__all__ = [ + "KillEmbedder", + "KillReranker", + "KillLLM", + "KillRetrievalCache", +] + +_KILLED = "chaos: {backend} backend killed (100% unavailable)" + + +class KillEmbedder(NoopEmbedder): + """Embedder whose ``bulk_embed`` always raises — the embedding service is down.""" + + async def bulk_embed( + self, + ctx: RequestContext, + texts: list[str], + chunk_ids: list[ChunkId], + ) -> list[Embedding]: + raise BackendError(_KILLED.format(backend="embedder"), backend="embedder") + + +class KillReranker(NoopReranker): + """Reranker whose stage calls always raise — the rerank service is down.""" + + async def fast_rerank( + self, + ctx: RequestContext, + query: str, + candidates: list[ChunkRef], + top_n: int = 50, + ) -> list[ChunkRef]: + raise RerankerError( + _KILLED.format(backend="reranker"), stage="fast", backend="KillReranker" + ) + + async def precise_rerank( + self, + ctx: RequestContext, + query: str, + candidates: list[Chunk], + top_n: int = 10, + ) -> list[Chunk]: + raise RerankerError( + _KILLED.format(backend="reranker"), stage="precise", backend="KillReranker" + ) + + +class KillLLM(NoopLLM): + """LLM whose ``complete`` always raises — the generation model API is down.""" + + async def complete( + self, + ctx: RequestContext, + messages: list[LLMMessage], + max_tokens: int = 1024, + temperature: float = 0.0, + stop: list[str] | None = None, + ) -> LLMResponse: + raise BackendError(_KILLED.format(backend="llm"), backend="llm") + + +class KillRetrievalCache(TieredRetrievalCache): + """Retrieval cache whose read+write always raise — e.g. the cache tier is down.""" + + async def get_best( + self, + ctx: RequestContext, + *, + query_text: str, + plan_hash: str, + corpus_version: int, + scope: str = "", + ) -> list[ChunkRef] | None: + raise BackendError(_KILLED.format(backend="retrieval_cache"), backend="retrieval_cache") + + async def put_with_query( + self, + ctx: RequestContext, + *, + query_text: str, + plan_hash: str, + corpus_version: int, + value: list[ChunkRef], + scope: str = "", + ttl_seconds: int | None = None, + ) -> None: + raise BackendError(_KILLED.format(backend="retrieval_cache"), backend="retrieval_cache") diff --git a/eval/gateway_chaos_v0/kill_matrix.json b/eval/gateway_chaos_v0/kill_matrix.json new file mode 100644 index 0000000..424fb7c --- /dev/null +++ b/eval/gateway_chaos_v0/kill_matrix.json @@ -0,0 +1,117 @@ +{ + "all_survived": true, + "results": [ + { + "target": "vector", + "requests": 120, + "status_counts": { + "200": 120 + }, + "success_rate": 1.0, + "server_errors": 0, + "breaker_states": { + "vector": "open", + "keyword": "closed", + "graph": "closed" + }, + "breaker_open": true, + "survived": true + }, + { + "target": "keyword", + "requests": 120, + "status_counts": { + "200": 120 + }, + "success_rate": 1.0, + "server_errors": 0, + "breaker_states": { + "vector": "closed", + "keyword": "open", + "graph": "closed" + }, + "breaker_open": true, + "survived": true + }, + { + "target": "graph", + "requests": 120, + "status_counts": { + "200": 120 + }, + "success_rate": 1.0, + "server_errors": 0, + "breaker_states": { + "vector": "closed", + "keyword": "closed", + "graph": "closed" + }, + "breaker_open": null, + "survived": true + }, + { + "target": "embedder", + "requests": 120, + "status_counts": { + "200": 120 + }, + "success_rate": 1.0, + "server_errors": 0, + "breaker_states": { + "vector": "closed", + "keyword": "closed", + "graph": "closed" + }, + "breaker_open": null, + "survived": true + }, + { + "target": "retrieval_cache", + "requests": 120, + "status_counts": { + "200": 120 + }, + "success_rate": 1.0, + "server_errors": 0, + "breaker_states": { + "vector": "closed", + "keyword": "closed", + "graph": "closed" + }, + "breaker_open": null, + "survived": true + }, + { + "target": "reranker", + "requests": 120, + "status_counts": { + "200": 120 + }, + "success_rate": 1.0, + "server_errors": 0, + "breaker_states": { + "vector": "closed", + "keyword": "closed", + "graph": "closed" + }, + "breaker_open": null, + "survived": true + }, + { + "target": "llm", + "requests": 120, + "status_counts": { + "200": 120 + }, + "success_rate": 1.0, + "server_errors": 0, + "breaker_states": { + "vector": "closed", + "keyword": "closed", + "graph": "closed" + }, + "breaker_open": null, + "survived": true + } + ] +} diff --git a/eval/gateway_chaos_v0/kill_matrix.md b/eval/gateway_chaos_v0/kill_matrix.md new file mode 100644 index 0000000..2c71e35 --- /dev/null +++ b/eval/gateway_chaos_v0/kill_matrix.md @@ -0,0 +1,17 @@ +# Gateway chaos kill-matrix — harness report + +Generated by `python -m eval.gateway_chaos_v0.kill_matrix`. Each row kills one backend (100 % unavailable) behind the real Phase-4 breakers + fallback ladder and the Step 7.2 gateway degrade-open guards, then drives `/v1/query` and checks the gateway never 5xx-es. + +**Verdict: PASS** — no single backend kill 5xx-es the gateway. + +| killed backend | requests | 2xx success | 5xx | breaker | survived | +|----------------|---------:|------------:|----:|---------|:--------:| +| vector | 120 | 1.0000 | 0 | open | yes | +| keyword | 120 | 1.0000 | 0 | open | yes | +| graph | 120 | 1.0000 | 0 | closed | yes | +| embedder | 120 | 1.0000 | 0 | n/a | yes | +| retrieval_cache | 120 | 1.0000 | 0 | n/a | yes | +| reranker | 120 | 1.0000 | 0 | n/a | yes | +| llm | 120 | 1.0000 | 0 | n/a | yes | + +**Survived** = no 5xx and success ≥ floor — and, for an *on-path* retrieval backend (vector, keyword), its circuit breaker tripped `open` (the gateway isolated the dead backend and kept serving). The **graph** backend is only consulted when the caller supplies graph seeds, so a standard query never calls it — its breaker stays `closed` and the kill is survivable by construction. diff --git a/eval/gateway_chaos_v0/kill_matrix.py b/eval/gateway_chaos_v0/kill_matrix.py new file mode 100644 index 0000000..1616618 --- /dev/null +++ b/eval/gateway_chaos_v0/kill_matrix.py @@ -0,0 +1,471 @@ +"""Step 7.2 — chaos kill-matrix harness. + +Kills **each** gateway backend in turn (100 % unavailable, one at a time) and +verifies the gateway still serves — no 5xx — proving the GA resilience bar: +*no single backend failure can 5xx the gateway*. + +Where Step 7.1 fault-injected only the three retrieval backends under load, 7.2 +walks the full hot-path backend set and asserts graceful degradation for each: + +| killed backend | survives via | +|-----------------|-------------------------------------------------------------| +| vector | breaker opens → fallback fuses the surviving sources | +| keyword | breaker opens → fallback → graceful empty answer | +| graph | breaker opens → fallback fuses the surviving sources | +| embedder | understanding's per-component catch + keyword-only fallback | +| retrieval_cache | Step 7.2 gateway cache degrade-open (treats outage as a miss)| +| reranker | Step 7.2 gateway rerank degrade (falls back to retrieval-only)| +| llm | the answer-generation guard (answer omitted, chunks survive) | + +The kills reuse Step 7.1's ``Chaos*`` retrieval wrappers (``failure_rate=1.0``) +for the retrieval backends and the Step 7.2 ``Kill*`` wrappers for the rest, all +behind the **real** Phase-4 breakers + fallback ladder — the harness *validates* +that machinery, it builds none. A tiny seeded keyword corpus + the store's real +``hydrate`` make rerank / pack / generate actually run, so every kill exercises a +genuine degrade path (an empty corpus would short-circuit them). + +Run:: + + uv run python -m eval.gateway_chaos_v0.kill_matrix + uv run python -m eval.gateway_chaos_v0.kill_matrix --target reranker + uv run python -m eval.gateway_chaos_v0.kill_matrix --check +""" + +from __future__ import annotations + +import argparse +import asyncio +import json +import sys +import time +from collections import Counter +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +import httpx +from fastapi import FastAPI +from rag_breaker import ( + GRAPH_BACKEND, + KEYWORD_BACKEND, + VECTOR_BACKEND, + BreakerGraphRetrievalBackend, + BreakerKeywordRetrievalBackend, + BreakerRegistry, + BreakerVectorRetrievalBackend, +) +from rag_cache import TieredRetrievalCache +from rag_core.spi.noop import ( + NoopEmbedder, + NoopGraphStore, + NoopKeywordStore, + NoopLLM, + NoopReranker, + NoopVectorStore, +) +from rag_core.types import ( + Chunk, + ChunkId, + CorpusId, + DocumentId, + Principal, + PrincipalId, + PrincipalKind, + RequestContext, + TenantId, +) +from rag_gateway.app import build_app +from rag_query.hyde import HyDEGenerator +from rag_query.pipeline import QueryUnderstandingPipeline +from rag_reranker.pipeline import RerankPipeline +from rag_retrieval import HybridRetriever, RetrievalRouter + +from eval.gateway_chaos_v0.faults import ( + ChaosGraphRetrievalBackend, + ChaosKeywordRetrievalBackend, + ChaosVectorRetrievalBackend, + FaultSpec, +) +from eval.gateway_chaos_v0.kill import ( + KillEmbedder, + KillLLM, + KillReranker, + KillRetrievalCache, +) + +HERE = Path(__file__).parent +REPORT_JSON = HERE / "kill_matrix.json" +REPORT_MD = HERE / "kill_matrix.md" + +_TENANT = "chaos-probe" +_SEED_TERM = "platform reliability degradation" + +# The retrieval backends sit behind circuit breakers. Vector + keyword are on +# the *default* read path, so killing one must trip its breaker (proving the +# breaker isolated the dead backend). Graph is only consulted when the caller +# supplies graph seeds (RetrievalRouter.decide: ``graph_input_available``), so a +# standard seed-less query never calls it — killing it cannot affect that query, +# and its breaker legitimately stays closed. +RETRIEVAL_TARGETS: tuple[str, ...] = ("vector", "keyword", "graph") +BREAKER_EXERCISED: tuple[str, ...] = ("vector", "keyword") +KILL_TARGETS: tuple[str, ...] = ( + *RETRIEVAL_TARGETS, + "embedder", + "retrieval_cache", + "reranker", + "llm", +) + +# Acceptance: every request must succeed (no 5xx) under a single-backend kill. +DEFAULT_SUCCESS_FLOOR = 1.0 + + +# --------------------------------------------------------------------------- +# Seeded corpus — gives the retrieve → rerank → pack → generate path real input +# --------------------------------------------------------------------------- +def _seed_ctx() -> RequestContext: + tid = TenantId(_TENANT) + return RequestContext( + tenant_id=tid, + principal=Principal( + id=PrincipalId("chaos-seed"), + kind=PrincipalKind.service, + display_name="chaos-seed", + tenant_id=tid, + ), + ) + + +async def _seed_keyword(store: NoopKeywordStore, *, n: int = 12) -> None: + """Index a tiny corpus so queries return refs and the full pipeline runs.""" + ctx = _seed_ctx() + chunks = [ + Chunk( + id=ChunkId(f"chaos-doc-{i}"), + document_id=DocumentId(f"chaos-doc-{i}"), + tenant_id=TenantId(_TENANT), + corpus_id=CorpusId("chaos"), + content=( + f"{_SEED_TERM} passage {i}. The gateway keeps serving and degrades " + "gracefully when a backend fails under load." + ), + position=i, + ) + for i in range(n) + ] + await store.bulk_index(ctx, chunks) + + +# --------------------------------------------------------------------------- +# Per-scenario wiring — kill exactly one backend, keep the rest healthy +# --------------------------------------------------------------------------- +def _wire(target: str, keyword_store: NoopKeywordStore) -> tuple[FastAPI, BreakerRegistry]: + """Build a gateway with ``target`` killed and every other backend healthy.""" + registry = BreakerRegistry() + embedder: Any = KillEmbedder() if target == "embedder" else NoopEmbedder(dimension=8) + + vector_inner: Any = NoopVectorStore() + keyword_inner: Any = keyword_store + graph_inner: Any = NoopGraphStore() + if target == "vector": + vector_inner = ChaosVectorRetrievalBackend(NoopVectorStore(), FaultSpec(failure_rate=1.0)) + elif target == "keyword": + keyword_inner = ChaosKeywordRetrievalBackend(keyword_store, FaultSpec(failure_rate=1.0)) + elif target == "graph": + graph_inner = ChaosGraphRetrievalBackend(NoopGraphStore(), FaultSpec(failure_rate=1.0)) + + vector = BreakerVectorRetrievalBackend(vector_inner, registry.get(VECTOR_BACKEND)) + keyword = BreakerKeywordRetrievalBackend(keyword_inner, registry.get(KEYWORD_BACKEND)) + graph = BreakerGraphRetrievalBackend(graph_inner, registry.get(GRAPH_BACKEND)) + hybrid = HybridRetriever(vector_backend=vector, keyword_backend=keyword, graph_backend=graph) + router = RetrievalRouter(hybrid=hybrid, embedder=embedder) + + understanding = QueryUnderstandingPipeline( + hyde=HyDEGenerator(llm=NoopLLM(), embedder=embedder, n=1) + ) + + reranker_spi: Any = KillReranker() if target == "reranker" else NoopReranker() + # The real hydrator (the seeded store) turns refs into chunks, so the rerank + # and downstream pack/generate stages actually run. + reranker = RerankPipeline(reranker=reranker_spi, hydrator=keyword_store.hydrate) + + llm: Any = KillLLM() if target == "llm" else NoopLLM() + cache: Any = KillRetrievalCache() if target == "retrieval_cache" else TieredRetrievalCache() + + app = build_app( + understanding=understanding, + retrieval_router=router, + reranker=reranker, + llm=llm, + retrieval_cache=cache, + breaker_registry=registry, + fallback_enabled=True, + ) + return app, registry + + +# --------------------------------------------------------------------------- +# Driver +# --------------------------------------------------------------------------- +def _body(idx: int) -> dict[str, Any]: + # A stable seed term drives retrieval; the nonce busts the retrieval cache so + # every request re-runs the pipeline (and the killed backend is exercised). + return { + "tenant_id": _TENANT, + "principal_id": "chaos-probe", + "query": f"{_SEED_TERM} probe {idx}", + "rerank": True, + "pack": True, + "generate": True, + } + + +async def _fire(client: httpx.AsyncClient, idx: int) -> int: + """Fire one /v1/query; return the status code (599 on transport error).""" + try: + resp = await client.post("/v1/query", json=_body(idx)) + return resp.status_code + except Exception: # noqa: BLE001 - a transport error counts as a server error + return 599 + + +async def _drive( + app: FastAPI, + registry: BreakerRegistry, + *, + requests: int, + concurrency: int, + warmup: int, +) -> tuple[dict[str, int], dict[str, str]]: + transport = httpx.ASGITransport(app=app) + statuses: Counter[int] = Counter() + async with httpx.AsyncClient(transport=transport, base_url="http://gateway") as client: + for i in range(warmup): + await _fire(client, -1 - i) + sem = asyncio.Semaphore(concurrency) + + async def _task(idx: int) -> None: + async with sem: + status = await _fire(client, idx) + statuses[status] += 1 + + await asyncio.gather(*[_task(i) for i in range(requests)]) + names = (VECTOR_BACKEND, KEYWORD_BACKEND, GRAPH_BACKEND) + breakers = {name: (await registry.get(name).snapshot()).state.value for name in names} + return {str(k): v for k, v in sorted(statuses.items())}, breakers + + +@dataclass +class KillResult: + """Outcome of killing one backend.""" + + target: str + requests: int + status_counts: dict[str, int] + success_rate: float + server_errors: int + breaker_states: dict[str, str] + success_floor: float + + @property + def is_retrieval(self) -> bool: + return self.target in RETRIEVAL_TARGETS + + @property + def breaker_required(self) -> bool: + """Whether this kill must trip a breaker (on-path retrieval backend).""" + return self.target in BREAKER_EXERCISED + + @property + def breaker_open(self) -> bool: + """Did the killed backend's own breaker trip open?""" + return self.breaker_states.get(self.target) == "open" + + @property + def survived(self) -> bool: + """No 5xx and success at/above the floor — and, for an on-path retrieval + backend, the killed backend's breaker opened (isolation, not luck).""" + ok = self.server_errors == 0 and self.success_rate >= self.success_floor + return ok and (self.breaker_open if self.breaker_required else True) + + def as_dict(self) -> dict[str, Any]: + return { + "target": self.target, + "requests": self.requests, + "status_counts": self.status_counts, + "success_rate": round(self.success_rate, 4), + "server_errors": self.server_errors, + "breaker_states": self.breaker_states, + "breaker_open": self.breaker_open if self.breaker_required else None, + "survived": self.survived, + } + + +@dataclass +class KillMatrix: + """The full kill matrix across every backend.""" + + results: list[KillResult] = field(default_factory=list) + + @property + def all_survived(self) -> bool: + return all(r.survived for r in self.results) + + def as_dict(self) -> dict[str, Any]: + return { + "all_survived": self.all_survived, + "results": [r.as_dict() for r in self.results], + } + + +async def _run_target( + target: str, *, requests: int, concurrency: int, warmup: int, success_floor: float +) -> KillResult: + keyword_store = NoopKeywordStore() + await _seed_keyword(keyword_store) + app, registry = _wire(target, keyword_store) + status_counts, breaker_states = await _drive( + app, registry, requests=requests, concurrency=concurrency, warmup=warmup + ) + total = sum(status_counts.values()) + ok = sum(v for k, v in status_counts.items() if int(k) < 400) + server_errors = sum(v for k, v in status_counts.items() if int(k) >= 500) + return KillResult( + target=target, + requests=requests, + status_counts=status_counts, + success_rate=(ok / total) if total else 0.0, + server_errors=server_errors, + breaker_states=breaker_states, + success_floor=success_floor, + ) + + +def run_kill_matrix( + *, + targets: tuple[str, ...] = KILL_TARGETS, + requests: int = 120, + concurrency: int = 16, + warmup: int = 10, + success_floor: float = DEFAULT_SUCCESS_FLOOR, +) -> KillMatrix: + """Kill each backend in ``targets`` in turn and summarise survival.""" + + async def _all() -> list[KillResult]: + out: list[KillResult] = [] + for target in targets: + out.append( + await _run_target( + target, + requests=requests, + concurrency=concurrency, + warmup=warmup, + success_floor=success_floor, + ) + ) + return out + + return KillMatrix(results=asyncio.run(_all())) + + +def run_one( + target: str, *, requests: int = 120, concurrency: int = 16, warmup: int = 10 +) -> KillResult: + """Kill a single backend (used by the perf gate for targeted assertions).""" + return asyncio.run( + _run_target( + target, + requests=requests, + concurrency=concurrency, + warmup=warmup, + success_floor=DEFAULT_SUCCESS_FLOOR, + ) + ) + + +# --------------------------------------------------------------------------- +# Report + CLI +# --------------------------------------------------------------------------- +def _write_report_md(path: Path, matrix: KillMatrix) -> None: + verdict = "PASS" if matrix.all_survived else "FAIL" + lines = [ + "# Gateway chaos kill-matrix — harness report", + "", + "Generated by `python -m eval.gateway_chaos_v0.kill_matrix`. Each row kills one " + "backend (100 % unavailable) behind the real Phase-4 breakers + fallback ladder and " + "the Step 7.2 gateway degrade-open guards, then drives `/v1/query` and checks the " + "gateway never 5xx-es.", + "", + f"**Verdict: {verdict}** — no single backend kill 5xx-es the gateway." + if matrix.all_survived + else f"**Verdict: {verdict}** — a backend kill produced 5xx (see below).", + "", + "| killed backend | requests | 2xx success | 5xx | breaker | survived |", + "|----------------|---------:|------------:|----:|---------|:--------:|", + ] + for r in matrix.results: + breaker = r.breaker_states.get(r.target, "—") if r.is_retrieval else "n/a" + lines.append( + f"| {r.target} | {r.requests} | {r.success_rate:.4f} | {r.server_errors} | " + f"{breaker} | {'yes' if r.survived else 'NO'} |" + ) + lines += [ + "", + "**Survived** = no 5xx and success ≥ floor — and, for an *on-path* retrieval backend " + "(vector, keyword), its circuit breaker tripped `open` (the gateway isolated the dead " + "backend and kept serving). The **graph** backend is only consulted when the caller " + "supplies graph seeds, so a standard query never calls it — its breaker stays `closed` " + "and the kill is survivable by construction.", + "", + ] + path.write_text("\n".join(lines), encoding="utf-8") + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--requests", type=int, default=120) + parser.add_argument("--concurrency", type=int, default=16) + parser.add_argument("--warmup", type=int, default=10) + parser.add_argument( + "--target", + choices=KILL_TARGETS, + help="Kill only this backend (default: the full matrix).", + ) + parser.add_argument("--no-write", action="store_true") + parser.add_argument( + "--check", action="store_true", help="Exit 1 if any backend kill 5xx-ed the gateway." + ) + args = parser.parse_args(argv) + + targets = (args.target,) if args.target else KILL_TARGETS + start = time.perf_counter() + matrix = run_kill_matrix( + targets=targets, + requests=args.requests, + concurrency=args.concurrency, + warmup=args.warmup, + ) + elapsed = time.perf_counter() - start + + if not args.no_write and not args.target: + REPORT_JSON.write_text(json.dumps(matrix.as_dict(), indent=2) + "\n", encoding="utf-8") + _write_report_md(REPORT_MD, matrix) + print(f"wrote {REPORT_JSON}") + print(f"wrote {REPORT_MD}") + + for r in matrix.results: + breaker = f" breaker={r.breaker_states.get(r.target)}" if r.is_retrieval else "" + print( + f"kill {r.target:<16} success_rate={r.success_rate:.4f} " + f"5xx={r.server_errors}{breaker} -> {'survived' if r.survived else 'FAILED'}" + ) + verdict = "PASS" if matrix.all_survived else "FAIL" + print(f"summary: killed {len(matrix.results)} backends in {elapsed:.1f}s -> {verdict}") + + if args.check and not matrix.all_survived: + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/infra/chaos/README.md b/infra/chaos/README.md new file mode 100644 index 0000000..62a8f54 --- /dev/null +++ b/infra/chaos/README.md @@ -0,0 +1,84 @@ +# Cluster chaos engineering (Step 7.2) + +LitmusChaos manifests that validate the GA resilience bar — **no single backend +failure can 5xx the gateway, and the fallback chain holds** — against a *real* +deployed cluster. + +These are the cluster-level counterpart to the deterministic in-process +**kill-matrix gate** (`eval/gateway_chaos_v0/kill_matrix.py` + +`tests/perf/test_chaos_kill_matrix.py`). The in-process gate proves resilience +on every PR with no infrastructure; these experiments prove it end-to-end on a +live cluster (a real network, real pods, real backends) — hardware/topology +bound, so a runbook, not a CI gate (same split as the Step 7.1 load runbook; +see [ADR-0044](../../docs/adr/ADR-0044-chaos-engineering.md)). + +## What each experiment proves + +| File | Fault | Asserted (via embedded httpProbe) | +|------|-------|-----------------------------------| +| `chaosengine-gateway-pod-delete.yaml` | SIGKILL gateway replicas | `/healthz` stays **200** — the PDB + multiple replicas survive instance loss | +| `chaosengine-backend-network-loss.yaml` | 100 % packet loss → a backend host | `/v1/query` stays **200** — the breaker opens, the fallback ladder serves on | +| `chaosengine-backend-network-latency.yaml` | +300 ms RTT → a backend host | `/healthz` stays **200** — latency is absorbed, no breaker trip on slowness | + +The **httpProbe** in each `ChaosEngine` is the acceptance check: LitmusChaos runs +it continuously for the chaos duration and **fails the experiment** if the +gateway ever returns a non-200. That is the cluster-level "no 5xx under chaos". + +## Prerequisites + +```bash +# 1. Install LitmusChaos (operator + CRDs) — see https://litmuschaos.io +kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-latest.yaml + +# 2. Install the generic experiments into the rag-platform namespace +kubectl apply -n rag-platform \ + -f https://hub.litmuschaos.io/api/chaos/latest?file=charts/generic/experiments.yaml + +# 3. Grant the chaos runner least-privilege RBAC (this directory) +kubectl apply -f infra/chaos/rbac.yaml +``` + +The gateway must run with **`replicaCount > 1`** (so pod-delete has survivors) +and the **PodDisruptionBudget** from the chart (`pdb.yaml`). Set +`DESTINATION_HOSTS` in the network experiments to the backend under test — +`rag-qdrant…`, `rag-redis…`, the Postgres/Elasticsearch service host, etc. — and +repeat per backend to validate each in isolation (or comma-separate for the +all-down worst case). + +## Run + observe + +```bash +kubectl apply -f infra/chaos/chaosengine-gateway-pod-delete.yaml +# Watch the verdict (Awaited → Pass/Fail): +kubectl get chaosresult -n rag-platform -w +kubectl describe chaosresult rag-gateway-pod-delete-pod-delete -n rag-platform +``` + +A **Pass** verdict means the probe held 200 throughout — the gateway degraded +gracefully. While an experiment runs, confirm the breaker tripped on the +severed backend: + +```bash +kubectl exec deploy/rag-platform -n rag-platform -- \ + curl -s localhost:8000/v1/status/breakers # the failing backend → "open" +``` + +## Acceptance targets + +| Metric | Target | +|--------|--------| +| User-facing 5xx during any single-backend fault | **0** (the httpProbe stays 200) | +| Circuit breaker on the severed backend | **open** during the fault, recovers after | +| Gateway availability during pod-delete | service stays up (PDB protects quorum) | +| Recovery after the experiment | breakers re-close, killed pods rescheduled | + +## Teardown + +```bash +kubectl delete -f infra/chaos/chaosengine-gateway-pod-delete.yaml +kubectl delete -f infra/chaos/chaosengine-backend-network-loss.yaml +kubectl delete -f infra/chaos/chaosengine-backend-network-latency.yaml +``` + +See [docs/guides/chaos-engineering.md](../../docs/guides/chaos-engineering.md) +for the full runbook and [reference/perf.md](../../docs/reference/perf.md). diff --git a/infra/chaos/chaosengine-backend-network-latency.yaml b/infra/chaos/chaosengine-backend-network-latency.yaml new file mode 100644 index 0000000..fb54ea6 --- /dev/null +++ b/infra/chaos/chaosengine-backend-network-latency.yaml @@ -0,0 +1,61 @@ +# LitmusChaos — backend network-latency (Step 7.2). +# +# Latency is not failure: a slow backend must NOT trip the breaker (which keys +# on errors, not slowness — see ADR-0043's deferred "latency-based tripping"). +# This experiment injects a large RTT on the gateway → backend link and asserts +# the gateway absorbs it (requests still 200, just slower) rather than erroring +# — the cluster analog of the in-process latency-injection test (Step 7.1). +# +# Prereqs: LitmusChaos + the `pod-network-latency` experiment installed; +# rbac.yaml applied. Run: kubectl apply -f chaosengine-backend-network-latency.yaml +apiVersion: litmuschaos.io/v1alpha1 +kind: ChaosEngine +metadata: + name: rag-backend-network-latency + namespace: rag-platform + labels: + app.kubernetes.io/part-of: rag-platform + app.kubernetes.io/component: chaos +spec: + appinfo: + appns: rag-platform + applabel: "app.kubernetes.io/name=rag-platform" + appkind: deployment + chaosServiceAccount: rag-chaos-runner + engineState: active + annotationCheck: "false" + experiments: + - name: pod-network-latency + spec: + components: + env: + - name: TOTAL_CHAOS_DURATION + value: "60" + - name: NETWORK_LATENCY + value: "300" # ms of injected RTT toward the backend + - name: DESTINATION_HOSTS + value: "rag-qdrant.rag-platform.svc.cluster.local" + - name: CONTAINER_RUNTIME + value: "containerd" + - name: SOCKET_PATH + value: "/run/containerd/containerd.sock" + - name: PODS_AFFECTED_PERC + value: "100" + probe: + # Acceptance: latency is absorbed — requests still succeed (no errors, + # no breaker trip on slowness). Timeout is generous to allow the RTT. + - name: query-absorbs-latency + type: httpProbe + mode: Continuous + runProperties: + probeTimeout: 5s + interval: 3s + attempt: 1 + probePollingInterval: 2s + httpProbe/inputs: + url: http://rag-platform.rag-platform.svc.cluster.local/healthz + insecureSkipVerify: false + method: + get: + criteria: == + responseCode: "200" diff --git a/infra/chaos/chaosengine-backend-network-loss.yaml b/infra/chaos/chaosengine-backend-network-loss.yaml new file mode 100644 index 0000000..6e85748 --- /dev/null +++ b/infra/chaos/chaosengine-backend-network-loss.yaml @@ -0,0 +1,72 @@ +# LitmusChaos — backend network-loss (Step 7.2). +# +# The backends (pgvector / qdrant / redis / elasticsearch) are reached over the +# network by host, so the cluster-level way to "kill a backend" from the +# gateway's point of view is to sever the gateway → backend link: 100 % packet +# loss toward that backend's host. The gateway's Phase-4 circuit breaker then +# opens on the unreachable backend and the fallback ladder + the Step 7.2 +# degrade-open guards keep it serving — exactly what the in-process kill-matrix +# proves deterministically, now validated end-to-end on a real cluster. +# +# Point DESTINATION_HOSTS at ONE backend to validate that backend's failure in +# isolation (repeat per backend). The embedded httpProbe drives a real +# /v1/query and asserts it stays 200 throughout — no user-facing 5xx. +# +# Prereqs: LitmusChaos + the `pod-network-loss` experiment installed; rbac.yaml +# applied. Run: kubectl apply -f chaosengine-backend-network-loss.yaml +apiVersion: litmuschaos.io/v1alpha1 +kind: ChaosEngine +metadata: + name: rag-backend-network-loss + namespace: rag-platform + labels: + app.kubernetes.io/part-of: rag-platform + app.kubernetes.io/component: chaos +spec: + appinfo: + appns: rag-platform + applabel: "app.kubernetes.io/name=rag-platform" + appkind: deployment + chaosServiceAccount: rag-chaos-runner + engineState: active + annotationCheck: "false" + experiments: + - name: pod-network-loss + spec: + components: + env: + - name: TOTAL_CHAOS_DURATION + value: "60" + - name: NETWORK_PACKET_LOSS_PERCENTAGE + value: "100" + # Sever the gateway → backend link. Set to the backend under test, + # e.g. the qdrant / redis / postgres / elasticsearch service host. + # Comma-separate to take down several at once (worst case). + - name: DESTINATION_HOSTS + value: "rag-qdrant.rag-platform.svc.cluster.local" + - name: CONTAINER_RUNTIME + value: "containerd" + - name: SOCKET_PATH + value: "/run/containerd/containerd.sock" + - name: PODS_AFFECTED_PERC + value: "100" + probe: + # Acceptance: a live /v1/query keeps returning 200 while the backend is + # unreachable — the breaker opens and the fallback ladder serves on. + - name: query-stays-200-under-backend-loss + type: httpProbe + mode: Continuous + runProperties: + probeTimeout: 3s + interval: 3s + attempt: 1 + probePollingInterval: 2s + httpProbe/inputs: + url: http://rag-platform.rag-platform.svc.cluster.local/v1/query + insecureSkipVerify: false + method: + post: + contentType: application/json + body: '{"tenant_id":"chaos","principal_id":"chaos","query":"resilience probe","rerank":true,"pack":true,"generate":false}' + criteria: == + responseCode: "200" diff --git a/infra/chaos/chaosengine-gateway-pod-delete.yaml b/infra/chaos/chaosengine-gateway-pod-delete.yaml new file mode 100644 index 0000000..eaabff9 --- /dev/null +++ b/infra/chaos/chaosengine-gateway-pod-delete.yaml @@ -0,0 +1,59 @@ +# LitmusChaos — gateway pod-delete (Step 7.2). +# +# Kills gateway replicas during steady traffic to prove the service survives +# instance loss: with replicaCount > 1 + the PodDisruptionBudget (pdb.yaml), +# the Service keeps routing to the survivors while Kubernetes reschedules the +# killed pod. This is the cluster analog of the in-process kill-matrix — the +# embedded httpProbe is the acceptance: /healthz must stay 200 *throughout*. +# +# Prereqs: LitmusChaos installed; the `pod-delete` experiment present in the +# namespace (kubectl apply -f https://hub.litmuschaos.io/api/chaos/?file=charts/generic/pod-delete/experiment.yaml); +# rbac.yaml applied. Run: kubectl apply -f chaosengine-gateway-pod-delete.yaml +apiVersion: litmuschaos.io/v1alpha1 +kind: ChaosEngine +metadata: + name: rag-gateway-pod-delete + namespace: rag-platform + labels: + app.kubernetes.io/part-of: rag-platform + app.kubernetes.io/component: chaos +spec: + appinfo: + appns: rag-platform + applabel: "app.kubernetes.io/name=rag-platform" + appkind: deployment + chaosServiceAccount: rag-chaos-runner + engineState: active + annotationCheck: "false" + experiments: + - name: pod-delete + spec: + components: + env: + - name: TOTAL_CHAOS_DURATION + value: "60" + - name: CHAOS_INTERVAL + value: "10" + # Hard kill (SIGKILL) — no graceful drain, the worst case. + - name: FORCE + value: "true" + # Kill half the replicas each interval; the PDB protects quorum. + - name: PODS_AFFECTED_PERC + value: "50" + probe: + # Acceptance: the gateway never serves a non-200 while replicas churn. + - name: gateway-healthz-stays-200 + type: httpProbe + mode: Continuous + runProperties: + probeTimeout: 2s + interval: 2s + attempt: 1 + probePollingInterval: 1s + httpProbe/inputs: + url: http://rag-platform.rag-platform.svc.cluster.local/healthz + insecureSkipVerify: false + method: + get: + criteria: == + responseCode: "200" diff --git a/infra/chaos/rbac.yaml b/infra/chaos/rbac.yaml new file mode 100644 index 0000000..dc20a9f --- /dev/null +++ b/infra/chaos/rbac.yaml @@ -0,0 +1,56 @@ +# RBAC for the LitmusChaos experiments in this directory (Step 7.2). +# +# The chaos-runner ServiceAccount needs permission to enumerate + delete the +# gateway pods (pod-delete) and to run the network-fault helper pods +# (pod-network-loss / pod-network-latency), plus to read/write the Litmus +# ChaosEngine / ChaosExperiment / ChaosResult CRDs and emit events. +# +# Namespaced (least privilege): it can only act within the rag-platform +# namespace. Adjust the namespace to match your release. +apiVersion: v1 +kind: ServiceAccount +metadata: + name: rag-chaos-runner + namespace: rag-platform + labels: + app.kubernetes.io/part-of: rag-platform + app.kubernetes.io/component: chaos +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: rag-chaos-runner + namespace: rag-platform + labels: + app.kubernetes.io/part-of: rag-platform + app.kubernetes.io/component: chaos +rules: + # Litmus bookkeeping: track experiment jobs + their state. + - apiGroups: ["", "batch", "litmuschaos.io"] + resources: ["jobs", "configmaps", "secrets", "events", "chaosengines", "chaosexperiments", "chaosresults"] + verbs: ["create", "list", "get", "patch", "update", "delete", "deletecollection"] + # The blast radius: enumerate target pods, read their logs, and (pod-delete) + # delete them / (network experiments) exec the fault into them. + - apiGroups: [""] + resources: ["pods", "pods/log", "pods/exec", "replicationcontrollers"] + verbs: ["create", "list", "get", "patch", "update", "delete"] + - apiGroups: ["apps"] + resources: ["deployments", "replicasets"] + verbs: ["list", "get"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: rag-chaos-runner + namespace: rag-platform + labels: + app.kubernetes.io/part-of: rag-platform + app.kubernetes.io/component: chaos +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: rag-chaos-runner +subjects: + - kind: ServiceAccount + name: rag-chaos-runner + namespace: rag-platform diff --git a/tests/perf/test_chaos_kill_matrix.py b/tests/perf/test_chaos_kill_matrix.py new file mode 100644 index 0000000..28ceb0b --- /dev/null +++ b/tests/perf/test_chaos_kill_matrix.py @@ -0,0 +1,98 @@ +"""Gateway chaos kill-matrix gate (Step 7.2). + +The gate: kill **each** backend in turn (100 % unavailable) and assert the +gateway still serves — *no single backend failure can 5xx the gateway*. This +extends the Step 7.1 chaos-under-load gate (which fault-injected only the three +retrieval backends) to the full hot-path backend set, validating the Phase-4 +breakers + fallback ladder *and* the Step 7.2 gateway degrade-open guards +(retrieval cache, reranker) + the answer-generation guard. + +``perf``-marked (runs in the dedicated perf CI job, not the cross-OS unit +sweep). The assertions are **timing-independent** — no 5xx, 100 % success, the +on-path retrieval breaker opens, and the expected degraded shape — so they are +deterministic regardless of CI-runner speed. + +See [docs/guides/chaos-engineering.md](../../docs/guides/chaos-engineering.md) +and [ADR-0044](../../docs/adr/ADR-0044-chaos-engineering.md). +""" + +from __future__ import annotations + +import asyncio + +import httpx +import pytest +from rag_core.spi.noop import NoopKeywordStore + +from eval.gateway_chaos_v0.kill_matrix import ( + BREAKER_EXERCISED, + KILL_TARGETS, + _body, + _seed_keyword, + _wire, + run_one, +) + +pytestmark = pytest.mark.perf + +_REQUESTS = 80 +_CONCURRENCY = 8 + + +@pytest.mark.parametrize("target", KILL_TARGETS) +def test_killing_each_backend_never_5xxes(target: str) -> None: + # Kill exactly one backend (100% unavailable); the gateway must keep serving. + result = run_one(target, requests=_REQUESTS, concurrency=_CONCURRENCY) + assert result.server_errors == 0, (target, result.status_counts) + assert result.success_rate == 1.0, (target, result.status_counts) + assert result.survived, (target, result.as_dict()) + + +@pytest.mark.parametrize("target", BREAKER_EXERCISED) +def test_on_path_retrieval_kill_trips_its_breaker(target: str) -> None: + # Killing an on-path retrieval backend must trip its circuit breaker open — + # the fallback then fuses the survivors (or returns a graceful empty answer). + result = run_one(target, requests=_REQUESTS, concurrency=_CONCURRENCY) + assert result.breaker_states[target] == "open", result.breaker_states + + +async def _probe(target: str) -> dict[str, object]: + keyword_store = NoopKeywordStore() + await _seed_keyword(keyword_store) + app, _registry = _wire(target, keyword_store) + transport = httpx.ASGITransport(app=app) + async with httpx.AsyncClient(transport=transport, base_url="http://gw") as client: + resp = await client.post("/v1/query", json=_body(1)) + return {"status": resp.status_code, "json": resp.json()} + + +def test_reranker_kill_degrades_to_retrieval_only() -> None: + # A killed reranker (which also hydrates) leaves no hydrated chunks: the + # gateway returns 200 with no citations rather than 5xx (retrieval-only). + out = asyncio.run(_probe("reranker")) + assert out["status"] == 200 + body = out["json"] + assert isinstance(body, dict) + assert body["citations"] == [] + + +def test_llm_kill_keeps_citations_but_omits_the_answer() -> None: + # A killed generation LLM must not lose the retrieved evidence: the gateway + # returns 200 with citations intact and the answer omitted. + out = asyncio.run(_probe("llm")) + assert out["status"] == 200 + body = out["json"] + assert isinstance(body, dict) + assert len(body["citations"]) > 0 + assert body["answer"] is None + + +def test_cache_kill_degrades_to_a_clean_miss() -> None: + # A killed retrieval cache is a non-authoritative miss: the full pipeline + # still runs (citations + answer present), the request never 5xx-es. + out = asyncio.run(_probe("retrieval_cache")) + assert out["status"] == 200 + body = out["json"] + assert isinstance(body, dict) + assert len(body["citations"]) > 0 + assert body["answer"] is not None diff --git a/tests/policy/coverage.py b/tests/policy/coverage.py index 77d6e58..f6ce3ed 100644 --- a/tests/policy/coverage.py +++ b/tests/policy/coverage.py @@ -133,6 +133,14 @@ # SPI wrappers above. They read no chunk themselves and add no policy # bypass — the policed read path is the HybridRetriever that drives them. Path("eval/gateway_chaos_v0/faults.py"), + # Step 7.2 — the chaos kill-matrix harness seeds a tiny synthetic corpus + # (``keyword_store.bulk_index``) into the in-memory noop stores so the + # rerank / generate stages actually run under a backend kill. Same + # benchmarking-tooling rationale as the golden_set_v0 / agent_loop_v0 + # harnesses above: not a production ingest path; the gateway/ingest + # pipeline remains the policy boundary. The Kill* wrappers themselves are + # pure-raise (no governed call), so only the harness needs the exemption. + Path("eval/gateway_chaos_v0/kill_matrix.py"), # Step 2.6 — query-understanding components that delegate to LLM / # Embedder SPIs. Query understanding is pre-retrieval text # transformation; the upstream gateway (Phase 3) is the