test(perf): chaos kill-matrix gate + cache/rerank degrade-open + LitmusChaos (Step 7.2)

[officialCodeWork]

Summary

Step 7.2 — Chaos engineering: kill each backend, verify the fallback chain holds. Extends the Step 7.1 chaos gate from the three retrieval backends to the full hot-path backend set and proves the GA resilience bar — no single backend failure can 5xx the gateway.

• Kill-matrix CI gate (eval/gateway_chaos_v0/kill_matrix.py + tests/perf/test_chaos_kill_matrix.py, task chaos-kill) — kills each of vector / keyword / graph / embedder / retrieval_cache / reranker / llm in turn (100% unavailable) behind the real Phase-4 breakers + fallback, drives /v1/query, and asserts no 5xx, 100% success, the on-path retrieval breaker opens, and the expected degraded shape. A seeded keyword corpus + the store's real hydrate make rerank/pack/generate actually run, so every kill exercises a genuine degrade path. Timing-independent → deterministic in the perf CI job.
• Chaos fixed what it found. The matrix exposed two real 5xx holes — a down retrieval cache and reranker propagated unhandled. Both gained minimal degrade-open guards at the gateway call site:
  • retrieval cache → treats an outage as a miss (gateway.cache.degraded, on /v1/query + /v1/retrieve)
  • reranker → falls back to retrieval-only (gateway.rerank.degraded, honouring RerankPipeline's documented "caller decides" contract)
  • the LLM (answer-generation guard) + embedder (understanding's per-component catch) already degraded.
• Graph is off the default read path (RetrievalRouter.decide needs caller seeds) — its kill is survivable by construction, recorded honestly rather than forcing a synthetic query.
• LitmusChaos cluster runbook (infra/chaos/) — gateway pod-delete + backend pod-network-loss/pod-network-latency with an embedded httpProbe asserting the gateway stays 200 throughout. Topology-bound, so a runbook, not a CI gate (same split as the 7.1 load runbook).

No dist/, SPI, config, or registered-event change (degrade kinds are log-only like gateway.answer.failed; the kill wrappers are pure-raise → no policy-coverage rule needed).

Verification

Empirical kill-matrix run — all seven survive with the expected degraded shape:

killed backend	5xx	degraded response
vector	0	breaker open → fallback fuses survivors
keyword	0	breaker open → graceful empty
graph	0	off the read path — unaffected
embedder	0	keyword-only fallback, full pipeline
retrieval_cache	0	clean miss, full pipeline
reranker	0	retrieval-only (no citations)
llm	0	citations kept, answer omitted

Documentation

• docs/adr/ADR-0044-chaos-engineering.md — the decision (kill-matrix gate, the two hardenings, cluster runbook, alternatives).
• docs/guides/chaos-engineering.md — the runbook (in-process gate + LitmusChaos).
• docs/reference/perf.md — kill-matrix section.
• infra/chaos/README.md — operator runbook for the manifests.
• docs/README.md — index rows. TRACKER.md — 7.2 ✅, Phase 7 2/10 (76/84), next: 7.3.

Test plan

• pytest -m perf tests/perf/test_chaos_kill_matrix.py — 12 tests green
• pytest -m perf tests/perf/ — full perf suite (incl. 7.1 chaos gate) green
• pytest apps/gateway/tests/ — 310 tests, no regressions
• task chaos-kill → PASS (all 7 backends survive)
• ruff, ruff format, mypy --strict (325 files), RAG001, policy-coverage, log/event-schema, schema/openapi drift (dist untouched)

🤖 Generated with Claude Code