Skip to content

test(perf): chaos kill-matrix gate + cache/rerank degrade-open + LitmusChaos (Step 7.2) #165

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
officialCodeWork merged 1 commit into from
Jun 8, 2026
Merged

test(perf): chaos kill-matrix gate + cache/rerank degrade-open + LitmusChaos (Step 7.2) #165

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 17 additions & 6 deletions TRACKER.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,13 @@
| | |
|---|---|
| **Last updated** | 2026-06-08 |
| **Current phase** | Phase 7 — Pilot, Harden, GA (**1 / 10 steps**) — Phases 0–6 ✅ complete |
| **Overall** | **75 / 84 steps** — Phases 0–6 complete |
| **Next action** | **Step 7.2 — Chaos engineering**: Chaos Monkey / LitmusChaos; kill each backend; verify the fallback chain holds (builds on the 7.1 in-process chaos harness). |
| **Current phase** | Phase 7 — Pilot, Harden, GA (**2 / 10 steps**) — Phases 0–6 ✅ complete |
| **Overall** | **76 / 84 steps** — Phases 0–6 complete |
| **Next action** | **Step 7.3 — Red-team**: prompt injection, ACL bypass, PII egress, and tenant-escape probes as an adversarial test suite. |

**Recently shipped**

- **7.2** ✅ Chaos engineering — a deterministic in-process **kill-matrix gate** (`eval/gateway_chaos_v0/kill_matrix.py` + `tests/perf/test_chaos_kill_matrix.py`) that extends 7.1 from the three retrieval backends to the **full hot-path set** (vector/keyword/graph/embedder/retrieval_cache/reranker/llm): kill each backend in turn (100% unavailable) behind the real breakers + fallback, drive `/v1/query`, and assert **no single failure 5xx-es the gateway** (no 5xx, on-path retrieval breaker opens, expected degraded shape; a seeded keyword corpus + real `hydrate` make rerank/generate actually run). **Chaos fixed what it found** — the matrix exposed that a down **retrieval cache** or **reranker** 5xx-ed, so the gateway gained two minimal **degrade-open** guards (`gateway.cache.degraded` → miss on `/v1/query` + `/v1/retrieve`; `gateway.rerank.degraded` → retrieval-only, honouring `RerankPipeline`'s "caller decides" contract); LLM + embedder already degraded. **LitmusChaos cluster manifests** (`infra/chaos/`: gateway pod-delete + backend `pod-network-loss`/`latency` with httpProbe acceptance) as the cluster runbook + `task chaos-kill`. No `dist/`/SPI/config change. [#165](https://github.com/officialCodeWork/AgentContextOS/pull/165)
- **7.1** ✅ Load + chaos testing — a **chaos-under-load** CI gate (`eval/gateway_chaos_v0/`): drives the in-process gateway under concurrent load while injecting backend faults (`FaultSpec` + `Chaos{Vector,Keyword,Graph}RetrievalBackend` SPI wrappers behind real breakers) and asserts **graceful degradation** — no 5xx, 100% success, the failing backend's breaker opens (validating the Phase-4 breakers + fallback, builds no new resilience); `tests/perf/test_chaos_under_load.py` (timing-independent, `perf`-marked) + `task chaos-test`. An extended **Locust v1 suite** (weighted read/write mix + varied queries + a ramp `LoadTestShape`) + documented **acceptance targets** (≥ 1000 RPS sustained, e2e p99 < 500 ms) as a cluster runbook — [#164](https://github.com/officialCodeWork/AgentContextOS/pull/164)

- **6.10** ✅ Compliance posture (Phase 6 capstone) — new `rag-compliance` package: `RetentionEnforcer` drives tenant-scoped `purge_*` (data retention + GDPR right-to-erasure) over the Feedback/Provenance stores (non-abstract `purge_before`/`purge_tenant` SPI capability with `dry_run` so a preview counts-without-deleting; audit chain never purged in place — its retention is the 6.6b WORM export); `POST /v1/compliance/erase` (tenant-scoped, dry-run default + two-flag delete); per-tenant `data_region` enforced at ingest (`ResidencyViolationError` → 403); `GET /v1/status/compliance` posture + `compliance_posture`/`residency_ok`; PII-free `compliance.*` events; `cfg.compliance` + `tenants[].data_region`/`retention_days`; `ragctl compliance report`/`demo`; SOC 2 + GDPR control-mapping docs backed by the live posture — [#162](https://github.com/officialCodeWork/AgentContextOS/pull/162)
Expand Down Expand Up @@ -67,8 +68,8 @@
| 4 | Reliability | 6 | **6** | 0 |
| 5 | Eval & Observability | 7 | **7** | 0 |
| 6 | Governance & Tenancy | 10 | **10** | 0 |
| 7 | Pilot, Harden, GA | 10 | **1** | 9 |
| **Total** | | **84** | **75** | **9** |
| 7 | Pilot, Harden, GA | 10 | **2** | 8 |
| **Total** | | **84** | **76** | **8** |

---

Expand Down Expand Up @@ -800,7 +801,7 @@ New ground — the only prior crypto was HMAC signing. The V1 plan calls for en
| Step | Title | Status | Planned deliverables |
|------|-------|:------:|----------------------|
| 7.1 | Load testing | ✅ | [#164](https://github.com/officialCodeWork/AgentContextOS/pull/164) — chaos-under-load CI gate (`eval/gateway_chaos_v0` fault backends + harness; `tests/perf/test_chaos_under_load.py`; graceful degradation: no 5xx + breaker opens); Locust v1 suite (varied-query mix + ramp shape); acceptance targets + runbook |
| 7.2 | Chaos engineering | ⏳ | Chaos Monkey / LitmusChaos; kill each backend; verify the fallback chain holds |
| 7.2 | Chaos engineering | ✅ | [#165](https://github.com/officialCodeWork/AgentContextOS/pull/165) — in-process **kill-matrix** gate (kill each hot-path backend → no 5xx, breaker opens, expected degraded shape); two new gateway **degrade-open** guards (retrieval cache + reranker); LitmusChaos cluster manifests (`infra/chaos/`) + `task chaos-kill` |
| 7.3 | Red-team | ⏳ | Prompt injection, ACL bypass, PII egress, tenant-escape probes |
| 7.4 | Design partner onboarding | ⏳ | 2–3 design partners; feedback incorporated; case study documented |
| 7.5 | Documentation site | ⏳ | Docusaurus / MkDocs site; API reference generated from OpenAPI; quickstart guides |
Expand All @@ -819,6 +820,15 @@ New ground — the only prior crypto was HMAC signing. The V1 plan calls for en
- **The 1000-QPS / p99 < 500 ms acceptance is a cluster runbook, not a CI gate** (hardware/backends-bound — same reasoning as ADR-0025). Extended **Locust v1 suite** (`eval/gateway_load_v0/locustfile.py`): a weighted read/write mix (`/v1/retrieve` · `/v1/query` · `/v1/feedback` · `/v1/status/metrics` · `/healthz`) + a `LoadTestShape` that ramps → holds a plateau → ramps down (env-tunable peak / hold for distributed workers). Acceptance targets + the runbook in [`guides/load-testing.md`](docs/guides/load-testing.md)
- **Scope:** in-process chaos gate (resilience) + the Locust suite + runbook for the throughput acceptance. **Deferred:** distributed-Locust-in-CI against an ephemeral cluster, latency-based breaker tripping (timeouts), storage/LLM-layer fault injection, soak/endurance tests (cluster-level chaos with LitmusChaos is Step 7.2). ~4 chaos perf tests; all gates green (ruff, mypy --strict, RAG001, policy-coverage, schema/openapi-drift, log-schema). [ADR-0043](docs/adr/ADR-0043-load-chaos-testing.md), [guides/load-testing.md](docs/guides/load-testing.md), [reference/perf.md](docs/reference/perf.md)

### 7.2 — Chaos engineering ✅ [#165](https://github.com/officialCodeWork/AgentContextOS/pull/165)

- Builds on Step 7.1 (ADR-0043). 7.1 fault-injected the three *retrieval* backends under load; 7.2 closes out the deliverable — "**kill each backend; verify the fallback chain holds**" — and the LitmusChaos piece 7.1 deferred. The GA bar is a sharp invariant: **no single backend failure can 5xx the gateway**
- **The kill-matrix is a deterministic in-process CI gate.** `eval/gateway_chaos_v0/kill_matrix.py` walks the full hot-path backend set — vector / keyword / graph / embedder / retrieval_cache / reranker / llm — kills **one at a time** (100% unavailable) behind the *real* Phase-4 breakers + fallback + the gateway degrade guards, drives `/v1/query`, and asserts **no 5xx, 100% success**, the killed *on-path* retrieval breaker **opens**, and the expected degraded shape (`tests/perf/test_chaos_kill_matrix.py`, `perf`-marked + timing-independent; `task chaos-kill`). A tiny seeded keyword corpus + the store's real `hydrate` make rerank/pack/generate actually run, so every kill exercises a genuine degrade path
- **Chaos *fixes* what it finds.** Running the matrix exposed two ways to 5xx: a down **retrieval cache** (`get_best`/`put_with_query`) and a down **reranker** (`reranker.rerank`) propagated unhandled. Both gained minimal **degrade-open** guards at the call site (matching the existing LLM guard): the cache treats an outage as a *miss* (`gateway.cache.degraded`, on `/v1/query` + `/v1/retrieve`); the reranker falls back to *retrieval-only* (`gateway.rerank.degraded`, honouring `RerankPipeline`'s documented "caller decides" contract — the reranker is also the hydration stage, so the degraded response carries no citations, same shape as `rerank=false`). The LLM (answer-generation guard) + embedder (understanding's per-component catch) already degraded
- **Graph is off the default read path.** `RetrievalRouter.decide` only sets `use_graph` when the caller supplies graph seeds; a standard query never does, so killing graph cannot affect it (breaker stays closed) — the matrix records this honestly (no 5xx asserted, breaker-open *not* required) rather than forcing a synthetic graph query
- **Cluster chaos is a runbook, not a CI gate** (topology-bound). `infra/chaos/` LitmusChaos manifests: gateway `pod-delete` (PDB-protected instance loss) + `pod-network-loss`/`pod-network-latency` toward a backend host (the cluster way to "kill a backend", since backends are reached by host), each with an embedded **httpProbe** asserting the gateway stays 200 throughout
- **Scope:** kill-matrix gate + the two degrade-open hardenings + the LitmusChaos runbook. No new core/wire types, SPI methods, config, or events (`dist/` untouched; the degrade kinds are log-only like `gateway.answer.failed`; the kill wrappers are pure-raise so the policy-coverage linter needs no new rule). **Deferred:** latency-based breaker tripping, multi-backend simultaneous kills as a gate, Litmus-in-CI against an ephemeral cluster, soak chaos. 12 kill-matrix perf tests; all gates green (ruff, mypy --strict 325 files, RAG001, policy-coverage, schema/openapi-drift, log-schema). [ADR-0044](docs/adr/ADR-0044-chaos-engineering.md), [guides/chaos-engineering.md](docs/guides/chaos-engineering.md), [reference/perf.md](docs/reference/perf.md)

---

## PR & Branch History
Expand Down Expand Up @@ -964,6 +974,7 @@ Complete log of every PR. Routine Dependabot bumps are grouped; everything else
| [#156](https://github.com/officialCodeWork/AgentContextOS/pull/156) | 2026-06-08 | feat(crypto): cfg.kms + key-manager factory + AWS KMS provider (Step 6.7b) |
| [#157](https://github.com/officialCodeWork/AgentContextOS/pull/157) | 2026-06-08 | feat(crypto): GCP / Azure / Vault KMS providers (Step 6.7c) |
| [#158](https://github.com/officialCodeWork/AgentContextOS/pull/158) | 2026-06-08 | feat(crypto): zero-downtime key rotation — RotatingKeyManager (Step 6.7d) |
| [#165](https://github.com/officialCodeWork/AgentContextOS/pull/165) | 2026-06-08 | test(perf): chaos kill-matrix gate + cache/rerank degrade-open + LitmusChaos (Step 7.2) |
| #78–#80, #116–#118 | Open | Dependabot bumps — awaiting merge |
| #81 | Closed | Dependabot bump — superseded |

Expand Down
5 changes: 5 additions & 0 deletions Taskfile.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,11 @@ tasks:
cmds:
- "{{.PYTHON}} -m eval.gateway_chaos_v0.harness --check"

chaos-kill:
desc: "Run the chaos kill-matrix — kill each backend in turn, assert no single failure 5xx-es the gateway"
cmds:
- "{{.PYTHON}} -m eval.gateway_chaos_v0.kill_matrix --check"

# ---------------------------------------------------------------------------
# Schemas
# env: sets PYTHONPATH cross-platform (Task handles Windows vs Unix syntax)
Expand Down
141 changes: 105 additions & 36 deletions apps/gateway/src/rag_gateway/query.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@
ProvenanceNotFoundError,
RagError,
RateLimitError,
RerankerError,
ResidencyViolationError,
RetrievalError,
ScimConflictError,
Expand Down Expand Up @@ -604,13 +605,28 @@ async def _handle_query(
shadow_expansion: dict[str, list[str]] = {}
cache_hit = False
if cache is not None:
cached_refs = await cache.get_best(
ctx,
query_text=body.query,
plan_hash=plan_hash,
corpus_version=corpus_version,
scope=scope,
)
try:
cached_refs = await cache.get_best(
ctx,
query_text=body.query,
plan_hash=plan_hash,
corpus_version=corpus_version,
scope=scope,
)
except Exception as exc:
# The retrieval cache is a non-authoritative optimisation: a
# backend outage (e.g. Redis down) must degrade to a cache *miss*,
# never fail the request. Chaos gate: Step 7.2 kills this backend.
_log.warning(
"gateway.cache.degraded",
extra={
"event_kind": "gateway.cache.degraded",
"stage": "get",
"tenant_id": str(ctx.tenant_id),
"error": repr(exc),
},
)
cached_refs = None
if cached_refs is not None:
cache_hit = True
chunk_refs = cached_refs
Expand Down Expand Up @@ -650,28 +666,56 @@ async def _handle_query(
raise RetrievalError(f"retrieval failed: {exc!r}") from exc

if cache is not None:
await cache.put_with_query(
ctx,
query_text=body.query,
plan_hash=plan_hash,
corpus_version=corpus_version,
value=chunk_refs,
)
try:
await cache.put_with_query(
ctx,
query_text=body.query,
plan_hash=plan_hash,
corpus_version=corpus_version,
value=chunk_refs,
)
except Exception as exc:
# A cache *write* failure is purely an optimisation loss —
# the fresh result is already in hand; never fail on it.
_log.warning(
"gateway.cache.degraded",
extra={
"event_kind": "gateway.cache.degraded",
"stage": "put",
"tenant_id": str(ctx.tenant_id),
"error": repr(exc),
},
)

# A/B routing (Step 5.7c) — record the served arm's outcome into the
# experiment tracker, once per fresh retrieval (not on a cache hit).
_record_ab_outcome(deps, variant, chunk_refs)

# Optional rerank (returns hydrated Chunks).
# Optional rerank (returns hydrated Chunks). A reranker outage is
# non-fatal: the RerankPipeline propagates RerankerError by contract and
# leaves the caller to fall back to retrieval-only. We degrade to the
# chunk-ref view (no hydrated chunks / citations this request) rather than
# 5xx — Step 7.2's chaos gate kills this backend and asserts the survival.
chunks: list[Chunk] = []
if body.rerank and chunk_refs:
with timings.stage("rerank"):
chunks = await deps.reranker.rerank(
ctx,
body.query,
chunk_refs,
top_k=body.top_k,
)
try:
chunks = await deps.reranker.rerank(
ctx,
body.query,
chunk_refs,
top_k=body.top_k,
)
except RerankerError as exc:
_log.warning(
"gateway.rerank.degraded",
extra={
"event_kind": "gateway.rerank.degraded",
"tenant_id": str(ctx.tenant_id),
"error": repr(exc),
},
)
chunks = []

# Optional packing.
packed: PackedContext | None = None
Expand Down Expand Up @@ -831,13 +875,26 @@ async def _handle_retrieve(
scope = _scope_for(body, variant)
corpus_version = await resolve_corpus_version(ctx, deps.corpus_store, list(body.corpus_ids))
if cache is not None:
cached_refs = await cache.get_best(
ctx,
query_text=body.query,
plan_hash=plan_hash,
corpus_version=corpus_version,
scope=scope,
)
try:
cached_refs = await cache.get_best(
ctx,
query_text=body.query,
plan_hash=plan_hash,
corpus_version=corpus_version,
scope=scope,
)
except Exception as exc:
# Non-authoritative cache: a backend outage degrades to a miss.
_log.warning(
"gateway.cache.degraded",
extra={
"event_kind": "gateway.cache.degraded",
"stage": "get",
"tenant_id": str(ctx.tenant_id),
"error": repr(exc),
},
)
cached_refs = None
if cached_refs is not None:
span.set_attribute("rag.gateway.cache_hit", True)
total_ms = (time.monotonic() - start) * 1000.0
Expand Down Expand Up @@ -886,14 +943,26 @@ async def _handle_retrieve(
raise RetrievalError(f"retrieval failed: {exc!r}") from exc

if cache is not None:
await cache.put_with_query(
ctx,
query_text=body.query,
plan_hash=plan_hash,
corpus_version=corpus_version,
value=chunk_refs,
scope=scope,
)
try:
await cache.put_with_query(
ctx,
query_text=body.query,
plan_hash=plan_hash,
corpus_version=corpus_version,
value=chunk_refs,
scope=scope,
)
except Exception as exc:
# A cache write failure is an optimisation loss, never fatal.
_log.warning(
"gateway.cache.degraded",
extra={
"event_kind": "gateway.cache.degraded",
"stage": "put",
"tenant_id": str(ctx.tenant_id),
"error": repr(exc),
},
)

# A/B routing (Step 5.7c) — record the served arm's outcome (fresh path).
_record_ab_outcome(deps, variant, chunk_refs)
Expand Down
Loading
Loading