test(perf): chaos-under-load gate + Locust v1 suite (Step 7.1) - #164
Merged
Conversation
Builds on Step 4.6 (ADR-0025: the in-process p99-overhead gate + Locust v0)
with the load + resilience side the GA bar needs, split into a deterministic
CI gate and a cluster runbook.
- Chaos-under-load is a deterministic CI gate (eval/gateway_chaos_v0/): drive
the in-process gateway under concurrent load while injecting backend faults
and assert graceful degradation — no 5xx, 100% success, the failing
backend's circuit breaker opens. The property is resilience (not raw
throughput), so it's timing-independent and runs in the perf CI job
(tests/perf/test_chaos_under_load.py); `task chaos-test` runs the harness.
- Reuses the Phase-4 breakers + fallback — it validates them, builds no new
resilience. FaultSpec (seeded failure prob + latency) + Chaos{Vector,
Keyword,Graph}RetrievalBackend are SPI wrappers like the breaker wrappers
(allowlisted in policy-coverage for the same reason), wired behind real
breakers. Under 100% vector failure the vector breaker trips and the
fallback fuses the survivors; all-fail degrades to graceful empty answers.
- Both the chaos harness and the Locust suite vary the query per request — a
constant query hits the retrieval cache and short-circuits, so it'd measure
the cache (and never exercise the chaos backends).
- The 1000-QPS / p99<500ms acceptance is a cluster runbook, not a CI gate
(hardware/backends-bound — same reasoning as ADR-0025): an extended Locust
v1 suite (weighted read/write mix + a ramp LoadTestShape) + documented
targets in docs/guides/load-testing.md.
All gates green: ruff, format, RAG001, policy-coverage, perf (chaos gate),
schema/openapi-drift (dist untouched — no source/types changed). Phase 7
opens: 7.1 done (75/84).
Docs: docs/guides/load-testing.md, docs/adr/ADR-0043-load-chaos-testing.md,
docs/reference/perf.md, docs/README.md. TRACKER updated.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Step 7.1 — Load testing (first Phase-7 hardening step): builds on Step 4.6 (ADR-0025: the in-process p99-overhead gate + Locust v0) with the load + resilience side the GA bar needs — 1000 QPS sustained, p99 < 500 ms, chaos under load — split into a deterministic CI gate and a cluster runbook.
eval/gateway_chaos_v0/): drives the in-process gateway (httpx ASGITransport, no socket / no cluster) under concurrent load while injecting backend faults, and asserts the gateway degrades gracefully — no 5xx, 100% success, the failing backend's circuit breaker opens. The asserted property is resilience, not throughput, so it's timing-independent and runs in theperfCI job (tests/perf/test_chaos_under_load.py);task chaos-testruns the harness--check.FaultSpec(seeded failure probability + injected latency) +Chaos{Vector,Keyword,Graph}RetrievalBackendare SPI wrappers exactly like the rag-breaker wrappers (allowlisted in policy-coverage for the same reason — they forward the already-policy-mergedretrieve_ids/expand), wired behind real breakers + the fallback ladder. Under 100% vector failure the vector breaker trips and the fallback fuses the survivors; all-backends-fail degrades to graceful empty answers — the client never sees a 5xx./v1/retrieve·/v1/query·/v1/feedback·/v1/status/metrics·/healthzmix + a rampLoadTestShape) + documented acceptance targets in the runbook.No
packages//apps/source ordist/changes — this iseval/+tests/perf/+ Taskfile + docs.Documentation
docs/guides/load-testing.md— the two-layer runbook (CI gates vs cluster Locust), running Locust at scale, acceptance targetsdocs/adr/ADR-0043-load-chaos-testing.md— decision + alternativesdocs/reference/perf.md(chaos section) +docs/README.md; TRACKER (7.1 ✅, Phase 7 opens, 75/84)Test plan
pytest -m perf tests/perf/test_chaos_under_load.py— vector-fail → breaker opens + no 5xx; all-fail → graceful empty 200; healthy → breakers closed; latency injection → no errorstask chaos-test/python -m eval.gateway_chaos_v0.harness --all-fail→ PASSpython -m eval.gateway_load_v0.harness --check(overhead gate) still passes🤖 Generated with Claude Code