Skip to content

test(perf): chaos-under-load gate + Locust v1 suite (Step 7.1) - #164

Merged
officialCodeWork merged 1 commit into
mainfrom
build/phase-7/step-7.1-load-testing
Jun 8, 2026
Merged

test(perf): chaos-under-load gate + Locust v1 suite (Step 7.1)#164
officialCodeWork merged 1 commit into
mainfrom
build/phase-7/step-7.1-load-testing

Conversation

@officialCodeWork

Copy link
Copy Markdown
Owner

Summary

Step 7.1 — Load testing (first Phase-7 hardening step): builds on Step 4.6 (ADR-0025: the in-process p99-overhead gate + Locust v0) with the load + resilience side the GA bar needs — 1000 QPS sustained, p99 < 500 ms, chaos under load — split into a deterministic CI gate and a cluster runbook.

  • Chaos-under-load is a deterministic CI gate (eval/gateway_chaos_v0/): drives the in-process gateway (httpx ASGITransport, no socket / no cluster) under concurrent load while injecting backend faults, and asserts the gateway degrades gracefully — no 5xx, 100% success, the failing backend's circuit breaker opens. The asserted property is resilience, not throughput, so it's timing-independent and runs in the perf CI job (tests/perf/test_chaos_under_load.py); task chaos-test runs the harness --check.
  • Reuses the Phase-4 machinery — it validates it. FaultSpec (seeded failure probability + injected latency) + Chaos{Vector,Keyword,Graph}RetrievalBackend are SPI wrappers exactly like the rag-breaker wrappers (allowlisted in policy-coverage for the same reason — they forward the already-policy-merged retrieve_ids/expand), wired behind real breakers + the fallback ladder. Under 100% vector failure the vector breaker trips and the fallback fuses the survivors; all-backends-fail degrades to graceful empty answers — the client never sees a 5xx.
  • The query is varied per request (a unique nonce) in both the chaos harness and the Locust suite — identical queries hit the retrieval cache and short-circuit before retrieval, so a constant-query load test would measure the cache (and never exercise the chaos backends).
  • The 1000-QPS / p99<500ms acceptance is a cluster runbook, not a CI gate (hardware/backends-bound — same reasoning as ADR-0025): an extended Locust v1 suite (weighted /v1/retrieve · /v1/query · /v1/feedback · /v1/status/metrics · /healthz mix + a ramp LoadTestShape) + documented acceptance targets in the runbook.

No packages//apps/ source or dist/ changes — this is eval/ + tests/perf/ + Taskfile + docs.

Documentation

Test plan

  • pytest -m perf tests/perf/test_chaos_under_load.py — vector-fail → breaker opens + no 5xx; all-fail → graceful empty 200; healthy → breakers closed; latency injection → no errors
  • task chaos-test / python -m eval.gateway_chaos_v0.harness --all-fail → PASS
  • python -m eval.gateway_load_v0.harness --check (overhead gate) still passes
  • ruff, format, RAG001, policy-coverage (chaos backends allowlisted), schema/openapi-drift (dist untouched)
  • Cluster: drive the Locust suite to ≥ 1000 RPS, confirm e2e p99 < 500 ms (operator, per the runbook)

🤖 Generated with Claude Code

Builds on Step 4.6 (ADR-0025: the in-process p99-overhead gate + Locust v0)
with the load + resilience side the GA bar needs, split into a deterministic
CI gate and a cluster runbook.

- Chaos-under-load is a deterministic CI gate (eval/gateway_chaos_v0/): drive
  the in-process gateway under concurrent load while injecting backend faults
  and assert graceful degradation — no 5xx, 100% success, the failing
  backend's circuit breaker opens. The property is resilience (not raw
  throughput), so it's timing-independent and runs in the perf CI job
  (tests/perf/test_chaos_under_load.py); `task chaos-test` runs the harness.
- Reuses the Phase-4 breakers + fallback — it validates them, builds no new
  resilience. FaultSpec (seeded failure prob + latency) + Chaos{Vector,
  Keyword,Graph}RetrievalBackend are SPI wrappers like the breaker wrappers
  (allowlisted in policy-coverage for the same reason), wired behind real
  breakers. Under 100% vector failure the vector breaker trips and the
  fallback fuses the survivors; all-fail degrades to graceful empty answers.
- Both the chaos harness and the Locust suite vary the query per request — a
  constant query hits the retrieval cache and short-circuits, so it'd measure
  the cache (and never exercise the chaos backends).
- The 1000-QPS / p99<500ms acceptance is a cluster runbook, not a CI gate
  (hardware/backends-bound — same reasoning as ADR-0025): an extended Locust
  v1 suite (weighted read/write mix + a ramp LoadTestShape) + documented
  targets in docs/guides/load-testing.md.

All gates green: ruff, format, RAG001, policy-coverage, perf (chaos gate),
schema/openapi-drift (dist untouched — no source/types changed). Phase 7
opens: 7.1 done (75/84).

Docs: docs/guides/load-testing.md, docs/adr/ADR-0043-load-chaos-testing.md,
docs/reference/perf.md, docs/README.md. TRACKER updated.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@officialCodeWork
officialCodeWork merged commit f5d84b1 into main Jun 8, 2026
12 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant