From 37a35d871f4f326d094c0943a42753791b7c3387 Mon Sep 17 00:00:00 2001 From: Deep Kumar Singh Kushwah Date: Sun, 7 Jun 2026 23:29:43 +0530 Subject: [PATCH] feat(admin-ui): A/B experiments console card + Phase-5 close-out (Step 5.7d) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add an "A/B experiments" card to the admin Live Status page surfacing GET /v1/status/experiments — one row per running experiment with the lift (relative, colour-coded), a verdict badge (candidate wins / candidate worse / no change / collecting), control->candidate means, the confidence interval, and per-arm sample counts. Live-wired via useLive (fetchExperiments + seedExperiments) with a seed/demo fallback when no gateway is configured. New ABAnalysis TS type mirrors rag_core.eval.ABAnalysisResult. No backend change — the endpoint shipped in 5.7a. Phase-5 close-out: Step 5.7 is complete, so Phase 5 — Eval & Observability is done (7/7); TRACKER milestone counts updated (64/84), next action points to Phase 6 (6.1 — Logical multi-tenancy). Deferred items remain documented. admin-ui gates green: tsc, eslint, 34 vitest (incl. the new card test), next build; browser-verified both the significant-win and insufficient-data states render. Co-Authored-By: Claude Opus 4.7 --- TRACKER.md | 31 ++++++--- apps/admin-ui/src/app/status/page.tsx | 98 +++++++++++++++++++++++++++ apps/admin-ui/src/lib/api.ts | 15 ++++ apps/admin-ui/src/lib/mock.ts | 43 ++++++++++++ apps/admin-ui/src/lib/types.ts | 30 ++++++++ apps/admin-ui/test/status.test.tsx | 13 ++++ docs/reference/admin-ui.md | 10 +-- docs/reference/experiments.md | 9 +++ 8 files changed, 234 insertions(+), 15 deletions(-) diff --git a/TRACKER.md b/TRACKER.md index 14da970..2624da1 100644 --- a/TRACKER.md +++ b/TRACKER.md @@ -14,12 +14,13 @@ | | | |---|---| | **Last updated** | 2026-06-07 | -| **Current phase** | Phase 5 — Eval & Observability (**6 / 7 steps**) | -| **Overall** | **63 / 84 steps** — Phases 0–4 complete | -| **Next action** | **Step 5.7d — Console + close-out** — admin-console A/B card surfacing `GET /v1/status/experiments` (lift + CI per experiment), plus Phase-5 close-out. Final slice of the final Phase-5 step (5.7a–d). | +| **Current phase** | **Phase 5 — Eval & Observability ✅ complete (7 / 7 steps)** → starting Phase 6 | +| **Overall** | **64 / 84 steps** — Phases 0–5 complete | +| **Next action** | **Step 6.1 — Logical multi-tenancy** (Phase 6 — Governance & Tenancy): namespace isolation; per-tenant config, quotas, ACLs in `rag.yaml`. First step of the governance phase. | **Recently shipped** +- **5.7d** ✅ Experiments console card + Phase-5 close-out — admin Live-Status A/B card over `GET /v1/status/experiments` (lift + CI per experiment) — [#147](https://github.com/officialCodeWork/AgentContextOS/pull/147) - **5.7c** ✅ A/B routing — `ABRouter` deterministically *serves* the candidate to a fraction of users (variant-partitioned cache, `ExperimentAssignment` response tag) — [#146](https://github.com/officialCodeWork/AgentContextOS/pull/146) - **5.7b** ✅ Shadow mode — observe-only candidate fan-out (`ShadowRunner`, background task) feeding the A/B tracker — [#145](https://github.com/officialCodeWork/AgentContextOS/pull/145) - **5.7a** ✅ A/B analyzer + experiment tracker + `GET /v1/status/experiments` dashboard — [#143](https://github.com/officialCodeWork/AgentContextOS/pull/143) @@ -52,10 +53,10 @@ | 2 | Retrieval Engine | 11 | **11** | 0 | | 3 | Gateway & Agent Runtime | 11 | **11** | 0 | | 4 | Reliability | 6 | **6** | 0 | -| 5 | Eval & Observability | 7 | **6** | 1 | +| 5 | Eval & Observability | 7 | **7** | 0 | | 6 | Governance & Tenancy | 10 | 0 | 10 | | 7 | Pilot, Harden, GA | 10 | 0 | 10 | -| **Total** | | **84** | **63** | **21** | +| **Total** | | **84** | **64** | **20** | --- @@ -507,7 +508,7 @@ - One reusable engine `rag_gateway.perf` (`measure_gateway_overhead` + `profile_gateway`) powers gate + harness + `ragctl perf`; `tests/contract/budgets.py` made real (drift-guarded); load test = in-process `eval/gateway_load_v0` + Locust `locustfile.py` - ~14 tests; [ADR-0025](docs/adr/ADR-0025-latency-load-testing.md), [reference/perf.md](docs/reference/perf.md), [architecture/latency.md](docs/architecture/latency.md) -## Phase 5 — Eval & Observability (Weeks 22–28) 🚧 (6 / 7) +## Phase 5 — Eval & Observability (Weeks 22–28) ✅ | Step | Title | Status | PR | |------|-------|:------:|----| @@ -523,11 +524,11 @@ | 5.6d | — Regression bisector | ✅ | [#140](https://github.com/officialCodeWork/AgentContextOS/pull/140) | | 5.6e | — Grafana dashboards | ✅ | [#141](https://github.com/officialCodeWork/AgentContextOS/pull/141) | | 5.6f | — Cross-links + close-out | ✅ | [#142](https://github.com/officialCodeWork/AgentContextOS/pull/142) | -| 5.7 | A/B testing & shadow mode | 🚧 | — | +| 5.7 | A/B testing & shadow mode | ✅ | [#143–#147](https://github.com/officialCodeWork/AgentContextOS/pull/147) | | 5.7a | — A/B analyzer + tracker + dashboard | ✅ | [#143](https://github.com/officialCodeWork/AgentContextOS/pull/143) | | 5.7b | — Shadow mode | ✅ | [#145](https://github.com/officialCodeWork/AgentContextOS/pull/145) | | 5.7c | — A/B routing | ✅ | [#146](https://github.com/officialCodeWork/AgentContextOS/pull/146) | -| 5.7d | — Console + close-out | ⏳ | — | +| 5.7d | — Console + close-out | ✅ | [#147](https://github.com/officialCodeWork/AgentContextOS/pull/147) | ### 5.1 — Per-query tracing & provenance ✅ [#132](https://github.com/officialCodeWork/AgentContextOS/pull/132) @@ -602,10 +603,10 @@ - Wires the observability surfaces together: the Trace viewer's *Logs* button → `/logs?trace=` (clearable violet trace-chip), a log detail drawer → *View this trace's logs*, a **Trace** quick-link on Live Status - Admin-ui only; 1 vitest cross-link spec; closes Step 5.6 -### 5.7 — A/B testing & shadow mode 🚧 +### 5.7 — A/B testing & shadow mode ✅ -- Final Phase-5 step, delivered in vertical slices **5.7a–d**: analyzer + tracker + dashboard (5.7a ✅), shadow mode (5.7b), A/B routing (5.7c), console + close-out (5.7d) -- Shadow N% of live queries; A/B variant routing; statistical analyzer with lift + confidence interval +- Final Phase-5 step, delivered in vertical slices **5.7a–d**: analyzer + tracker + dashboard (5.7a ✅), shadow mode (5.7b ✅), A/B routing (5.7c ✅), console + close-out (5.7d ✅) +- Shadow N% of live queries; A/B variant routing; statistical analyzer with lift + confidence interval — **all delivered; Phase 5 complete** #### 5.7a — A/B analyzer + tracker + dashboard ✅ [#143](https://github.com/officialCodeWork/AgentContextOS/pull/143) @@ -631,6 +632,13 @@ - Same `read_chunk` PDP site (no coverage-linter entry); gated separately by `cfg.experiments.routing_enabled` (requires `enabled`) — independent of `shadow_enabled` — since routing can change a response; `ShadowCandidateConfig` generalised to `CandidateConfig` (shared by `shadow_candidate` + new `routing_candidate`); `build_app(ab_router=…)` inject seam; `ragctl ab` drives assign → serve → record → analyze; ~25 new tests; all gates green (ruff, mypy --strict, RAG001, schema/openapi-drift, proto-compat, policy-coverage) - [ADR-0032](docs/adr/ADR-0032-ab-testing-shadow-mode.md) (5.7c update), [reference/experiments.md](docs/reference/experiments.md), [architecture/ab-routing.md](docs/architecture/ab-routing.md) +#### 5.7d — Console + close-out ✅ [#147](https://github.com/officialCodeWork/AgentContextOS/pull/147) + +- New **A/B experiments** card on the admin **Live Status** page (`apps/admin-ui`) surfacing `GET /v1/status/experiments` — one row per running experiment with the **lift** (relative, colour-coded), a verdict badge (**candidate wins** / **candidate worse** / **no change** / **collecting**), control→candidate means, the confidence interval, and per-arm sample counts; live-vs-seed via `useLive` (`fetchExperiments` + `seedExperiments`), demo badge when no gateway +- `ABAnalysis` TS type mirrors `rag_core.eval.ABAnalysisResult`; admin-ui gates green (tsc / eslint / 34 vitest / `next build`); browser-verified both the significant-win + insufficient-data states render +- **Phase-5 close-out:** Step 5.7 ✅ → **Phase 5 complete (7 / 7)**; deferred items remain documented (per-tenant drift / per-dimension embedding PSI; feedback/breaker/quota Grafana export + Loki-events dashboard; gRPC proto mirror of `corpus_decision` + `experiment`; sequential / multi-metric experiments) +- [reference/experiments.md](docs/reference/experiments.md), [reference/admin-ui.md](docs/reference/admin-ui.md) + ## Phase 6 — Governance & Tenancy (Weeks 28–34) ⏳ | Step | Title | Status | Planned deliverables | @@ -796,6 +804,7 @@ Complete log of every PR. Routine Dependabot bumps are grouped; everything else | [#144](https://github.com/officialCodeWork/AgentContextOS/pull/144) | 2026-06-05 | docs(tracker): restructure for readability + complete PR history | | [#145](https://github.com/officialCodeWork/AgentContextOS/pull/145) | 2026-06-05 | feat(experiments): shadow mode — observe-only candidate fan-out (Step 5.7b) | | [#146](https://github.com/officialCodeWork/AgentContextOS/pull/146) | 2026-06-07 | feat(experiments): A/B routing — serve the candidate to a fraction of users (Step 5.7c) | +| [#147](https://github.com/officialCodeWork/AgentContextOS/pull/147) | 2026-06-07 | feat(admin-ui): A/B experiments console card + Phase-5 close-out (Step 5.7d) | | #78–#80, #116–#118 | Open | Dependabot bumps — awaiting merge | | #81 | Closed | Dependabot bump — superseded | diff --git a/apps/admin-ui/src/app/status/page.tsx b/apps/admin-ui/src/app/status/page.tsx index a69cc71..e7e1225 100644 --- a/apps/admin-ui/src/app/status/page.tsx +++ b/apps/admin-ui/src/app/status/page.tsx @@ -3,6 +3,7 @@ import { Activity, ChevronRight, + FlaskConical, Gauge, MessageSquare, Route, @@ -23,6 +24,7 @@ import { fetchConnectorStatus, fetchCost, fetchDrift, + fetchExperiments, fetchFeedback, fetchQuotas, forceCloseBreaker, @@ -35,11 +37,13 @@ import { seedConnectorStatus, seedCost, seedDrift, + seedExperiments, seedFeedback, seedQuotas, } from "@/lib/mock"; import { relTime } from "@/lib/time"; import type { + ABAnalysis, BreakerState, BreakerView, ComponentHealth, @@ -570,6 +574,99 @@ function CostCard() { ); } +/** A relative figure as a signed percentage (e.g. `+24.0%`). */ +function liftPct(x: number): string { + return `${x >= 0 ? "+" : ""}${(x * 100).toFixed(1)}%`; +} + +/** Verdict badge for one A/B comparison: collecting → still gathering samples; + * otherwise a significant candidate win / regression, or no detectable change. */ +function abVerdict(r: ABAnalysis): { + label: string; + variant: "success" | "error" | "secondary"; +} { + if (r.status === "insufficient_data") return { label: "collecting", variant: "secondary" }; + if (!r.significant) return { label: "no change", variant: "secondary" }; + return r.diff >= 0 + ? { label: "candidate wins", variant: "success" } + : { label: "candidate worse", variant: "error" }; +} + +function liftColor(r: ABAnalysis): string { + if (r.status === "insufficient_data" || !r.significant) return "text-muted-foreground"; + return r.diff >= 0 + ? "text-emerald-600 dark:text-emerald-400" + : "text-red-600 dark:text-red-400"; +} + +function signed(x: number): string { + return `${x >= 0 ? "+" : ""}${x.toFixed(3)}`; +} + +/** A/B experiments (Step 5.7) — control-vs-candidate lift + CI per running experiment. */ +function ExperimentsCard() { + const { rows } = useLive(seedExperiments, fetchExperiments, { + fallbackOnEmpty: true, + }); + + return ( + +
+
A/B experiments
+
+ {rows.length > 0 ? `${rows.length} running` : "control vs candidate"} +
+
+ {rows.length === 0 ? ( +
+ + No experiments running. +
+ ) : ( +
    + {rows.map((r) => { + const verdict = abVerdict(r); + const collecting = r.status === "insufficient_data"; + return ( +
  • +
    +
    +
    {r.experiment}
    +
    {r.metric}
    +
    + {verdict.label} +
    +
    +
    +
    + {collecting ? "—" : liftPct(r.lift)} +
    +
    lift vs control
    +
    +
    + {collecting ? ( +
    {r.n_a + r.n_b} samples · need more
    + ) : ( + <> +
    + {r.variant_a} {r.mean_a.toFixed(3)} → {r.variant_b} {r.mean_b.toFixed(3)} +
    +
    + CI [{signed(r.ci_lower)}, {signed(r.ci_upper)}] · n={r.n_a}/{r.n_b} +
    + + )} +
    +
    +
    • + ); + })} +
+ )} + + ); +} + export default function LiveStatusPage() { const router = useRouter(); const { health, source, connected } = useStatusStream(); @@ -704,6 +801,7 @@ export default function LiveStatusPage() { + ); } diff --git a/apps/admin-ui/src/lib/api.ts b/apps/admin-ui/src/lib/api.ts index 0c23b29..cafe7ac 100644 --- a/apps/admin-ui/src/lib/api.ts +++ b/apps/admin-ui/src/lib/api.ts @@ -7,6 +7,7 @@ import { GATEWAY_URL } from "./config"; import type { + ABAnalysis, BreakerView, ConnectorStatusLive, Corpus, @@ -226,6 +227,20 @@ export async function fetchCost(tenantId: string): Promise { return data.status === "insufficient_data" ? [] : [data]; } +// --------------------------------------------------------------------------- +// A/B experiments (Step 5.7) +// --------------------------------------------------------------------------- +/** One control-vs-candidate comparison per running experiment. Empty when + * experiments are disabled or none have been observed, so `useLive` falls back + * to seed data (with the demo badge) rather than rendering an empty card. */ +export async function fetchExperiments(tenantId: string): Promise { + const data = await request<{ experiments: ABAnalysis[]; total: number }>( + tenantId, + "/v1/status/experiments", + ); + return data.experiments; +} + // --------------------------------------------------------------------------- // Per-query trace & provenance (Step 5.1) // --------------------------------------------------------------------------- diff --git a/apps/admin-ui/src/lib/mock.ts b/apps/admin-ui/src/lib/mock.ts index e03c7dc..e4d9c43 100644 --- a/apps/admin-ui/src/lib/mock.ts +++ b/apps/admin-ui/src/lib/mock.ts @@ -1,6 +1,7 @@ /** Seed data for the console — ported from the design handoff (deterministic). */ import type { + ABAnalysis, ApiKey, AuditEvent, BreakerView, @@ -563,3 +564,45 @@ export function seedFeedback(tenantId: string): FeedbackStats[] { }, ]; } + +/** A/B experiments (Step 5.7) — one significant win + one still-collecting comparison. */ +export function seedExperiments(_tenantId: string): ABAnalysis[] { + return [ + { + experiment: "router_v2", + metric: "retrieval_score", + variant_a: "control", + variant_b: "candidate", + status: "analyzed", + n_a: 240, + n_b: 255, + mean_a: 0.5, + mean_b: 0.62, + diff: 0.12, + lift: 0.24, + ci_lower: 0.08, + ci_upper: 0.16, + p_value: 0.0009, + confidence: 0.95, + significant: true, + }, + { + experiment: "reranker_v3", + metric: "retrieval_score", + variant_a: "control", + variant_b: "candidate", + status: "insufficient_data", + n_a: 12, + n_b: 9, + mean_a: 0.0, + mean_b: 0.0, + diff: 0.0, + lift: 0.0, + ci_lower: 0.0, + ci_upper: 0.0, + p_value: 1.0, + confidence: 0.95, + significant: false, + }, + ]; +} diff --git a/apps/admin-ui/src/lib/types.ts b/apps/admin-ui/src/lib/types.ts index b8409c8..c240f7c 100644 --- a/apps/admin-ui/src/lib/types.ts +++ b/apps/admin-ui/src/lib/types.ts @@ -383,3 +383,33 @@ export interface CostSnapshot { recent_cost_micros: number; baseline_cost_micros: number; } + +// --------------------------------------------------------------------------- +// A/B experiments (Step 5.7) — mirror rag_core.eval.ABAnalysisResult +// --------------------------------------------------------------------------- +export type ABStatus = "analyzed" | "insufficient_data"; + +/** One experiment's control-vs-candidate comparison (`GET /v1/status/experiments`). + * `lift` is the *relative* change of the candidate (`variant_b`) over the control + * (`variant_a`); `diff` is the absolute difference; `ci_lower`/`ci_upper` bound + * `diff` at `confidence` (a normal-approximation Welch interval); `significant` + * is true when that interval excludes 0. `insufficient_data` means either side is + * below the configured `min_samples`, so the statistics are not yet meaningful. */ +export interface ABAnalysis { + experiment: string; + metric: string; + variant_a: string; + variant_b: string; + status: ABStatus; + n_a: number; + n_b: number; + mean_a: number; + mean_b: number; + diff: number; + lift: number; + ci_lower: number; + ci_upper: number; + p_value: number; + confidence: number; + significant: boolean; +} diff --git a/apps/admin-ui/test/status.test.tsx b/apps/admin-ui/test/status.test.tsx index d4eb899..a41264a 100644 --- a/apps/admin-ui/test/status.test.tsx +++ b/apps/admin-ui/test/status.test.tsx @@ -116,4 +116,17 @@ describe("Live Status page", () => { expect(within(card).getByText("Recent spend / request")).toBeInTheDocument(); expect(within(card).getByText("$0.126")).toBeInTheDocument(); // 126_000 micro-dollars }); + + it("renders the A/B experiments card with a significant win + a collecting run (Step 5.7)", () => { + renderWithProviders(); + const card = screen.getByText("A/B experiments").closest("div.rounded-lg")! as HTMLElement; + // the significant experiment shows its lift + a "candidate wins" verdict + const winRow = within(card).getByText("router_v2").closest("li")! as HTMLElement; + expect(within(winRow).getByText("+24.0%")).toBeInTheDocument(); // seed lift 0.24 + expect(within(winRow).getByText("candidate wins")).toBeInTheDocument(); + // the under-sampled experiment shows a collecting verdict, no lift figure + const collectRow = within(card).getByText("reranker_v3").closest("li")! as HTMLElement; + expect(within(collectRow).getByText("collecting")).toBeInTheDocument(); + expect(within(collectRow).getByText(/21 samples/)).toBeInTheDocument(); // 12 + 9 + }); }); diff --git a/docs/reference/admin-ui.md b/docs/reference/admin-ui.md index 1f0cde7..ac3962b 100644 --- a/docs/reference/admin-ui.md +++ b/docs/reference/admin-ui.md @@ -13,7 +13,7 @@ state control): | Page | Route | Data source | |------|-------|-------------| | Dashboard | `/` | seed (stats, recent activity, connector health) + live system-status banner | -| Live Status | `/status` | **live** `WS /v1/status/ws` (health) + breakers / quotas / **drift** / **feedback** / **cost** cards (REST) → seed fallback | +| Live Status | `/status` | **live** `WS /v1/status/ws` (health) + breakers / quotas / **drift** / **feedback** / **cost** / **experiments** cards (REST) → seed fallback | | Metrics | `/metrics` | **live** `WS /v1/status/ws` → seed fallback (Step 3.11) | | Logs | `/logs` | **live** SSE `/v1/status/logs/stream` → seed fallback (Step 3.11) | | Query Trace | `/trace` | **live** `GET /v1/query/{id}/trace` → seed fallback (Step 5.1/5.6b) | @@ -35,10 +35,12 @@ that surface each reliability/quality subsystem — **Circuit breakers** (Step 4 one-click force-close), **Quotas & rate limits** (Step 4.5, one-click reset), **Drift monitors** (Step 5.5/5.6a, per-metric verdict + one-click *Rebaseline* to "accept the new normal"), **Feedback & satisfaction** (Step 5.4/5.6a, the -per-tenant mean satisfaction, sentiment split, and headline signal counts), and +per-tenant mean satisfaction, sentiment split, and headline signal counts), **Cost anomaly** (Step 5.6c, the per-tenant recent-spend verdict — recent vs -rolling baseline, ratio + z-score). Each card is live-wired with a seed fallback -so it always renders. +rolling baseline, ratio + z-score), and **A/B experiments** (Step 5.7d, one row +per running experiment — the lift, a verdict badge, control→candidate means, the +confidence interval, and per-arm sample counts, from `GET /v1/status/experiments`). +Each card is live-wired with a seed fallback so it always renders. The Observability surfaces **cross-link** (Step 5.6f): Live Status has a **Trace** quick-link, the **Query Trace** viewer's *Logs* button jumps to `/logs?trace=` diff --git a/docs/reference/experiments.md b/docs/reference/experiments.md index 269816d..340c71a 100644 --- a/docs/reference/experiments.md +++ b/docs/reference/experiments.md @@ -59,6 +59,15 @@ tracker.samples("routing_v2", "candidate") # the window (a copy) Each entry compares the control against the candidate variant of one experiment. Empty when experiments are disabled or none have been observed. +### Console (Step 5.7d) + +The admin **Live Status** page (`apps/admin-ui`, `/status`) renders an **A/B +experiments** card over this endpoint: one row per running experiment showing the +**lift** (relative, colour-coded), a verdict badge (*candidate wins* / *candidate +worse* / *no change* / *collecting*), the control→candidate means, the confidence +interval, and per-arm sample counts. Live-wired with a seed fallback (the demo +badge) when no gateway is configured. See [admin-ui.md](admin-ui.md). + ### CLI ```bash