officialCodeWork · officialCodeWork · Jun 7, 2026 · Jun 7, 2026
diff --git a/TRACKER.md b/TRACKER.md
@@ -14,12 +14,13 @@
 | | |
 |---|---|
 | **Last updated** | 2026-06-07 |
-| **Current phase** | Phase 5 — Eval & Observability (**6 / 7 steps**) |
-| **Overall** | **63 / 84 steps** — Phases 0–4 complete |
-| **Next action** | **Step 5.7d — Console + close-out** — admin-console A/B card surfacing `GET /v1/status/experiments` (lift + CI per experiment), plus Phase-5 close-out. Final slice of the final Phase-5 step (5.7a–d). |
+| **Current phase** | **Phase 5 — Eval & Observability ✅ complete (7 / 7 steps)** → starting Phase 6 |
+| **Overall** | **64 / 84 steps** — Phases 0–5 complete |
+| **Next action** | **Step 6.1 — Logical multi-tenancy** (Phase 6 — Governance & Tenancy): namespace isolation; per-tenant config, quotas, ACLs in `rag.yaml`. First step of the governance phase. |
 
 **Recently shipped**
 
+- **5.7d** ✅ Experiments console card + Phase-5 close-out — admin Live-Status A/B card over `GET /v1/status/experiments` (lift + CI per experiment) — [#147](https://github.com/officialCodeWork/AgentContextOS/pull/147)
 - **5.7c** ✅ A/B routing — `ABRouter` deterministically *serves* the candidate to a fraction of users (variant-partitioned cache, `ExperimentAssignment` response tag) — [#146](https://github.com/officialCodeWork/AgentContextOS/pull/146)
 - **5.7b** ✅ Shadow mode — observe-only candidate fan-out (`ShadowRunner`, background task) feeding the A/B tracker — [#145](https://github.com/officialCodeWork/AgentContextOS/pull/145)
 - **5.7a** ✅ A/B analyzer + experiment tracker + `GET /v1/status/experiments` dashboard — [#143](https://github.com/officialCodeWork/AgentContextOS/pull/143)
@@ -52,10 +53,10 @@
 | 2 | Retrieval Engine | 11 | **11** | 0 |
 | 3 | Gateway & Agent Runtime | 11 | **11** | 0 |
 | 4 | Reliability | 6 | **6** | 0 |
-| 5 | Eval & Observability | 7 | **6** | 1 |
+| 5 | Eval & Observability | 7 | **7** | 0 |
 | 6 | Governance & Tenancy | 10 | 0 | 10 |
 | 7 | Pilot, Harden, GA | 10 | 0 | 10 |
-| **Total** | | **84** | **63** | **21** |
+| **Total** | | **84** | **64** | **20** |
 
 ---
 
@@ -507,7 +508,7 @@
 - One reusable engine `rag_gateway.perf` (`measure_gateway_overhead` + `profile_gateway`) powers gate + harness + `ragctl perf`; `tests/contract/budgets.py` made real (drift-guarded); load test = in-process `eval/gateway_load_v0` + Locust `locustfile.py`
 - ~14 tests; [ADR-0025](docs/adr/ADR-0025-latency-load-testing.md), [reference/perf.md](docs/reference/perf.md), [architecture/latency.md](docs/architecture/latency.md)
 
-## Phase 5 — Eval & Observability (Weeks 22–28) 🚧 (6 / 7)
+## Phase 5 — Eval & Observability (Weeks 22–28) ✅
 
 | Step | Title | Status | PR |
 |------|-------|:------:|----|
@@ -523,11 +524,11 @@
 | 5.6d | — Regression bisector | ✅ | [#140](https://github.com/officialCodeWork/AgentContextOS/pull/140) |
 | 5.6e | — Grafana dashboards | ✅ | [#141](https://github.com/officialCodeWork/AgentContextOS/pull/141) |
 | 5.6f | — Cross-links + close-out | ✅ | [#142](https://github.com/officialCodeWork/AgentContextOS/pull/142) |
-| 5.7 | A/B testing & shadow mode | 🚧 | — |
+| 5.7 | A/B testing & shadow mode | ✅ | [#143–#147](https://github.com/officialCodeWork/AgentContextOS/pull/147) |
 | 5.7a | — A/B analyzer + tracker + dashboard | ✅ | [#143](https://github.com/officialCodeWork/AgentContextOS/pull/143) |
 | 5.7b | — Shadow mode | ✅ | [#145](https://github.com/officialCodeWork/AgentContextOS/pull/145) |
 | 5.7c | — A/B routing | ✅ | [#146](https://github.com/officialCodeWork/AgentContextOS/pull/146) |
-| 5.7d | — Console + close-out | ⏳ | — |
+| 5.7d | — Console + close-out | ✅ | [#147](https://github.com/officialCodeWork/AgentContextOS/pull/147) |
 
 ### 5.1 — Per-query tracing & provenance ✅ [#132](https://github.com/officialCodeWork/AgentContextOS/pull/132)
 
@@ -602,10 +603,10 @@
 - Wires the observability surfaces together: the Trace viewer's *Logs* button → `/logs?trace=<id>` (clearable violet trace-chip), a log detail drawer → *View this trace's logs*, a **Trace** quick-link on Live Status
 - Admin-ui only; 1 vitest cross-link spec; closes Step 5.6
 
-### 5.7 — A/B testing & shadow mode 🚧
+### 5.7 — A/B testing & shadow mode ✅
 
-- Final Phase-5 step, delivered in vertical slices **5.7a–d**: analyzer + tracker + dashboard (5.7a ✅), shadow mode (5.7b), A/B routing (5.7c), console + close-out (5.7d)
-- Shadow N% of live queries; A/B variant routing; statistical analyzer with lift + confidence interval
+- Final Phase-5 step, delivered in vertical slices **5.7a–d**: analyzer + tracker + dashboard (5.7a ✅), shadow mode (5.7b ✅), A/B routing (5.7c ✅), console + close-out (5.7d ✅)
+- Shadow N% of live queries; A/B variant routing; statistical analyzer with lift + confidence interval — **all delivered; Phase 5 complete**
 
 #### 5.7a — A/B analyzer + tracker + dashboard ✅ [#143](https://github.com/officialCodeWork/AgentContextOS/pull/143)
 
@@ -631,6 +632,13 @@
 - Same `read_chunk` PDP site (no coverage-linter entry); gated separately by `cfg.experiments.routing_enabled` (requires `enabled`) — independent of `shadow_enabled` — since routing can change a response; `ShadowCandidateConfig` generalised to `CandidateConfig` (shared by `shadow_candidate` + new `routing_candidate`); `build_app(ab_router=…)` inject seam; `ragctl ab` drives assign → serve → record → analyze; ~25 new tests; all gates green (ruff, mypy --strict, RAG001, schema/openapi-drift, proto-compat, policy-coverage)
 - [ADR-0032](docs/adr/ADR-0032-ab-testing-shadow-mode.md) (5.7c update), [reference/experiments.md](docs/reference/experiments.md), [architecture/ab-routing.md](docs/architecture/ab-routing.md)
 
+#### 5.7d — Console + close-out ✅ [#147](https://github.com/officialCodeWork/AgentContextOS/pull/147)
+
+- New **A/B experiments** card on the admin **Live Status** page (`apps/admin-ui`) surfacing `GET /v1/status/experiments` — one row per running experiment with the **lift** (relative, colour-coded), a verdict badge (**candidate wins** / **candidate worse** / **no change** / **collecting**), control→candidate means, the confidence interval, and per-arm sample counts; live-vs-seed via `useLive` (`fetchExperiments` + `seedExperiments`), demo badge when no gateway
+- `ABAnalysis` TS type mirrors `rag_core.eval.ABAnalysisResult`; admin-ui gates green (tsc / eslint / 34 vitest / `next build`); browser-verified both the significant-win + insufficient-data states render
+- **Phase-5 close-out:** Step 5.7 ✅ → **Phase 5 complete (7 / 7)**; deferred items remain documented (per-tenant drift / per-dimension embedding PSI; feedback/breaker/quota Grafana export + Loki-events dashboard; gRPC proto mirror of `corpus_decision` + `experiment`; sequential / multi-metric experiments)
+- [reference/experiments.md](docs/reference/experiments.md), [reference/admin-ui.md](docs/reference/admin-ui.md)
+
 ## Phase 6 — Governance & Tenancy (Weeks 28–34) ⏳
 
 | Step | Title | Status | Planned deliverables |
@@ -796,6 +804,7 @@ Complete log of every PR. Routine Dependabot bumps are grouped; everything else
 | [#144](https://github.com/officialCodeWork/AgentContextOS/pull/144) | 2026-06-05 | docs(tracker): restructure for readability + complete PR history |
 | [#145](https://github.com/officialCodeWork/AgentContextOS/pull/145) | 2026-06-05 | feat(experiments): shadow mode — observe-only candidate fan-out (Step 5.7b) |
 | [#146](https://github.com/officialCodeWork/AgentContextOS/pull/146) | 2026-06-07 | feat(experiments): A/B routing — serve the candidate to a fraction of users (Step 5.7c) |
+| [#147](https://github.com/officialCodeWork/AgentContextOS/pull/147) | 2026-06-07 | feat(admin-ui): A/B experiments console card + Phase-5 close-out (Step 5.7d) |
 | #78–#80, #116–#118 | Open | Dependabot bumps — awaiting merge |
 | #81 | Closed | Dependabot bump — superseded |
 

diff --git a/apps/admin-ui/src/app/status/page.tsx b/apps/admin-ui/src/app/status/page.tsx
@@ -3,6 +3,7 @@
 import {
   Activity,
   ChevronRight,
+  FlaskConical,
   Gauge,
   MessageSquare,
   Route,
@@ -23,6 +24,7 @@ import {
   fetchConnectorStatus,
   fetchCost,
   fetchDrift,
+  fetchExperiments,
   fetchFeedback,
   fetchQuotas,
   forceCloseBreaker,
@@ -35,11 +37,13 @@ import {
   seedConnectorStatus,
   seedCost,
   seedDrift,
+  seedExperiments,
   seedFeedback,
   seedQuotas,
 } from "@/lib/mock";
 import { relTime } from "@/lib/time";
 import type {
+  ABAnalysis,
   BreakerState,
   BreakerView,
   ComponentHealth,
@@ -570,6 +574,99 @@ function CostCard() {
   );
 }
 
+/** A relative figure as a signed percentage (e.g. `+24.0%`). */
+function liftPct(x: number): string {
+  return `${x >= 0 ? "+" : ""}${(x * 100).toFixed(1)}%`;
+}
+
+/** Verdict badge for one A/B comparison: collecting → still gathering samples;
+ *  otherwise a significant candidate win / regression, or no detectable change. */
+function abVerdict(r: ABAnalysis): {
+  label: string;
+  variant: "success" | "error" | "secondary";
+} {
+  if (r.status === "insufficient_data") return { label: "collecting", variant: "secondary" };
+  if (!r.significant) return { label: "no change", variant: "secondary" };
+  return r.diff >= 0
+    ? { label: "candidate wins", variant: "success" }
+    : { label: "candidate worse", variant: "error" };
+}
+
+function liftColor(r: ABAnalysis): string {
+  if (r.status === "insufficient_data" || !r.significant) return "text-muted-foreground";
+  return r.diff >= 0
+    ? "text-emerald-600 dark:text-emerald-400"
+    : "text-red-600 dark:text-red-400";
+}
+
+function signed(x: number): string {
+  return `${x >= 0 ? "+" : ""}${x.toFixed(3)}`;
+}
+
+/** A/B experiments (Step 5.7) — control-vs-candidate lift + CI per running experiment. */
+function ExperimentsCard() {
+  const { rows } = useLive<ABAnalysis>(seedExperiments, fetchExperiments, {
+    fallbackOnEmpty: true,
+  });
+
+  return (
+    <Card className="mt-4">
+      <div className="flex items-center justify-between border-b px-5 py-3">
+        <div className="text-sm font-semibold">A/B experiments</div>
+        <div className="text-xs text-muted-foreground">
+          {rows.length > 0 ? `${rows.length} running` : "control vs candidate"}
+        </div>
+      </div>
+      {rows.length === 0 ? (
+        <div className="px-5 py-8 text-center text-sm text-muted-foreground">
+          <FlaskConical className="mx-auto mb-2 h-5 w-5" />
+          No experiments running.
+        </div>
+      ) : (
+        <ul className="divide-y">
+          {rows.map((r) => {
+            const verdict = abVerdict(r);
+            const collecting = r.status === "insufficient_data";
+            return (
+              <li key={r.experiment} className="px-5 py-3">
+                <div className="flex items-center justify-between gap-3">
+                  <div className="min-w-0">
+                    <div className="truncate text-sm font-medium">{r.experiment}</div>
+                    <div className="truncate text-xs text-muted-foreground">{r.metric}</div>
+                  </div>
+                  <Badge variant={verdict.variant}>{verdict.label}</Badge>
+                </div>
+                <div className="mt-2 flex items-end justify-between gap-3">
+                  <div>
+                    <div className={cn("text-2xl font-semibold tabular-nums", liftColor(r))}>
+                      {collecting ? "—" : liftPct(r.lift)}
+                    </div>
+                    <div className="text-xs text-muted-foreground">lift vs control</div>
+                  </div>
+                  <div className="text-right text-xs tabular-nums text-muted-foreground">
+                    {collecting ? (
+                      <div>{r.n_a + r.n_b} samples · need more</div>
+                    ) : (
+                      <>
+                        <div>
+                          {r.variant_a} {r.mean_a.toFixed(3)} → {r.variant_b} {r.mean_b.toFixed(3)}
+                        </div>
+                        <div>
+                          CI [{signed(r.ci_lower)}, {signed(r.ci_upper)}] · n={r.n_a}/{r.n_b}
+                        </div>
+                      </>
+                    )}
+                  </div>
+                </div>
+              </li>
+            );
+          })}
+        </ul>
+      )}
+    </Card>
+  );
+}
+
 export default function LiveStatusPage() {
   const router = useRouter();
   const { health, source, connected } = useStatusStream();
@@ -704,6 +801,7 @@ export default function LiveStatusPage() {
       <DriftCard />
       <FeedbackCard />
       <CostCard />
+      <ExperimentsCard />
     </div>
   );
 }
diff --git a/apps/admin-ui/src/lib/api.ts b/apps/admin-ui/src/lib/api.ts
@@ -7,6 +7,7 @@
 
 import { GATEWAY_URL } from "./config";
 import type {
+  ABAnalysis,
   BreakerView,
   ConnectorStatusLive,
   Corpus,
@@ -226,6 +227,20 @@ export async function fetchCost(tenantId: string): Promise<CostSnapshot[]> {
   return data.status === "insufficient_data" ? [] : [data];
 }
 
+// ---------------------------------------------------------------------------
+// A/B experiments (Step 5.7)
+// ---------------------------------------------------------------------------
+/** One control-vs-candidate comparison per running experiment. Empty when
+ *  experiments are disabled or none have been observed, so `useLive` falls back
+ *  to seed data (with the demo badge) rather than rendering an empty card. */
+export async function fetchExperiments(tenantId: string): Promise<ABAnalysis[]> {
+  const data = await request<{ experiments: ABAnalysis[]; total: number }>(
+    tenantId,
+    "/v1/status/experiments",
+  );
+  return data.experiments;
+}
+
 // ---------------------------------------------------------------------------
 // Per-query trace & provenance (Step 5.1)
 // ---------------------------------------------------------------------------

diff --git a/apps/admin-ui/src/lib/mock.ts b/apps/admin-ui/src/lib/mock.ts
@@ -1,6 +1,7 @@
 /** Seed data for the console — ported from the design handoff (deterministic). */
 
 import type {
+  ABAnalysis,
   ApiKey,
   AuditEvent,
   BreakerView,
@@ -563,3 +564,45 @@ export function seedFeedback(tenantId: string): FeedbackStats[] {
     },
   ];
 }
+
+/** A/B experiments (Step 5.7) — one significant win + one still-collecting comparison. */
+export function seedExperiments(_tenantId: string): ABAnalysis[] {
+  return [
+    {
+      experiment: "router_v2",
+      metric: "retrieval_score",
+      variant_a: "control",
+      variant_b: "candidate",
+      status: "analyzed",
+      n_a: 240,
+      n_b: 255,
+      mean_a: 0.5,
+      mean_b: 0.62,
+      diff: 0.12,
+      lift: 0.24,
+      ci_lower: 0.08,
+      ci_upper: 0.16,
+      p_value: 0.0009,
+      confidence: 0.95,
+      significant: true,
+    },
+    {
+      experiment: "reranker_v3",
+      metric: "retrieval_score",
+      variant_a: "control",
+      variant_b: "candidate",
+      status: "insufficient_data",
+      n_a: 12,
+      n_b: 9,
+      mean_a: 0.0,
+      mean_b: 0.0,
+      diff: 0.0,
+      lift: 0.0,
+      ci_lower: 0.0,
+      ci_upper: 0.0,
+      p_value: 1.0,
+      confidence: 0.95,
+      significant: false,
+    },
+  ];
+}
diff --git a/apps/admin-ui/src/lib/types.ts b/apps/admin-ui/src/lib/types.ts
@@ -383,3 +383,33 @@ export interface CostSnapshot {
   recent_cost_micros: number;
   baseline_cost_micros: number;
 }
+
+// ---------------------------------------------------------------------------
+// A/B experiments (Step 5.7) — mirror rag_core.eval.ABAnalysisResult
+// ---------------------------------------------------------------------------
+export type ABStatus = "analyzed" | "insufficient_data";
+
+/** One experiment's control-vs-candidate comparison (`GET /v1/status/experiments`).
+ *  `lift` is the *relative* change of the candidate (`variant_b`) over the control
+ *  (`variant_a`); `diff` is the absolute difference; `ci_lower`/`ci_upper` bound
+ *  `diff` at `confidence` (a normal-approximation Welch interval); `significant`
+ *  is true when that interval excludes 0. `insufficient_data` means either side is
+ *  below the configured `min_samples`, so the statistics are not yet meaningful. */
+export interface ABAnalysis {
+  experiment: string;
+  metric: string;
+  variant_a: string;
+  variant_b: string;
+  status: ABStatus;
+  n_a: number;
+  n_b: number;
+  mean_a: number;
+  mean_b: number;
+  diff: number;
+  lift: number;
+  ci_lower: number;
+  ci_upper: number;
+  p_value: number;
+  confidence: number;
+  significant: boolean;
+}
diff --git a/apps/admin-ui/test/status.test.tsx b/apps/admin-ui/test/status.test.tsx
@@ -116,4 +116,17 @@ describe("Live Status page", () => {
     expect(within(card).getByText("Recent spend / request")).toBeInTheDocument();
     expect(within(card).getByText("$0.126")).toBeInTheDocument(); // 126_000 micro-dollars
   });
+
+  it("renders the A/B experiments card with a significant win + a collecting run (Step 5.7)", () => {
+    renderWithProviders(<LiveStatusPage />);
+    const card = screen.getByText("A/B experiments").closest("div.rounded-lg")! as HTMLElement;
+    // the significant experiment shows its lift + a "candidate wins" verdict
+    const winRow = within(card).getByText("router_v2").closest("li")! as HTMLElement;
+    expect(within(winRow).getByText("+24.0%")).toBeInTheDocument(); // seed lift 0.24
+    expect(within(winRow).getByText("candidate wins")).toBeInTheDocument();
+    // the under-sampled experiment shows a collecting verdict, no lift figure
+    const collectRow = within(card).getByText("reranker_v3").closest("li")! as HTMLElement;
+    expect(within(collectRow).getByText("collecting")).toBeInTheDocument();
+    expect(within(collectRow).getByText(/21 samples/)).toBeInTheDocument(); // 12 + 9
+  });
 });
diff --git a/docs/reference/admin-ui.md b/docs/reference/admin-ui.md
@@ -13,7 +13,7 @@ state control):
 | Page | Route | Data source |
 |------|-------|-------------|
 | Dashboard | `/` | seed (stats, recent activity, connector health) + live system-status banner |
-| Live Status | `/status` | **live** `WS /v1/status/ws` (health) + breakers / quotas / **drift** / **feedback** / **cost** cards (REST) → seed fallback |
+| Live Status | `/status` | **live** `WS /v1/status/ws` (health) + breakers / quotas / **drift** / **feedback** / **cost** / **experiments** cards (REST) → seed fallback |
 | Metrics | `/metrics` | **live** `WS /v1/status/ws` → seed fallback (Step 3.11) |
 | Logs | `/logs` | **live** SSE `/v1/status/logs/stream` → seed fallback (Step 3.11) |
 | Query Trace | `/trace` | **live** `GET /v1/query/{id}/trace` → seed fallback (Step 5.1/5.6b) |
@@ -35,10 +35,12 @@ that surface each reliability/quality subsystem — **Circuit breakers** (Step 4
 one-click force-close), **Quotas & rate limits** (Step 4.5, one-click reset),
 **Drift monitors** (Step 5.5/5.6a, per-metric verdict + one-click *Rebaseline* to
 "accept the new normal"), **Feedback & satisfaction** (Step 5.4/5.6a, the
-per-tenant mean satisfaction, sentiment split, and headline signal counts), and
+per-tenant mean satisfaction, sentiment split, and headline signal counts),
 **Cost anomaly** (Step 5.6c, the per-tenant recent-spend verdict — recent vs
-rolling baseline, ratio + z-score). Each card is live-wired with a seed fallback
-so it always renders.
+rolling baseline, ratio + z-score), and **A/B experiments** (Step 5.7d, one row
+per running experiment — the lift, a verdict badge, control→candidate means, the
+confidence interval, and per-arm sample counts, from `GET /v1/status/experiments`).
+Each card is live-wired with a seed fallback so it always renders.
 
 The Observability surfaces **cross-link** (Step 5.6f): Live Status has a **Trace**
 quick-link, the **Query Trace** viewer's *Logs* button jumps to `/logs?trace=<id>`