feat(experiments): A/B routing — serve the candidate to a fraction of users (Step 5.7c)

TRACKER.md

@@ -16,10 +16,11 @@
 | **Last updated** | 2026-06-07 |
 | **Current phase** | Phase 5 — Eval & Observability (**6 / 7 steps**) |
 | **Overall** | **63 / 84 steps** — Phases 0–4 complete |
-| **Next action** | **Step 5.7c — A/B routing** — deterministic variant assignment that actually *serves* the candidate to a fraction of users (the first slice that can change a response). Final Phase-5 step, slices 5.7a–d. |
+| **Next action** | **Step 5.7d — Console + close-out** — admin-console A/B card surfacing `GET /v1/status/experiments` (lift + CI per experiment), plus Phase-5 close-out. Final slice of the final Phase-5 step (5.7a–d). |
 
 **Recently shipped**
 
+- **5.7c** ✅ A/B routing — `ABRouter` deterministically *serves* the candidate to a fraction of users (variant-partitioned cache, `ExperimentAssignment` response tag) — [#146](https://github.com/officialCodeWork/AgentContextOS/pull/146)
 - **5.7b** ✅ Shadow mode — observe-only candidate fan-out (`ShadowRunner`, background task) feeding the A/B tracker — [#145](https://github.com/officialCodeWork/AgentContextOS/pull/145)
 - **5.7a** ✅ A/B analyzer + experiment tracker + `GET /v1/status/experiments` dashboard — [#143](https://github.com/officialCodeWork/AgentContextOS/pull/143)
 - **5.6** ✅ Status & Metrics GUI (full build) — drift/feedback/cost cards, query-trace viewer, regression bisector, Grafana dashboards, cross-links — [#137–#142](https://github.com/officialCodeWork/AgentContextOS/pull/142)
@@ -525,7 +526,7 @@
 | 5.7 | A/B testing & shadow mode | 🚧 | — |
 | 5.7a | — A/B analyzer + tracker + dashboard | ✅ | [#143](https://github.com/officialCodeWork/AgentContextOS/pull/143) |
 | 5.7b | — Shadow mode | ✅ | [#145](https://github.com/officialCodeWork/AgentContextOS/pull/145) |
-| 5.7c | — A/B routing | ⏳ | — |
+| 5.7c | — A/B routing | ✅ | [#146](https://github.com/officialCodeWork/AgentContextOS/pull/146) |
 | 5.7d | — Console + close-out | ⏳ | — |
 
 ### 5.1 — Per-query tracing & provenance ✅ [#132](https://github.com/officialCodeWork/AgentContextOS/pull/132)
@@ -621,9 +622,14 @@
 - `cfg.experiments` gains `shadow_enabled` / `shadow_sample_rate` / `shadow_experiment` / `shadow_candidate`; doubly opt-in (`enabled` **and** `shadow_enabled`); `ragctl shadow`; 27 tests
 - [ADR-0032](docs/adr/ADR-0032-ab-testing-shadow-mode.md), [reference/experiments.md](docs/reference/experiments.md), [architecture/ab-shadow-mode.md](docs/architecture/ab-shadow-mode.md)
 
-#### 5.7c — A/B routing ⏳
+#### 5.7c — A/B routing ✅ [#146](https://github.com/officialCodeWork/AgentContextOS/pull/146)
 
-- **Next up.** Deterministic variant assignment that serves the candidate to a fraction of users (the first slice that can change a response) and tags it
+- New `ABRouter` (`rag_gateway.experiments`, beside `ShadowRunner`) — deterministic `hash(request_id) < routing_sample_rate` variant assignment that **serves** the candidate retrieval config to the assigned fraction of users (the first slice that can change a response); `assign` / `is_candidate` / `candidate` / `record` / `tag`
+- **Served-path wiring** in `/v1/query` + `/v1/retrieve`: the candidate arm is routed **inline** via `_route_query` (one retrieval, no added latency); **not degrade-open** (a candidate failure → 502, never a silent fallback to control that would corrupt the experiment); the served arm's `outcome_metric` is recorded once per **fresh** retrieval (skipped on a cache hit)
+- **Variant-partitioned retrieval cache** — `compute_plan_hash` / `compute_params_hash` fold in the assigned `variant`, so control and candidate never share a cache entry (byte-identical key when routing is off)
+- New frozen **`ExperimentAssignment`** (`experiment` / `variant` / `is_candidate`) on the `QueryResponse` / `RetrieveResponse` `experiment` field (additive, `dist/schemas` + `dist/openapi` regenerated; REST-only — gRPC proto mirror deferred like `corpus_decision`); served variant also on the `gateway.query_complete` log
+- Same `read_chunk` PDP site (no coverage-linter entry); gated separately by `cfg.experiments.routing_enabled` (requires `enabled`) — independent of `shadow_enabled` — since routing can change a response; `ShadowCandidateConfig` generalised to `CandidateConfig` (shared by `shadow_candidate` + new `routing_candidate`); `build_app(ab_router=…)` inject seam; `ragctl ab` drives assign → serve → record → analyze; ~25 new tests; all gates green (ruff, mypy --strict, RAG001, schema/openapi-drift, proto-compat, policy-coverage)
+- [ADR-0032](docs/adr/ADR-0032-ab-testing-shadow-mode.md) (5.7c update), [reference/experiments.md](docs/reference/experiments.md), [architecture/ab-routing.md](docs/architecture/ab-routing.md)
 
 ## Phase 6 — Governance & Tenancy (Weeks 28–34) ⏳
 
@@ -787,6 +793,9 @@ Complete log of every PR. Routine Dependabot bumps are grouped; everything else
 | [#141](https://github.com/officialCodeWork/AgentContextOS/pull/141) | 2026-06-05 | feat(observability): drift + cost Prometheus metrics + Grafana dashboard (Step 5.6e) |
 | [#142](https://github.com/officialCodeWork/AgentContextOS/pull/142) | 2026-06-05 | feat(admin-ui): observability cross-links + Step 5.6 close-out (5.6f) |
 | [#143](https://github.com/officialCodeWork/AgentContextOS/pull/143) | 2026-06-05 | feat(experiments): A/B analyzer + tracker + dashboard (Step 5.7a) |
+| [#144](https://github.com/officialCodeWork/AgentContextOS/pull/144) | 2026-06-05 | docs(tracker): restructure for readability + complete PR history |
+| [#145](https://github.com/officialCodeWork/AgentContextOS/pull/145) | 2026-06-05 | feat(experiments): shadow mode — observe-only candidate fan-out (Step 5.7b) |
+| [#146](https://github.com/officialCodeWork/AgentContextOS/pull/146) | 2026-06-07 | feat(experiments): A/B routing — serve the candidate to a fraction of users (Step 5.7c) |
 | #78–#80, #116–#118 | Open | Dependabot bumps — awaiting merge |
 | #81 | Closed | Dependabot bump — superseded |
 

apps/gateway/src/rag_gateway/app.py

@@ -366,6 +366,7 @@ def build_app(
     cost_tracker: Any | None = None,
     experiment_tracker: Any | None = None,
     shadow_runner: Any | None = None,
+    ab_router: Any | None = None,
     enable_cors: bool = True,
     default_tenant_id: TenantId | None = None,
 ) -> FastAPI:
@@ -565,6 +566,13 @@ def build_app(
     # ``cfg.experiments`` when ``shadow_enabled``.
     app.state.shadow_runner = shadow_runner
 
+    # A/B routing (Step 5.7c) — deterministically assigns each query to a variant
+    # and *serves* the candidate config to the assigned fraction of users (the
+    # first experiment slice that can change a response), feeding the same
+    # tracker.  ``None`` (inert) in the plain ``build_app``; the config-driven
+    # wiring builds it from ``cfg.experiments`` when ``routing_enabled``.
+    app.state.ab_router = ab_router
+
     # Outbound webhooks (Step 3.9) — the subscription registry + the delivery
     # dispatcher behind ``/v1/webhooks/subscriptions`` and the
     # ``ingest.completed`` emission.  Defaults: in-memory store + an HTTP

apps/gateway/src/rag_gateway/cache_keys.py

@@ -35,16 +35,25 @@ def compute_plan_hash(
     corpus_ids: list[str],
     top_k: int,
     filters: dict[str, Any],
+    variant: str | None = None,
 ) -> str:
-    """Return a stable SHA-256 hex digest over the retrieval-determining inputs."""
-    payload = {
+    """Return a stable SHA-256 hex digest over the retrieval-determining inputs.
+
+    ``variant`` partitions the cache by A/B-routing arm (Step 5.7c): the control
+    and candidate configs retrieve different chunks for the same query, so they
+    must never share a cache entry.  Omitted (``None``) when A/B routing is off,
+    which leaves the key byte-identical to the pre-5.7c hash.
+    """
+    payload: dict[str, Any] = {
         "t": tenant_id,
         "p": principal_id,
         "q": query,
         "c": sorted(corpus_ids),
         "k": top_k,
         "f": filters,
     }
+    if variant is not None:
+        payload["v"] = variant
     blob = json.dumps(payload, sort_keys=True, separators=(",", ":"), default=str)
     return hashlib.sha256(blob.encode("utf-8")).hexdigest()
 
@@ -56,20 +65,28 @@ def compute_params_hash(
     corpus_ids: list[str],
     top_k: int,
     filters: dict[str, Any],
+    variant: str | None = None,
 ) -> str:
     """SHA-256 over every retrieval-determining input *except the query text*.
 
     This is the L2 semantic-cache ``scope``: a similar query may reuse a prior
     result only when these parameters match exactly, so the semantic tier can
     never serve a result built with a different top_k / filter / corpus / ACL.
+
+    ``variant`` partitions the scope by A/B-routing arm (Step 5.7c) for the same
+    reason as :func:`compute_plan_hash` — the semantic tier must not serve a
+    control result to a candidate-assigned request (or vice versa).  Omitted when
+    routing is off, leaving the scope byte-identical to the pre-5.7c value.
     """
-    payload = {
+    payload: dict[str, Any] = {
         "t": tenant_id,
         "p": principal_id,
         "c": sorted(corpus_ids),
         "k": top_k,
         "f": filters,
     }
+    if variant is not None:
+        payload["v"] = variant
     blob = json.dumps(payload, sort_keys=True, separators=(",", ":"), default=str)
     return hashlib.sha256(blob.encode("utf-8")).hexdigest()
 

apps/gateway/src/rag_gateway/experiments.py

@@ -1,31 +1,41 @@
-"""Shadow-mode A/B fan-out (Step 5.7b).
-
-On a sampled fraction of live queries, run a **candidate** retriever alongside
-the served (**control**) path — *observe-only*, never affecting the response —
-and feed both variants' outcome metric into the
-:class:`~rag_observability.experiments.ABExperimentTracker`, which the dashboard
-(``GET /v1/status/experiments``) compares with the pure
-``analyze_ab_experiment`` (lift + confidence interval, Step 5.7a).
-
-Design (mirrors the Phase-5 observe-only features — drift, cost):
-
-* **Never alters the request.** The candidate retrieval is scheduled as a
-  FastAPI ``BackgroundTask``, so it runs *after* the response is sent — the
-  served path's latency is untouched.
-* **Degrade-open.** Any failure in the candidate retrieval is logged
+"""Shadow-mode fan-out (Step 5.7b) + A/B routing (Step 5.7c).
+
+Two ways the gateway compares a **candidate** retrieval config against the live
+**control** on real traffic, both feeding the
+:class:`~rag_observability.experiments.ABExperimentTracker` the dashboard
+(``GET /v1/status/experiments``) reads through the pure ``analyze_ab_experiment``
+(lift + confidence interval, Step 5.7a):
+
+* :class:`ShadowRunner` (5.7b) is **observe-only** — it runs the candidate
+  *alongside* the served control on a sampled fraction of queries, in the
+  background, and never affects the response.
+* :class:`ABRouter` (5.7c) is **live routing** — it deterministically assigns
+  each query to one variant and, when that variant is the candidate, **serves**
+  the candidate's retrieval to the user.  This is the first slice that can change
+  which config a user gets, so it is separately opt-in (``routing_enabled``).
+
+Shared design (mirrors the Phase-5 observe-only features — drift, cost):
+
+* **Deterministic assignment.** Whether a query is shadowed / which variant it is
+  routed to is a pure hash of its ``request_id`` against the sample rate — no RNG
+  in the hot path, so the decision is reproducible and testable.
+* **Degrade-open (shadow).** A candidate failure in the *shadow* path is logged
   (``experiment.shadow_failed``) and swallowed; the response was already sent.
-* **Deterministic sampling.** Whether a query is shadowed is a pure hash of its
-  ``request_id`` against the sample rate, so the decision is reproducible and
-  testable (no RNG in the hot path).
-
-The candidate is any :class:`~rag_retrieval.router.SupportsRoute` (a second
-:class:`~rag_retrieval.router.RetrievalRouter` with different fusion weights is
-the config-driven default; production injects one over the real backends).
-A/B *routing* — actually serving the candidate to a fraction of users — is the
-next slice (5.7c); this slice is strictly observe-only.
-
-See [docs/reference/experiments.md](../../../../docs/reference/experiments.md)
-and [docs/architecture/ab-shadow-mode.md](../../../../docs/architecture/ab-shadow-mode.md).
+  A/B *routing* serves the candidate inline, so a candidate failure surfaces as a
+  normal retrieval error (the variant is the served path) — there is nothing to
+  degrade *to* without silently swapping the user back to control mid-experiment.
+* **One ``read_chunk`` PDP site.** The candidate is any
+  :class:`~rag_retrieval.router.SupportsRoute` and routes through the same
+  ``HybridRetriever.retrieve`` (the canonical policy site), so ACLs are enforced
+  on the candidate exactly as on the control — no coverage-linter entry needed.
+
+The candidate is a second :class:`~rag_retrieval.router.RetrievalRouter` with
+different fusion weights in the config-driven default; production injects one over
+the real backends via ``build_app(shadow_runner=…)`` / ``build_app(ab_router=…)``.
+
+See [docs/reference/experiments.md](../../../../docs/reference/experiments.md),
+[docs/architecture/ab-shadow-mode.md](../../../../docs/architecture/ab-shadow-mode.md),
+and [docs/architecture/ab-routing.md](../../../../docs/architecture/ab-routing.md).
 """
 
 from __future__ import annotations
@@ -34,12 +44,13 @@
 from typing import Any
 
 from rag_core import get_logger
+from rag_core.gateway_types import ExperimentAssignment
 from rag_core.types import RequestContext
 from rag_observability.experiments import ABExperimentTracker
 
 _log = get_logger(__name__)
 
-__all__ = ["ShadowRunner", "outcome_metric"]
+__all__ = ["ABRouter", "ShadowRunner", "outcome_metric"]
 
 # Largest value of an 8-hex-digit digest prefix — the denominator that maps a
 # hash into the half-open unit interval ``[0, 1)`` for sampling.
@@ -161,3 +172,80 @@ async def run(
                     "error": repr(exc),
                 },
             )
+
+
+class ABRouter:
+    """Deterministic A/B variant assignment that *serves* the candidate (Step 5.7c).
+
+    Unlike :class:`ShadowRunner` (observe-only, background), ``ABRouter`` is on
+    the served path and can change the response: :meth:`assign` deterministically
+    hashes the ``request_id`` to a variant, and when that variant is the
+    candidate the gateway routes retrieval through :attr:`candidate` (a second
+    :class:`~rag_retrieval.router.SupportsRoute`) instead of the control router —
+    so the configured fraction of users is actually served the candidate config.
+
+    The runner stays a thin policy object: it *decides* the variant, *holds* the
+    candidate router, *records* the served variant's :func:`outcome_metric` into
+    the tracker, and *tags* the response.  The query handler owns the wiring (it
+    is the one place that knows the served control path), so ``ABRouter`` makes no
+    governed SPI call of its own — the candidate's own ``read_chunk`` PDP site
+    enforces ACLs.  Stateless beyond its dependencies, so one instance is shared
+    across requests (the tracker it feeds is itself thread-safe).
+    """
+
+    def __init__(
+        self,
+        *,
+        candidate_router: Any,  # SupportsRoute — Any to match the gateway's deps style.
+        tracker: ABExperimentTracker,
+        sample_rate: float = 0.1,
+        experiment: str = "ab",
+        control_variant: str = "control",
+        candidate_variant: str = "candidate",
+    ) -> None:
+        self._candidate = candidate_router
+        self._tracker = tracker
+        self._sample_rate = max(0.0, min(1.0, sample_rate))
+        self._experiment = experiment
+        self._control = control_variant
+        self._candidate_variant = candidate_variant
+
+    @property
+    def experiment(self) -> str:
+        return self._experiment
+
+    @property
+    def candidate(self) -> Any:
+        """The candidate :class:`~rag_retrieval.router.SupportsRoute` to serve."""
+        return self._candidate
+
+    def assign(self, request_id: str) -> str:
+        """Return the variant label for ``request_id`` (deterministic).
+
+        The candidate is served to the ``sample_rate`` fraction of requests whose
+        ``request_id`` hashes into ``[0, sample_rate)``; everyone else gets the
+        control.  ``rate <= 0`` assigns everyone to control (routing a no-op);
+        ``rate >= 1`` assigns everyone to the candidate.
+        """
+        return self._candidate_variant if _sampled(request_id, self._sample_rate) else self._control
+
+    def is_candidate(self, variant: str) -> bool:
+        """Whether ``variant`` is the candidate arm (the one served the alternative)."""
+        return variant == self._candidate_variant
+
+    def record(self, variant: str, chunk_refs: list[Any]) -> None:
+        """Record the served ``variant``'s :func:`outcome_metric` into the tracker.
+
+        Called once per freshly-retrieved query (the served arm only), so the
+        per-``(experiment, variant)`` windows the dashboard analyses fill from
+        real served traffic.  An O(1) append — never raises into the request.
+        """
+        self._tracker.observe(self._experiment, variant, outcome_metric(chunk_refs))
+
+    def tag(self, variant: str) -> ExperimentAssignment:
+        """Build the :class:`ExperimentAssignment` stamped on the response."""
+        return ExperimentAssignment(
+            experiment=self._experiment,
+            variant=variant,
+            is_candidate=self.is_candidate(variant),
+        )