feat(tenancy): physical tenancy — dedicated index per tenant (Step 6.2)

TRACKER.md

@@ -14,12 +14,13 @@
 | | |
 |---|---|
 | **Last updated** | 2026-06-08 |
-| **Current phase** | Phase 6 — Governance & Tenancy (**1 / 10 steps**) |
-| **Overall** | **65 / 84 steps** — Phases 0–5 complete |
-| **Next action** | **Step 6.2 — Physical tenancy (dedicated index)**: dedicated vector index per tenant; cross-tenant probe gate. Builds on the 6.1 namespace primitive. |
+| **Current phase** | Phase 6 — Governance & Tenancy (**2 / 10 steps**) |
+| **Overall** | **66 / 84 steps** — Phases 0–5 complete |
+| **Next action** | **Step 6.3 — ACL push-down at retrieval**: inject the ACL filter into every vector / BM25 / graph query; `acl.egress_denied` event. Consumes the per-tenant `acl_labels` resolved in 6.1. |
 
 **Recently shipped**
 
+- **6.2** ✅ Physical tenancy — per-tenant *dedicated* vector index/collection; `TenantConfig.dedicated_index` → `ctx.physical_index` → backend `<base>-<key>` (Noop/Pinecone/Qdrant); cross-tenant probe gate — [#149](https://github.com/officialCodeWork/AgentContextOS/pull/149)
 - **6.1** ✅ Logical multi-tenancy — `TenantResolver` resolves per-tenant `rag.yaml` config (namespace / PII / ACL labels) → `TenantSettings` applied at the gateway boundary; `RequestContext.namespace`; `GET /v1/status/tenant`; `ragctl tenant` — [#148](https://github.com/officialCodeWork/AgentContextOS/pull/148)
 - **5.7d** ✅ Experiments console card + Phase-5 close-out — admin Live-Status A/B card over `GET /v1/status/experiments` (lift + CI per experiment) — [#147](https://github.com/officialCodeWork/AgentContextOS/pull/147)
 - **5.7c** ✅ A/B routing — `ABRouter` deterministically *serves* the candidate to a fraction of users (variant-partitioned cache, `ExperimentAssignment` response tag) — [#146](https://github.com/officialCodeWork/AgentContextOS/pull/146)
@@ -55,9 +56,9 @@
 | 3 | Gateway & Agent Runtime | 11 | **11** | 0 |
 | 4 | Reliability | 6 | **6** | 0 |
 | 5 | Eval & Observability | 7 | **7** | 0 |
-| 6 | Governance & Tenancy | 10 | **1** | 9 |
+| 6 | Governance & Tenancy | 10 | **2** | 8 |
 | 7 | Pilot, Harden, GA | 10 | 0 | 10 |
-| **Total** | | **84** | **65** | **19** |
+| **Total** | | **84** | **66** | **18** |
 
 ---
 
@@ -640,12 +641,12 @@
 - **Phase-5 close-out:** Step 5.7 ✅ → **Phase 5 complete (7 / 7)**; deferred items remain documented (per-tenant drift / per-dimension embedding PSI; feedback/breaker/quota Grafana export + Loki-events dashboard; gRPC proto mirror of `corpus_decision` + `experiment`; sequential / multi-metric experiments)
 - [reference/experiments.md](docs/reference/experiments.md), [reference/admin-ui.md](docs/reference/admin-ui.md)
 
-## Phase 6 — Governance & Tenancy (Weeks 28–34) 🚧 (1 / 10)
+## Phase 6 — Governance & Tenancy (Weeks 28–34) 🚧 (2 / 10)
 
 | Step | Title | Status | Planned deliverables |
 |------|-------|:------:|----------------------|
 | 6.1 | Logical multi-tenancy | ✅ | [#148](https://github.com/officialCodeWork/AgentContextOS/pull/148) — `TenantResolver` → `TenantSettings` (namespace / PII / ACL labels) applied at the boundary; `RequestContext.namespace`; `GET /v1/status/tenant`; `ragctl tenant` |
-| 6.2 | Physical tenancy (dedicated index) | ⏳ | Dedicated vector index per tenant; cross-tenant probe gate |
+| 6.2 | Physical tenancy (dedicated index) | ✅ | [#149](https://github.com/officialCodeWork/AgentContextOS/pull/149) — `dedicated_index` → `ctx.physical_index` → backend `<base>-<key>` (Noop/Pinecone/Qdrant, lazy create); cross-tenant probe gate |
 | 6.3 | ACL push-down at retrieval | ⏳ | ACL filter injected into every vector / BM25 / graph query; `acl.egress_denied` event |
 | 6.4 | ACL egress verifier | ⏳ | Post-retrieval re-check; defense-in-depth; zero-ACL-violation-rate gate |
 | 6.5 | PII policies | ⏳ | Per-tenant PII enforcement (block / redact / allow); egress redaction; `pii.egress_blocked` event |
@@ -665,6 +666,15 @@
 - **Scope:** resolution + threading only — ACL push-down enforcement is 6.3, PII egress is 6.5, physical tenancy is 6.2; ~21 new tests (resolver, config, gateway, cross-tenant red-team, CLI); all gates green (ruff, mypy --strict 293 files, RAG001, schema/openapi-drift, proto-compat, policy-coverage)
 - [ADR-0033](docs/adr/ADR-0033-logical-multi-tenancy.md), [reference/tenancy.md](docs/reference/tenancy.md), [architecture/multi-tenancy.md](docs/architecture/multi-tenancy.md)
 
+### 6.2 — Physical tenancy (dedicated index) ✅ [#149](https://github.com/officialCodeWork/AgentContextOS/pull/149)
+
+- *Logical* tenancy (6.1) shares one index + a `tenant_id` filter; **physical** tenancy gives a tenant its **own** vector index/collection, so its data is separate even if a filter were bypassed.  Opt-in per tenant via **`TenantConfig.dedicated_index`** (+ optional `dedicated_index_name`)
+- Same resolve-once-at-the-boundary pattern as 6.1: the resolver computes a **`physical_index`** key (`dedicated_index_name or tenant_id`, else `None`) on `TenantSettings`, threaded onto new **`RequestContext.physical_index`**; the gateway middleware applies it
+- **Backends read only `ctx`** (graph is backends→core, never rag-config), so the decision arrives via `ctx.physical_index`; each namespaces its base under the key (**`<base>-<key>`**) and **lazily creates** the dedicated index/collection on first use, falling back to the base index when `None`.  Implemented for **Noop** (CI oracle, partitioned by `physical_index`) + **Pinecone** (dedicated index) + **Qdrant** (dedicated collection); pgvector/ES/Weaviate follow later
+- **Cross-tenant probe gate** (`tests/redteam/test_cross_tenant_dedicated_index.py`, CI, Noop) proves a dedicated tenant's data is invisible to another **independent of the tenant filter** (a matching `tenant_id` pointed at a different index still returns nothing); live backends are integration-tested
+- One instance + per-tenant derivation (**no** per-tenant backend instances, **no** SPI change); `GET /v1/status/tenant` + `ragctl tenant resolve` report `dedicated_index` / `physical_index`; `TenantConfig` fields → `rag.schema`, `TenantSettings` + `RequestContext` regenerated, `/v1/status/tenant` → `dist/openapi`; ~30 new/updated tests; all gates green
+- [ADR-0034](docs/adr/ADR-0034-physical-multi-tenancy.md), [reference/tenancy.md](docs/reference/tenancy.md#physical-tenancy-step-62), [architecture/multi-tenancy.md](docs/architecture/multi-tenancy.md)
+
 ---
 
 ## Phase 7 — Pilot, Harden, GA (Weeks 34–40) ⏳
@@ -817,6 +827,7 @@ Complete log of every PR. Routine Dependabot bumps are grouped; everything else
 | [#146](https://github.com/officialCodeWork/AgentContextOS/pull/146) | 2026-06-07 | feat(experiments): A/B routing — serve the candidate to a fraction of users (Step 5.7c) |
 | [#147](https://github.com/officialCodeWork/AgentContextOS/pull/147) | 2026-06-07 | feat(admin-ui): A/B experiments console card + Phase-5 close-out (Step 5.7d) |
 | [#148](https://github.com/officialCodeWork/AgentContextOS/pull/148) | 2026-06-08 | feat(tenancy): logical multi-tenancy — per-tenant config resolution (Step 6.1) |
+| [#149](https://github.com/officialCodeWork/AgentContextOS/pull/149) | 2026-06-08 | feat(tenancy): physical tenancy — dedicated index per tenant (Step 6.2) |
 | #78–#80, #116–#118 | Open | Dependabot bumps — awaiting merge |
 | #81 | Closed | Dependabot bump — superseded |
 

apps/gateway/src/rag_gateway/middleware.py

@@ -167,10 +167,12 @@ async def build_gateway_context(
     if principal is not None and tenant_id is not None:
         namespace = str(tenant_id)
         pii_policy = None
+        physical_index: str | None = None
         if tenant_resolver is not None:
             tenant_settings = tenant_resolver.resolve(str(tenant_id))
             namespace = tenant_settings.namespace
             pii_policy = tenant_settings.pii_policy
+            physical_index = tenant_settings.physical_index
             if tenant_settings.acl_labels:
                 principal = principal.model_copy(
                     update={"acl_labels": principal.acl_labels | tenant_settings.acl_labels}
@@ -180,6 +182,7 @@ async def build_gateway_context(
             tenant_id=tenant_id,
             principal=principal,
             namespace=namespace,
+            physical_index=physical_index,
             **({"pii_policy": pii_policy} if pii_policy is not None else {}),
             trace=trace,
         )

apps/gateway/src/rag_gateway/status.py

@@ -332,6 +332,10 @@ class TenantStatusResponse(BaseModel):
     namespace: str = ""
     pii_action: str = "redact"
     acl_labels: list[str] = Field(default_factory=list)
+    # Physical tenancy (Step 6.2): whether this tenant has a dedicated index, and
+    # its key (``None`` when it shares the base index).
+    dedicated_index: bool = False
+    physical_index: str | None = None
 
 
 class ExperimentsStatusResponse(BaseModel):
@@ -898,6 +902,8 @@ async def status_tenant(request: Request, tenant_id: str | None = None) -> Tenan
             namespace=settings.namespace,
             pii_action=settings.pii_policy.action.value,
             acl_labels=sorted(settings.acl_labels),
+            dedicated_index=settings.physical_index is not None,
+            physical_index=settings.physical_index,
         )
 
     @router.websocket("/v1/status/ws")

apps/gateway/tests/test_tenancy.py

@@ -100,9 +100,22 @@ def test_status_tenant_known() -> None:
         "namespace": "acme-prod",
         "pii_action": "block",
         "acl_labels": ["region:eu"],
+        "dedicated_index": False,
+        "physical_index": None,
     }
 
 
+def test_status_tenant_reports_dedicated_index() -> None:
+    cfg = {
+        "version": "1",
+        "tenants": [{"id": "vault", "name": "Vault", "dedicated_index": True}],
+    }
+    client = TestClient(build_app_from_config(RagConfig.model_validate(cfg)))
+    body = client.get("/v1/status/tenant", headers={"X-Tenant-Id": "vault"}).json()
+    assert body["dedicated_index"] is True
+    assert body["physical_index"] == "vault"  # auto-derived from the id
+
+
 def test_status_tenant_unknown_is_defaulted() -> None:
     client = TestClient(build_app_from_config(RagConfig.model_validate(_resolver_cfg())))
     body = client.get("/v1/status/tenant", headers={"X-Tenant-Id": "ghost"}).json()

dist/openapi.json

@@ -3674,6 +3674,11 @@
             "title": "Acl Labels",
             "type": "array"
           },
+          "dedicated_index": {
+            "default": false,
+            "title": "Dedicated Index",
+            "type": "boolean"
+          },
           "known": {
             "default": false,
             "title": "Known",
@@ -3684,6 +3689,17 @@
             "title": "Namespace",
             "type": "string"
           },
+          "physical_index": {
+            "anyOf": [
+              {
+                "type": "string"
+              },
+              {
+                "type": "null"
+              }
+            ],
+            "title": "Physical Index"
+          },
           "pii_action": {
             "default": "redact",
             "title": "Pii Action",

dist/openapi.yaml

@@ -3255,6 +3255,10 @@ components:
             type: string
           title: Acl Labels
           type: array
+        dedicated_index:
+          default: false
+          title: Dedicated Index
+          type: boolean
         known:
           default: false
           title: Known
@@ -3263,6 +3267,11 @@ components:
           default: ''
           title: Namespace
           type: string
+        physical_index:
+          anyOf:
+          - type: string
+          - type: 'null'
+          title: Physical Index
         pii_action:
           default: redact
           title: Pii Action

dist/rag.schema.json

@@ -1325,6 +1325,24 @@
           },
           "title": "Acl Labels",
           "type": "array"
+        },
+        "dedicated_index": {
+          "default": false,
+          "title": "Dedicated Index",
+          "type": "boolean"
+        },
+        "dedicated_index_name": {
+          "anyOf": [
+            {
+              "minLength": 1,
+              "type": "string"
+            },
+            {
+              "type": "null"
+            }
+          ],
+          "default": null,
+          "title": "Dedicated Index Name"
         }
       },
       "required": [

dist/rag.schema.yaml

@@ -1231,6 +1231,17 @@ $defs:
           type: string
         title: Acl Labels
         type: array
+      dedicated_index:
+        default: false
+        title: Dedicated Index
+        type: boolean
+      dedicated_index_name:
+        anyOf:
+        - minLength: 1
+          type: string
+        - type: 'null'
+        default: null
+        title: Dedicated Index Name
     required:
     - id
     - name

dist/schemas/RequestContext.json

@@ -206,6 +206,18 @@
       "title": "Namespace",
       "type": "string"
     },
+    "physical_index": {
+      "anyOf": [
+        {
+          "type": "string"
+        },
+        {
+          "type": "null"
+        }
+      ],
+      "default": null,
+      "title": "Physical Index"
+    },
     "pii_policy": {
       "$ref": "#/$defs/PiiPolicy"
     },

dist/schemas/TenantSettings.json

@@ -58,6 +58,18 @@
       "title": "Acl Labels",
       "type": "array",
       "uniqueItems": true
+    },
+    "physical_index": {
+      "anyOf": [
+        {
+          "type": "string"
+        },
+        {
+          "type": "null"
+        }
+      ],
+      "default": null,
+      "title": "Physical Index"
     }
   },
   "required": [

docs/README.md

@@ -172,6 +172,7 @@ broken, and what to fix before committing to the next phase.
 | [ADR-0027-offline-eval-harness.md](adr/ADR-0027-offline-eval-harness.md) | Decision (Step 5.2): complete the Step 0.8 eval skeleton with a deterministic, offline harness — synthetic 5-domain corpus on the noop SPIs (a `HashingEmbedder` gives the zero-vector dense path real signal, fused with `NoopKeywordStore` token overlap via the real `HybridRetriever`); 500-query golden set committed under `tests/eval/golden/` and generated reproducibly from the corpus (drift-gated), 3 passages/concept so nDCG + Citation Precision aren't degenerate; nDCG added + dependency-free `lexical_faithfulness` default (RAGAS optional) keep CI ML-free; harness in `eval/golden_set_v0/` (not `rag_config`) so config stays light; additive report fields + self-contained HTML; `--check` enforces the threshold floor now (regression-delta is 5.3); rejected real backends in CI, RAGAS-as-default, hand-authored queries |
 | [ADR-0032-ab-testing-shadow-mode.md](adr/ADR-0032-ab-testing-shadow-mode.md) | Decision (Step 5.7): compare two configs on live traffic, delivered in slices (5.7a analyzer+tracker+surface, 5.7b shadow, 5.7c routing, 5.7d console); the analyzer is pure + stdlib-only in `rag_config.eval` (normal-approx Welch via `statistics.NormalDist`, no numpy/scipy — same spirit as drift PSI / cost z-score); the `ABExperimentTracker` is a pure *sample holder* in `rag_observability` (so it doesn't import `rag_config`), and the gateway composes the two for `GET /v1/status/experiments`; opt-in by default (A/B routing can change responses); `ABAnalysisResult` additive in `rag_core.eval` (not in `dist/schemas`); rejected numpy/scipy, a new package, putting the analyzer in observability, defaulting on |
 | [ADR-0033-logical-multi-tenancy.md](adr/ADR-0033-logical-multi-tenancy.md) | Decision (Step 6.1): make per-tenant `rag.yaml` config drive requests. `TenantResolver` (`rag_config`) maps a tenant id → frozen `TenantSettings` (`rag_core`), applied once at the gateway boundary; resolver in config / settings type in core keeps the `config → core` direction; unknown tenants → safe defaults (isolated not privileged); `RequestContext.namespace` defaults to `tenant_id` (a backend-partition primitive — Pinecone uses it — not a chunk field, so `filter_pushdown` is unchanged); scope stops at resolution + threading (ACL push-down 6.3, PII egress 6.5, physical tenancy 6.2); additive + inert in `build_app` |
+| [ADR-0034-physical-multi-tenancy.md](adr/ADR-0034-physical-multi-tenancy.md) | Decision (Step 6.2): a *dedicated* vector index/collection per tenant. `TenantConfig.dedicated_index` resolves to a `physical_index` key on `TenantSettings`, threaded onto `RequestContext.physical_index`; backends read only `ctx` (graph is backends→core, never rag-config) and namespace their base under it (`<base>-<key>`), lazily creating it; one instance + per-tenant derivation (no per-tenant instances, no SPI change); Noop is the CI conformance oracle (keyed by `physical_index`) for a cross-tenant probe gate that proves isolation independent of the tenant filter; Noop + Pinecone + Qdrant this step, others later |
 | [ADR-0031-cost-anomaly.md](adr/ADR-0031-cost-anomaly.md) | Decision (Step 5.6c): detect per-tenant spend spikes with a rolling `CostTracker` (not the cumulative quota counter); detect scale-free on the token series (cost = tokens × a constant price) so detection is decoupled from quota pricing and works with quotas off; two gates (ratio + z-score, z relaxed on a flat baseline) → tri-state verdict; put it in `rag-observability` as a `dataclass` (gateway wraps it in a Pydantic `CostStatusResponse`) so there's **no `rag-core` type / `dist/schemas` churn**; feed O(1) from `record_request_usage` before the quota block; pull-based `GET /v1/status/cost` (no per-request span/event); rejected folding into the infra-scoped drift registry, a new package, a `cost.anomaly_detected` push event (deferred), per-model pricing, a time-series DB |
 | [ADR-0030-drift-monitors.md](adr/ADR-0030-drift-monitors.md) | Decision (Step 5.5): detect retrieval degradation with five drift monitors in a new `rag-drift` package (mirroring rag-feedback); two statistics — PSI (pure, binned, dependency-free) for the distribution monitors + mean-drop for the rate/score monitors — over one scalar-window `DriftMonitor`; infra-scoped registry (like breakers) fed via `observe` from the signals the gateway already computes (query length / retrieval score / HyDE-embedding norm / guard grounded-claim fraction / feedback citation clicks); detection on dashboard-poll `evaluate()` with transition-edge `drift.detected` (structured event + the Step 3.9 webhook, targeting `alert_tenant`); observe-only / inert-by-default / rebaseline; rejected per-tenant monitors, per-dimension embedding PSI, a stats library, a background scheduler, hot-path detection |
 | [ADR-0029-online-feedback.md](adr/ADR-0029-online-feedback.md) | Decision (Step 5.4): capture online feedback + implicit signals in a new `rag-feedback` package mirroring `rag-provenance` (SPI + types in rag-core; recorder + pure aggregator in the package); one polymorphic `POST /v1/feedback` (a `signal` enum spanning explicit thumbs/rating/comment + implicit citation-click/copy/regenerate/dwell, `kind` inferred); normalise every signal to a `[-1,1]` score so the dashboard has one satisfaction number; **redact-don't-hash** free-text comments via an injected `PIIDetector` (default `NoopPIIDetector` seam, `comment_redacted` flag, PII-free event) — opposite of provenance's hashing; body identity like `/v1/query`; degrade-open + inert-by-default; `GET /v1/status/feedback` dashboard (event-only, no per-call span); admin-UI card deferred to 5.6; rejected separate per-signal endpoints, header-auth, hashing/raw comments, an OTel span per submission, folding into provenance |