feat(tenancy): physical tenancy — dedicated index per tenant (Step 6.2)

[officialCodeWork]

Summary

Step 6.2 — Physical tenancy: give a tenant its own vector index/collection, so its data is physically separate even if a filter were bypassed — for tenants that require hard isolation. Built as a single PR.

Where 6.1 was logical (one shared index + tenant_id filter + namespace), 6.2 is physical (a dedicated index per tenant), opt-in per tenant.

• Config: TenantConfig.dedicated_index: bool (+ optional dedicated_index_name). The resolver computes a physical_index key (dedicated_index_name or tenant_id, else None) on TenantSettings, threaded onto the new RequestContext.physical_index by the gateway middleware — the same resolve-once-at-the-boundary pattern as 6.1.
• Backends read only ctx (the dependency graph is backends→core, never rag-config), namespace their base index under the key (<base>-<key>), and lazily create the dedicated index/collection on first use; None keeps the shared base index. Implemented for Noop (the CI conformance oracle, partitioned by physical_index), Pinecone (dedicated index), and Qdrant (dedicated collection). pgvector / Elasticsearch / Weaviate follow the same pattern later.
• Cross-tenant probe gate (tests/redteam/test_cross_tenant_dedicated_index.py, CI, Noop) proves a dedicated tenant's data is invisible to another independent of the tenant filter — a matching tenant_id pointed at a different index still returns nothing. Live-service isolation is covered by the Pinecone / Qdrant integration tests.
• One backend instance + per-tenant derivation — no per-tenant backend instances, no SPI change (the methods already take ctx).
• GET /v1/status/tenant + ragctl tenant resolve <id> report dedicated_index / physical_index.

TenantConfig fields → rag.schema; TenantSettings + RequestContext regenerated; /v1/status/tenant → dist/openapi.

Key decisions (see ADR-0034)

• The per-tenant index decision reaches the backend via ctx.physical_index (not a resolver call), because rag-backends may import rag-core only.
• Uniform <base>-<key> naming (collision-safe under the deployment base); lazy check-and-create; Noop as the CI oracle so the probe gate runs without live services.

Documentation

• ADR-0034 — the decision + trade-offs
• docs/reference/tenancy.md — config + physical_index + probe gate
• docs/architecture/multi-tenancy.md — design + the updated boundary table
• docs/architecture/request-context.md — physical_index field
• docs/README.md — ADR-0034 entry
• TRACKER.md — 6.2 ✅, Phase 6 2/10, overall 66/84, next action → 6.3, PR history

Test plan

• ruff check + ruff format --check clean
• mypy --strict clean (293 source files)
• RAG001 logging gate clean
• schema-drift + openapi-drift regenerated (committed); proto-compat + policy-coverage pass
• full suite 2529 passed, 4 skipped (stable order)
• ~30 new/updated tests, incl. the cross-tenant dedicated-index probe gate (physical isolation independent of the tenant filter), resolver/config (tests/config/test_tenancy.py), /v1/status/tenant dedicated-index reporting (apps/gateway/tests/test_tenancy.py), and ragctl tenant (packages/ragctl/tests/test_tenant.py)
• ragctl tenant resolve acme -f rag.yaml shows index: dedicated (key=acme) end-to-end

🤖 Generated with Claude Code