feat(policy): ACL push-down at retrieval — AclPolicyEngine (Step 6.3)

TRACKER.md

@@ -14,12 +14,13 @@
 | | |
 |---|---|
 | **Last updated** | 2026-06-08 |
-| **Current phase** | Phase 6 — Governance & Tenancy (**2 / 10 steps**) |
-| **Overall** | **66 / 84 steps** — Phases 0–5 complete |
-| **Next action** | **Step 6.3 — ACL push-down at retrieval**: inject the ACL filter into every vector / BM25 / graph query; `acl.egress_denied` event. Consumes the per-tenant `acl_labels` resolved in 6.1. |
+| **Current phase** | Phase 6 — Governance & Tenancy (**3 / 10 steps**) |
+| **Overall** | **67 / 84 steps** — Phases 0–5 complete |
+| **Next action** | **Step 6.4 — ACL egress verifier**: post-retrieval re-check of returned chunks against the principal's ACLs (defense-in-depth); zero-ACL-violation-rate gate. Backstops the 6.3 push-down. |
 
 **Recently shipped**
 
+- **6.3** ✅ ACL push-down — opt-in `AclPolicyEngine` And-merges `any_in("acl_labels", principal.acl_labels)` into every `read_chunk` query (overlap, fail-closed); `acl.egress_denied` event — [#150](https://github.com/officialCodeWork/AgentContextOS/pull/150)
 - **6.2** ✅ Physical tenancy — per-tenant *dedicated* vector index/collection; `TenantConfig.dedicated_index` → `ctx.physical_index` → backend `<base>-<key>` (Noop/Pinecone/Qdrant); cross-tenant probe gate — [#149](https://github.com/officialCodeWork/AgentContextOS/pull/149)
 - **6.1** ✅ Logical multi-tenancy — `TenantResolver` resolves per-tenant `rag.yaml` config (namespace / PII / ACL labels) → `TenantSettings` applied at the gateway boundary; `RequestContext.namespace`; `GET /v1/status/tenant`; `ragctl tenant` — [#148](https://github.com/officialCodeWork/AgentContextOS/pull/148)
 - **5.7d** ✅ Experiments console card + Phase-5 close-out — admin Live-Status A/B card over `GET /v1/status/experiments` (lift + CI per experiment) — [#147](https://github.com/officialCodeWork/AgentContextOS/pull/147)
@@ -56,9 +57,9 @@
 | 3 | Gateway & Agent Runtime | 11 | **11** | 0 |
 | 4 | Reliability | 6 | **6** | 0 |
 | 5 | Eval & Observability | 7 | **7** | 0 |
-| 6 | Governance & Tenancy | 10 | **2** | 8 |
+| 6 | Governance & Tenancy | 10 | **3** | 7 |
 | 7 | Pilot, Harden, GA | 10 | 0 | 10 |
-| **Total** | | **84** | **66** | **18** |
+| **Total** | | **84** | **67** | **17** |
 
 ---
 
@@ -641,13 +642,13 @@
 - **Phase-5 close-out:** Step 5.7 ✅ → **Phase 5 complete (7 / 7)**; deferred items remain documented (per-tenant drift / per-dimension embedding PSI; feedback/breaker/quota Grafana export + Loki-events dashboard; gRPC proto mirror of `corpus_decision` + `experiment`; sequential / multi-metric experiments)
 - [reference/experiments.md](docs/reference/experiments.md), [reference/admin-ui.md](docs/reference/admin-ui.md)
 
-## Phase 6 — Governance & Tenancy (Weeks 28–34) 🚧 (2 / 10)
+## Phase 6 — Governance & Tenancy (Weeks 28–34) 🚧 (3 / 10)
 
 | Step | Title | Status | Planned deliverables |
 |------|-------|:------:|----------------------|
 | 6.1 | Logical multi-tenancy | ✅ | [#148](https://github.com/officialCodeWork/AgentContextOS/pull/148) — `TenantResolver` → `TenantSettings` (namespace / PII / ACL labels) applied at the boundary; `RequestContext.namespace`; `GET /v1/status/tenant`; `ragctl tenant` |
 | 6.2 | Physical tenancy (dedicated index) | ✅ | [#149](https://github.com/officialCodeWork/AgentContextOS/pull/149) — `dedicated_index` → `ctx.physical_index` → backend `<base>-<key>` (Noop/Pinecone/Qdrant, lazy create); cross-tenant probe gate |
-| 6.3 | ACL push-down at retrieval | ⏳ | ACL filter injected into every vector / BM25 / graph query; `acl.egress_denied` event |
+| 6.3 | ACL push-down at retrieval | ✅ | [#150](https://github.com/officialCodeWork/AgentContextOS/pull/150) — opt-in `AclPolicyEngine` And-merges `any_in("acl_labels", …)` into every `read_chunk` push-down (overlap, fail-closed); `acl.egress_denied` |
 | 6.4 | ACL egress verifier | ⏳ | Post-retrieval re-check; defense-in-depth; zero-ACL-violation-rate gate |
 | 6.5 | PII policies | ⏳ | Per-tenant PII enforcement (block / redact / allow); egress redaction; `pii.egress_blocked` event |
 | 6.6 | Immutable audit log | ⏳ | Hash-chain audit log; WORM export; tamper-evident verification; `GET /v1/audit` |
@@ -675,6 +676,15 @@
 - One instance + per-tenant derivation (**no** per-tenant backend instances, **no** SPI change); `GET /v1/status/tenant` + `ragctl tenant resolve` report `dedicated_index` / `physical_index`; `TenantConfig` fields → `rag.schema`, `TenantSettings` + `RequestContext` regenerated, `/v1/status/tenant` → `dist/openapi`; ~30 new/updated tests; all gates green
 - [ADR-0034](docs/adr/ADR-0034-physical-multi-tenancy.md), [reference/tenancy.md](docs/reference/tenancy.md#physical-tenancy-step-62), [architecture/multi-tenancy.md](docs/architecture/multi-tenancy.md)
 
+### 6.3 — ACL push-down at retrieval ✅ [#150](https://github.com/officialCodeWork/AgentContextOS/pull/150)
+
+- Enforces **label-based ACLs** at retrieval: 6.1 resolved each tenant's `acl_labels` onto the principal, but nothing enforced them.  New **`AclPolicyEngine`** (`rag-policy`) — a decorator like `QuotaPolicyEngine` — And-merges **`any_in("acl_labels", sorted(principal.acl_labels))`** into every `read_chunk` push-down; a chunk is retrievable only when its labels **overlap** the principal's
+- **Overlap via the existing `AnyIn` predicate** — the semantics the codebase already assumed and every backend translator already speaks (pgvector `&&`, Qdrant `MatchAny`, Cypher `ANY()`, noop `evaluate`), so **zero new predicate / translator changes**.  Injected at the canonical `HybridRetriever` `read_chunk` PDP site (already merges `filter_pushdown`), so no new call site / coverage-linter entry
+- **Fail-closed**: the clause is uniform — `any_in([])` matches nothing, so a label-less principal retrieves nothing (model "public" as a shared label granted to all); each label-less request emits one PII-free **`acl.egress_denied`** event (pre-registered `EVT_ACL_DENIED`)
+- **Opt-in** via new `cfg.acl.enabled` (default off → pre-6.3 behaviour, tenant scoping only); `build_app(acl_enabled=…)` wraps the engine, `build_app_from_config` from config.  Decorates without absorbing the inner engine (composes with a production PDP)
+- `AclConfig` → `rag.schema`; ~17 new tests incl. an **end-to-end ACL red-team** through `HybridRetriever` (overlap / disjoint / fail-closed / public-label / no-tenant-bypass) + engine unit + event + wiring; all gates green (ruff, mypy --strict, RAG001, schema-drift, policy-coverage, log-schema)
+- **Scope:** push-down enforcement only — post-retrieval egress re-verification is 6.4, graph **edge** ACLs deferred.  [ADR-0035](docs/adr/ADR-0035-acl-pushdown.md), [architecture/policy-engine.md](docs/architecture/policy-engine.md), [reference/tenancy.md](docs/reference/tenancy.md)
+
 ---
 
 ## Phase 7 — Pilot, Harden, GA (Weeks 34–40) ⏳
@@ -828,6 +838,7 @@ Complete log of every PR. Routine Dependabot bumps are grouped; everything else
 | [#147](https://github.com/officialCodeWork/AgentContextOS/pull/147) | 2026-06-07 | feat(admin-ui): A/B experiments console card + Phase-5 close-out (Step 5.7d) |
 | [#148](https://github.com/officialCodeWork/AgentContextOS/pull/148) | 2026-06-08 | feat(tenancy): logical multi-tenancy — per-tenant config resolution (Step 6.1) |
 | [#149](https://github.com/officialCodeWork/AgentContextOS/pull/149) | 2026-06-08 | feat(tenancy): physical tenancy — dedicated index per tenant (Step 6.2) |
+| [#150](https://github.com/officialCodeWork/AgentContextOS/pull/150) | 2026-06-08 | feat(policy): ACL push-down at retrieval — AclPolicyEngine (Step 6.3) |
 | #78–#80, #116–#118 | Open | Dependabot bumps — awaiting merge |
 | #81 | Closed | Dependabot bump — superseded |
 

apps/gateway/src/rag_gateway/app.py

@@ -368,6 +368,7 @@ def build_app(
     shadow_runner: Any | None = None,
     ab_router: Any | None = None,
     tenant_resolver: Any | None = None,
+    acl_enabled: bool = False,
     enable_cors: bool = True,
     default_tenant_id: TenantId | None = None,
 ) -> FastAPI:
@@ -492,7 +493,16 @@ def build_app(
     # quotas and the entry checks resolve to the inner engine's allow.
     if quota_enforcer is not None:
         policy_engine = QuotaPolicyEngine(quota_enforcer, inner=policy_engine)
+    # Label-based ACL push-down (Step 6.3) — when enabled, decorate the engine so
+    # every ``read_chunk`` filter is And-merged with an ``acl_labels`` overlap
+    # clause; fail-closed (a label-less principal retrieves nothing).  Opt-in, so
+    # the default keeps the pre-6.3 behaviour (tenant scoping only).
+    if acl_enabled:
+        from rag_policy import AclPolicyEngine
+
+        policy_engine = AclPolicyEngine(inner=policy_engine)
     app.state.policy_engine = policy_engine
+    app.state.acl_enabled = acl_enabled
     app.state.quota_enforcer = quota_enforcer
 
     # Hallucination guard (Step 4.3) — post-generation faithfulness check over

apps/gateway/src/rag_gateway/wiring.py

@@ -702,6 +702,9 @@ def build_app_from_config(cfg: RagConfig, **overrides: Any) -> FastAPI:
     # process (the instrument is global, the callback reads current state).
     _register_platform_metrics_once(overrides.get("drift_registry"), overrides.get("cost_tracker"))
 
+    # Label-based ACL push-down (Step 6.3) — opt-in; off by default.
+    acl_enabled = overrides.pop("acl_enabled", cfg.acl.enabled)
+
     return build_app(
         corpus_store=corpus_store,
         retrieval_router=retrieval_router,
@@ -711,6 +714,7 @@ def build_app_from_config(cfg: RagConfig, **overrides: Any) -> FastAPI:
         guard_enabled=guard_enabled,
         breaker_registry=breaker_registry,
         quota_enforcer=quota_enforcer,
+        acl_enabled=acl_enabled,
         **overrides,
     )
 

apps/gateway/tests/test_acl.py

@@ -0,0 +1,39 @@
+"""ACL push-down config + gateway wiring (Step 6.3)."""
+
+from __future__ import annotations
+
+from rag_config import RagConfig
+from rag_config.schema import AclConfig
+from rag_gateway import build_app
+from rag_gateway.wiring import build_app_from_config
+from rag_policy import AclPolicyEngine, NoopPolicyEngine
+
+
+def test_acl_config_defaults_off() -> None:
+    assert RagConfig().acl.enabled is False
+
+
+def test_inert_by_default() -> None:
+    app = build_app()
+    assert app.state.acl_enabled is False
+    assert not isinstance(app.state.policy_engine, AclPolicyEngine)
+
+
+def test_build_from_config_wraps_when_enabled() -> None:
+    app = build_app_from_config(RagConfig(acl=AclConfig(enabled=True)))
+    assert app.state.acl_enabled is True
+    assert isinstance(app.state.policy_engine, AclPolicyEngine)
+
+
+def test_build_from_config_inert_when_disabled() -> None:
+    app = build_app_from_config(RagConfig())
+    assert not isinstance(app.state.policy_engine, AclPolicyEngine)
+
+
+def test_acl_decorates_without_absorbing_inner() -> None:
+    app = build_app_from_config(RagConfig(acl=AclConfig(enabled=True)))
+    engine = app.state.policy_engine
+    assert isinstance(engine, AclPolicyEngine)
+    # the inner engine is preserved (here the default Noop), so ACL composes with
+    # — rather than replaces — the production PDP.
+    assert isinstance(engine.inner, NoopPolicyEngine)

dist/rag.schema.json

@@ -1,5 +1,18 @@
 {
   "$defs": {
+    "AclConfig": {
+      "additionalProperties": false,
+      "description": "Label-based ACL push-down enforcement (Step 6.3).\n\nWhen ``enabled`` the gateway wraps its PolicyEngine in an ``AclPolicyEngine``\nthat injects ``any_in(\"acl_labels\", principal.acl_labels)`` into every\n``read_chunk`` push-down \u2014 a chunk is retrievable only when its labels overlap\nthe principal's (the labels resolved per tenant in Step 6.1).\n\n**Disabled by default** \u2014 turning it on changes which chunks a principal can\nretrieve, and is **fail-closed**: a principal carrying no ACL labels matches\nno labeled chunk, so it retrieves nothing (model \"public\" data as a shared\nlabel granted to all principals).  Each label-less request emits a PII-free\n``acl.egress_denied`` event.  ACL *egress* re-verification is Step 6.4.",
+      "properties": {
+        "enabled": {
+          "default": false,
+          "title": "Enabled",
+          "type": "boolean"
+        }
+      },
+      "title": "AclConfig",
+      "type": "object"
+    },
     "AuthConfig": {
       "additionalProperties": false,
       "properties": {
@@ -1600,6 +1613,9 @@
     "quotas": {
       "$ref": "#/$defs/QuotaConfig"
     },
+    "acl": {
+      "$ref": "#/$defs/AclConfig"
+    },
     "webhooks": {
       "$ref": "#/$defs/WebhooksConfig"
     },

dist/rag.schema.yaml

@@ -1,4 +1,34 @@
 $defs:
+  AclConfig:
+    additionalProperties: false
+    description: 'Label-based ACL push-down enforcement (Step 6.3).
+
+
+      When ``enabled`` the gateway wraps its PolicyEngine in an ``AclPolicyEngine``
+
+      that injects ``any_in("acl_labels", principal.acl_labels)`` into every
+
+      ``read_chunk`` push-down — a chunk is retrievable only when its labels overlap
+
+      the principal''s (the labels resolved per tenant in Step 6.1).
+
+
+      **Disabled by default** — turning it on changes which chunks a principal can
+
+      retrieve, and is **fail-closed**: a principal carrying no ACL labels matches
+
+      no labeled chunk, so it retrieves nothing (model "public" data as a shared
+
+      label granted to all principals).  Each label-less request emits a PII-free
+
+      ``acl.egress_denied`` event.  ACL *egress* re-verification is Step 6.4.'
+    properties:
+      enabled:
+        default: false
+        title: Enabled
+        type: boolean
+    title: AclConfig
+    type: object
   AuthConfig:
     additionalProperties: false
     properties:
@@ -1444,6 +1474,8 @@ properties:
     $ref: '#/$defs/BreakerConfig'
   quotas:
     $ref: '#/$defs/QuotaConfig'
+  acl:
+    $ref: '#/$defs/AclConfig'
   webhooks:
     $ref: '#/$defs/WebhooksConfig'
   provenance:

docs/README.md

@@ -173,6 +173,7 @@ broken, and what to fix before committing to the next phase.
 | [ADR-0032-ab-testing-shadow-mode.md](adr/ADR-0032-ab-testing-shadow-mode.md) | Decision (Step 5.7): compare two configs on live traffic, delivered in slices (5.7a analyzer+tracker+surface, 5.7b shadow, 5.7c routing, 5.7d console); the analyzer is pure + stdlib-only in `rag_config.eval` (normal-approx Welch via `statistics.NormalDist`, no numpy/scipy — same spirit as drift PSI / cost z-score); the `ABExperimentTracker` is a pure *sample holder* in `rag_observability` (so it doesn't import `rag_config`), and the gateway composes the two for `GET /v1/status/experiments`; opt-in by default (A/B routing can change responses); `ABAnalysisResult` additive in `rag_core.eval` (not in `dist/schemas`); rejected numpy/scipy, a new package, putting the analyzer in observability, defaulting on |
 | [ADR-0033-logical-multi-tenancy.md](adr/ADR-0033-logical-multi-tenancy.md) | Decision (Step 6.1): make per-tenant `rag.yaml` config drive requests. `TenantResolver` (`rag_config`) maps a tenant id → frozen `TenantSettings` (`rag_core`), applied once at the gateway boundary; resolver in config / settings type in core keeps the `config → core` direction; unknown tenants → safe defaults (isolated not privileged); `RequestContext.namespace` defaults to `tenant_id` (a backend-partition primitive — Pinecone uses it — not a chunk field, so `filter_pushdown` is unchanged); scope stops at resolution + threading (ACL push-down 6.3, PII egress 6.5, physical tenancy 6.2); additive + inert in `build_app` |
 | [ADR-0034-physical-multi-tenancy.md](adr/ADR-0034-physical-multi-tenancy.md) | Decision (Step 6.2): a *dedicated* vector index/collection per tenant. `TenantConfig.dedicated_index` resolves to a `physical_index` key on `TenantSettings`, threaded onto `RequestContext.physical_index`; backends read only `ctx` (graph is backends→core, never rag-config) and namespace their base under it (`<base>-<key>`), lazily creating it; one instance + per-tenant derivation (no per-tenant instances, no SPI change); Noop is the CI conformance oracle (keyed by `physical_index`) for a cross-tenant probe gate that proves isolation independent of the tenant filter; Noop + Pinecone + Qdrant this step, others later |
+| [ADR-0035-acl-pushdown.md](adr/ADR-0035-acl-pushdown.md) | Decision (Step 6.3): label-based ACL push-down at retrieval. `AclPolicyEngine` (a decorator like `QuotaPolicyEngine`) And-merges `any_in("acl_labels", principal.acl_labels)` into every `read_chunk` push-down at the canonical `HybridRetriever` PDP site — overlap semantics via the existing `AnyIn` predicate (zero backend/translator changes), **fail-closed** (label-less principal matches nothing; "public" = a shared label), **opt-in** via `cfg.acl.enabled`; emits `acl.egress_denied` on a request-level denial; graph edge ACLs + post-retrieval re-verification (6.4) deferred |
 | [ADR-0031-cost-anomaly.md](adr/ADR-0031-cost-anomaly.md) | Decision (Step 5.6c): detect per-tenant spend spikes with a rolling `CostTracker` (not the cumulative quota counter); detect scale-free on the token series (cost = tokens × a constant price) so detection is decoupled from quota pricing and works with quotas off; two gates (ratio + z-score, z relaxed on a flat baseline) → tri-state verdict; put it in `rag-observability` as a `dataclass` (gateway wraps it in a Pydantic `CostStatusResponse`) so there's **no `rag-core` type / `dist/schemas` churn**; feed O(1) from `record_request_usage` before the quota block; pull-based `GET /v1/status/cost` (no per-request span/event); rejected folding into the infra-scoped drift registry, a new package, a `cost.anomaly_detected` push event (deferred), per-model pricing, a time-series DB |
 | [ADR-0030-drift-monitors.md](adr/ADR-0030-drift-monitors.md) | Decision (Step 5.5): detect retrieval degradation with five drift monitors in a new `rag-drift` package (mirroring rag-feedback); two statistics — PSI (pure, binned, dependency-free) for the distribution monitors + mean-drop for the rate/score monitors — over one scalar-window `DriftMonitor`; infra-scoped registry (like breakers) fed via `observe` from the signals the gateway already computes (query length / retrieval score / HyDE-embedding norm / guard grounded-claim fraction / feedback citation clicks); detection on dashboard-poll `evaluate()` with transition-edge `drift.detected` (structured event + the Step 3.9 webhook, targeting `alert_tenant`); observe-only / inert-by-default / rebaseline; rejected per-tenant monitors, per-dimension embedding PSI, a stats library, a background scheduler, hot-path detection |
 | [ADR-0029-online-feedback.md](adr/ADR-0029-online-feedback.md) | Decision (Step 5.4): capture online feedback + implicit signals in a new `rag-feedback` package mirroring `rag-provenance` (SPI + types in rag-core; recorder + pure aggregator in the package); one polymorphic `POST /v1/feedback` (a `signal` enum spanning explicit thumbs/rating/comment + implicit citation-click/copy/regenerate/dwell, `kind` inferred); normalise every signal to a `[-1,1]` score so the dashboard has one satisfaction number; **redact-don't-hash** free-text comments via an injected `PIIDetector` (default `NoopPIIDetector` seam, `comment_redacted` flag, PII-free event) — opposite of provenance's hashing; body identity like `/v1/query`; degrade-open + inert-by-default; `GET /v1/status/feedback` dashboard (event-only, no per-call span); admin-UI card deferred to 5.6; rejected separate per-signal endpoints, header-auth, hashing/raw comments, an OTel span per submission, folding into provenance |