Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 17 additions & 6 deletions TRACKER.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,13 @@
| | |
|---|---|
| **Last updated** | 2026-06-08 |
| **Current phase** | Phase 6 — Governance & Tenancy (**3 / 10 steps**) |
| **Overall** | **67 / 84 steps** — Phases 0–5 complete |
| **Next action** | **Step 6.4ACL egress verifier**: post-retrieval re-check of returned chunks against the principal's ACLs (defense-in-depth); zero-ACL-violation-rate gate. Backstops the 6.3 push-down. |
| **Current phase** | Phase 6 — Governance & Tenancy (**4 / 10 steps**) |
| **Overall** | **68 / 84 steps** — Phases 0–5 complete |
| **Next action** | **Step 6.5PII policies**: per-tenant PII enforcement at egress (block / redact / allow) through the PolicyEngine `egress_text` decision; `pii.egress_blocked` event. Builds on the per-tenant `pii_policy` already resolved onto `RequestContext` in 6.1. |

**Recently shipped**

- **6.4** ✅ ACL egress verifier — `AclEgressVerifier` re-checks returned `ChunkRef`s against the principal's labels at the gateway router boundary (defense-in-depth, same overlap semantics, independent of the PDP); `acl.egress_violation` event; red-team zero-violation-rate gate — [#151](https://github.com/officialCodeWork/AgentContextOS/pull/151)
- **6.3** ✅ ACL push-down — opt-in `AclPolicyEngine` And-merges `any_in("acl_labels", principal.acl_labels)` into every `read_chunk` query (overlap, fail-closed); `acl.egress_denied` event — [#150](https://github.com/officialCodeWork/AgentContextOS/pull/150)
- **6.2** ✅ Physical tenancy — per-tenant *dedicated* vector index/collection; `TenantConfig.dedicated_index` → `ctx.physical_index` → backend `<base>-<key>` (Noop/Pinecone/Qdrant); cross-tenant probe gate — [#149](https://github.com/officialCodeWork/AgentContextOS/pull/149)
- **6.1** ✅ Logical multi-tenancy — `TenantResolver` resolves per-tenant `rag.yaml` config (namespace / PII / ACL labels) → `TenantSettings` applied at the gateway boundary; `RequestContext.namespace`; `GET /v1/status/tenant`; `ragctl tenant` — [#148](https://github.com/officialCodeWork/AgentContextOS/pull/148)
Expand Down Expand Up @@ -57,9 +58,9 @@
| 3 | Gateway & Agent Runtime | 11 | **11** | 0 |
| 4 | Reliability | 6 | **6** | 0 |
| 5 | Eval & Observability | 7 | **7** | 0 |
| 6 | Governance & Tenancy | 10 | **3** | 7 |
| 6 | Governance & Tenancy | 10 | **4** | 6 |
| 7 | Pilot, Harden, GA | 10 | 0 | 10 |
| **Total** | | **84** | **67** | **17** |
| **Total** | | **84** | **68** | **16** |

---

Expand Down Expand Up @@ -649,7 +650,7 @@
| 6.1 | Logical multi-tenancy | ✅ | [#148](https://github.com/officialCodeWork/AgentContextOS/pull/148) — `TenantResolver` → `TenantSettings` (namespace / PII / ACL labels) applied at the boundary; `RequestContext.namespace`; `GET /v1/status/tenant`; `ragctl tenant` |
| 6.2 | Physical tenancy (dedicated index) | ✅ | [#149](https://github.com/officialCodeWork/AgentContextOS/pull/149) — `dedicated_index` → `ctx.physical_index` → backend `<base>-<key>` (Noop/Pinecone/Qdrant, lazy create); cross-tenant probe gate |
| 6.3 | ACL push-down at retrieval | ✅ | [#150](https://github.com/officialCodeWork/AgentContextOS/pull/150) — opt-in `AclPolicyEngine` And-merges `any_in("acl_labels", …)` into every `read_chunk` push-down (overlap, fail-closed); `acl.egress_denied` |
| 6.4 | ACL egress verifier | | Post-retrieval re-check; defense-in-depth; zero-ACL-violation-rate gate |
| 6.4 | ACL egress verifier | | [#151](https://github.com/officialCodeWork/AgentContextOS/pull/151) — `AclEgressVerifier` re-checks returned chunks at the gateway router boundary (defense-in-depth above the 6.3 push-down); `acl.egress_violation`; zero-violation-rate red-team gate |
| 6.5 | PII policies | ⏳ | Per-tenant PII enforcement (block / redact / allow); egress redaction; `pii.egress_blocked` event |
| 6.6 | Immutable audit log | ⏳ | Hash-chain audit log; WORM export; tamper-evident verification; `GET /v1/audit` |
| 6.7 | BYOK (Bring Your Own Key) | ⏳ | KMS integration (AWS KMS, GCP KMS, HashiCorp Vault); envelope encryption for embeddings |
Expand Down Expand Up @@ -685,6 +686,15 @@
- `AclConfig` → `rag.schema`; ~17 new tests incl. an **end-to-end ACL red-team** through `HybridRetriever` (overlap / disjoint / fail-closed / public-label / no-tenant-bypass) + engine unit + event + wiring; all gates green (ruff, mypy --strict, RAG001, schema-drift, policy-coverage, log-schema)
- **Scope:** push-down enforcement only — post-retrieval egress re-verification is 6.4, graph **edge** ACLs deferred. [ADR-0035](docs/adr/ADR-0035-acl-pushdown.md), [architecture/policy-engine.md](docs/architecture/policy-engine.md), [reference/tenancy.md](docs/reference/tenancy.md)

### 6.4 — ACL egress verifier ✅ [#151](https://github.com/officialCodeWork/AgentContextOS/pull/151)

- The 6.3 push-down filters ACLs at the source (one layer, re-implemented by each backend translator); 6.4 adds the **defense-in-depth second layer** that the `tests/policy/coverage.py` linter docstring already anticipated. New **`AclEgressVerifier`** (`rag_policy.egress`) — `verify(ctx, refs)` keeps only the returned `ChunkRef`s whose `acl_labels` **overlap** the principal's, dropping the rest — so a filter-translation bug, a backend that ignores the predicate, or a path wired without the policy engine **cannot leak an over-privileged chunk past the boundary**
- **Identical overlap semantics to the push-down** (same `∩ ≠ ∅`, same fail-closed / "public is a shared label" model), so it's a **no-op on a correctly-filtered result** and only ever acts on a genuine leak. Reads `ChunkRef.acl_labels` (which every backend populates regardless of the filter applied) → **no re-hydration**, an O(results) set-intersection per call (hot-path-safe)
- **Independent of the PDP** — it consults only `ctx.principal.acl_labels`, so it backstops the push-down even when the push-down isn't wired. **Layered at the retrieval router boundary**: a thin `SupportsRoute` wrapper (`AclEgressVerifyingRouter`, in the gateway so `rag-retrieval` keeps its no-`rag-policy` property) decorates `app.state.retrieval_router` — the single attribute every surface reads — so query / retrieve / corpus / OpenAI / agent all inherit it, a layer **above** the `HybridRetriever` where the push-down merges
- **Opt-in, on-by-default behind ACL**: new `cfg.acl.verify_egress` (default **true**) gated on `cfg.acl.enabled` — turning ACLs on gives **both** layers; set it false to run the push-down alone (the strict drop is meaningless without the label model, so it never fires for a non-ACL deployment). `build_app(acl_verify_egress=…)` / `build_app_from_config` wire it; `app.state.acl_verify_egress` reports the effective state
- **`acl.egress_violation`** (pre-registered `EVT_ACL_EGRESS_VIOLATION`, `error` level, PII-free: tenant / principal / counts / dropped chunk **ids** only) fires once per call that drops ≥1 ref — the *unexpected* push-down failure, distinct from 6.3's *expected* `acl.egress_denied`; a clean pass is silent
- **Scope:** gateway retrieval surfaces (everything reading `app.state.retrieval_router`); trusts the labels the backend reports on each `ChunkRef` (catching a *mislabelling* backend needs authoritative re-hydration — deferred), per-tenant/per-label violation metrics deferred to the 6.x governance dashboards. `AclConfig` → `rag.schema` regenerated; ~21 new tests incl. a **red-team zero-violation-rate gate** (`tests/redteam/test_acl_egress_verifier.py` — bypassed push-down + leaky backend → verifier drops every violation) + verifier unit + gateway wiring + event-schema; all gates green (ruff, mypy --strict 296 files, RAG001, schema-drift, policy-coverage, log-schema). [ADR-0036](docs/adr/ADR-0036-acl-egress-verifier.md), [architecture/policy-engine.md](docs/architecture/policy-engine.md), [reference/tenancy.md](docs/reference/tenancy.md)

---

## Phase 7 — Pilot, Harden, GA (Weeks 34–40) ⏳
Expand Down Expand Up @@ -839,6 +849,7 @@ Complete log of every PR. Routine Dependabot bumps are grouped; everything else
| [#148](https://github.com/officialCodeWork/AgentContextOS/pull/148) | 2026-06-08 | feat(tenancy): logical multi-tenancy — per-tenant config resolution (Step 6.1) |
| [#149](https://github.com/officialCodeWork/AgentContextOS/pull/149) | 2026-06-08 | feat(tenancy): physical tenancy — dedicated index per tenant (Step 6.2) |
| [#150](https://github.com/officialCodeWork/AgentContextOS/pull/150) | 2026-06-08 | feat(policy): ACL push-down at retrieval — AclPolicyEngine (Step 6.3) |
| [#151](https://github.com/officialCodeWork/AgentContextOS/pull/151) | 2026-06-08 | feat(policy): ACL egress verifier — defense-in-depth re-check (Step 6.4) |
| #78–#80, #116–#118 | Open | Dependabot bumps — awaiting merge |
| #81 | Closed | Dependabot bump — superseded |

Expand Down
64 changes: 64 additions & 0 deletions apps/gateway/src/rag_gateway/_acl_egress.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
"""Gateway glue: wrap the retrieval router with the Step 6.4 ACL egress verifier.

A thin :class:`~rag_retrieval.router.SupportsRoute` decorator that runs the inner
router, then re-checks the returned chunks with an
:class:`~rag_policy.AclEgressVerifier` before they leave the retrieval boundary.
Wrapping ``app.state.retrieval_router`` — the single attribute every retrieval
surface reads — gives query / retrieve / corpus / OpenAI / agent the
defense-in-depth backstop uniformly, a layer above the ``HybridRetriever`` where
the 6.3 push-down is merged. Lives in the gateway (the composition root) so
``rag-retrieval`` stays free of any ``rag-policy`` dependency.
"""

from __future__ import annotations

from collections.abc import Sequence

from rag_core.filter import FilterExpr
from rag_core.types import ChunkRef, CorpusId, RequestContext, RoutingDecision
from rag_policy import AclEgressVerifier
from rag_retrieval.router import SupportsRoute

__all__ = ["AclEgressVerifyingRouter"]


class AclEgressVerifyingRouter:
"""Decorate a ``SupportsRoute`` retriever with a post-retrieval ACL re-check."""

def __init__(self, *, inner: SupportsRoute, verifier: AclEgressVerifier) -> None:
self._inner = inner
self._verifier = verifier

@property
def inner(self) -> SupportsRoute:
return self._inner

async def route(
self,
ctx: RequestContext,
*,
text: str,
expansion_terms: dict[str, list[str]] | None = None,
hyde_vector: list[float] | None = None,
vector: list[float] | None = None,
graph_seeds: list[str] | None = None,
keyword_query: str | None = None,
corpus_ids: Sequence[CorpusId] | None = None,
top_k: int = 10,
filters: FilterExpr | None = None,
graph_hops: int = 1,
) -> tuple[RoutingDecision, list[ChunkRef]]:
decision, refs = await self._inner.route(
ctx,
text=text,
expansion_terms=expansion_terms,
hyde_vector=hyde_vector,
vector=vector,
graph_seeds=graph_seeds,
keyword_query=keyword_query,
corpus_ids=corpus_ids,
top_k=top_k,
filters=filters,
graph_hops=graph_hops,
)
return decision, self._verifier.verify(ctx, refs)
19 changes: 18 additions & 1 deletion apps/gateway/src/rag_gateway/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -369,6 +369,7 @@ def build_app(
ab_router: Any | None = None,
tenant_resolver: Any | None = None,
acl_enabled: bool = False,
acl_verify_egress: bool = True,
enable_cors: bool = True,
default_tenant_id: TenantId | None = None,
) -> FastAPI:
Expand Down Expand Up @@ -435,11 +436,27 @@ def build_app(
# consumer (corpus router, direct path, OpenAI surface, agent loop) reads
# this one attribute, so all of them gain fallback transparently.
base_router = retrieval_router or build_default_retrieval_router()
app.state.retrieval_router = (
routed = (
build_default_fallback_chain(base_router, config=fallback)
if fallback_enabled
else base_router
)
# ACL egress verifier (Step 6.4) — defense-in-depth backstop *above* the
# router, where the 6.3 push-down lives inside HybridRetriever. When wired,
# it re-checks the chunks each ``route`` returns against the principal's ACL
# labels and drops any over-privileged leak (emitting ``acl.egress_violation``).
# Gated on ``acl_enabled`` because the strict overlap drop only makes sense
# under the ACL label model; every downstream consumer (corpus router, agent,
# OpenAI surface) reads this one attribute, so all gain it transparently.
acl_verify_egress = acl_enabled and acl_verify_egress
if acl_verify_egress:
from rag_policy import AclEgressVerifier

from rag_gateway._acl_egress import AclEgressVerifyingRouter

routed = AclEgressVerifyingRouter(inner=routed, verifier=AclEgressVerifier())
app.state.retrieval_router = routed
app.state.acl_verify_egress = acl_verify_egress
app.state.reranker = reranker or build_default_reranker()
app.state.packer = packer or build_default_packer()
app.state.llm = llm if llm is not None else NoopLLM()
Expand Down
5 changes: 4 additions & 1 deletion apps/gateway/src/rag_gateway/wiring.py
Original file line number Diff line number Diff line change
Expand Up @@ -702,8 +702,10 @@ def build_app_from_config(cfg: RagConfig, **overrides: Any) -> FastAPI:
# process (the instrument is global, the callback reads current state).
_register_platform_metrics_once(overrides.get("drift_registry"), overrides.get("cost_tracker"))

# Label-based ACL push-down (Step 6.3) — opt-in; off by default.
# Label-based ACL push-down (Step 6.3) + egress verifier (Step 6.4) — opt-in;
# off by default. The verifier defaults on but only fires when ``enabled``.
acl_enabled = overrides.pop("acl_enabled", cfg.acl.enabled)
acl_verify_egress = overrides.pop("acl_verify_egress", cfg.acl.verify_egress)

return build_app(
corpus_store=corpus_store,
Expand All @@ -715,6 +717,7 @@ def build_app_from_config(cfg: RagConfig, **overrides: Any) -> FastAPI:
breaker_registry=breaker_registry,
quota_enforcer=quota_enforcer,
acl_enabled=acl_enabled,
acl_verify_egress=acl_verify_egress,
**overrides,
)

Expand Down
105 changes: 104 additions & 1 deletion apps/gateway/tests/test_acl.py
Original file line number Diff line number Diff line change
@@ -1,14 +1,31 @@
"""ACL push-down config + gateway wiring (Step 6.3)."""
"""ACL push-down (Step 6.3) + egress verifier (Step 6.4) config + gateway wiring."""

from __future__ import annotations

from typing import Any

from rag_config import RagConfig
from rag_config.schema import AclConfig
from rag_core.types import (
ChunkId,
ChunkRef,
Principal,
PrincipalId,
PrincipalKind,
QueryShape,
RequestContext,
RoutingDecision,
TenantId,
)
from rag_gateway import build_app
from rag_gateway._acl_egress import AclEgressVerifyingRouter
from rag_gateway.wiring import build_app_from_config
from rag_policy import AclPolicyEngine, NoopPolicyEngine


# ---------------------------------------------------------------------------
# Step 6.3 — push-down config + engine wrapping
# ---------------------------------------------------------------------------
def test_acl_config_defaults_off() -> None:
assert RagConfig().acl.enabled is False

Expand Down Expand Up @@ -37,3 +54,89 @@ def test_acl_decorates_without_absorbing_inner() -> None:
# the inner engine is preserved (here the default Noop), so ACL composes with
# — rather than replaces — the production PDP.
assert isinstance(engine.inner, NoopPolicyEngine)


# ---------------------------------------------------------------------------
# Step 6.4 — egress verifier config + router wrapping
# ---------------------------------------------------------------------------
def test_verify_egress_config_defaults_on() -> None:
assert RagConfig().acl.verify_egress is True


def test_verify_egress_inert_by_default() -> None:
# ACL off (default) → verifier off regardless of its own flag.
app = build_app()
assert app.state.acl_verify_egress is False
assert not isinstance(app.state.retrieval_router, AclEgressVerifyingRouter)


def test_verify_egress_wraps_router_when_acl_enabled() -> None:
app = build_app_from_config(RagConfig(acl=AclConfig(enabled=True)))
assert app.state.acl_verify_egress is True
assert isinstance(app.state.retrieval_router, AclEgressVerifyingRouter)


def test_verify_egress_off_runs_pushdown_only() -> None:
app = build_app_from_config(RagConfig(acl=AclConfig(enabled=True, verify_egress=False)))
assert app.state.acl_verify_egress is False
assert not isinstance(app.state.retrieval_router, AclEgressVerifyingRouter)
# push-down (6.3) is still wired
assert isinstance(app.state.policy_engine, AclPolicyEngine)


def test_verify_egress_inert_when_acl_disabled() -> None:
# verify_egress defaults True, but acl.enabled=False gates it off.
app = build_app_from_config(RagConfig(acl=AclConfig(enabled=False, verify_egress=True)))
assert app.state.acl_verify_egress is False
assert not isinstance(app.state.retrieval_router, AclEgressVerifyingRouter)


# ---------------------------------------------------------------------------
# Step 6.4 — wrapped router actually drops over-privileged chunks
# ---------------------------------------------------------------------------
class _StubRouter:
"""A SupportsRoute stub that returns whatever refs it's given (push-down bypassed)."""

def __init__(self, refs: list[ChunkRef]) -> None:
self._refs = refs

async def route(
self, ctx: RequestContext, *, text: str, **_: Any
) -> tuple[RoutingDecision, list[ChunkRef]]:
decision = RoutingDecision(
shape=QueryShape.MIXED, use_vector=True, use_keyword=False, use_graph=False
)
return decision, list(self._refs)


def _ref(chunk_id: str, labels: list[str]) -> ChunkRef:
return ChunkRef(
chunk_id=ChunkId(chunk_id),
tenant_id=TenantId("acme"),
score=1.0,
acl_labels=frozenset(labels),
)


def _ctx(labels: list[str]) -> RequestContext:
tid = TenantId("acme")
return RequestContext(
tenant_id=tid,
principal=Principal(
id=PrincipalId("p"),
kind=PrincipalKind.user,
display_name="p",
tenant_id=tid,
acl_labels=frozenset(labels),
),
)


async def test_wrapped_router_drops_over_privileged_chunks() -> None:
leaky = _StubRouter([_ref("c-eng", ["eng"]), _ref("c-fin", ["fin"]), _ref("c-none", [])])
app = build_app(acl_enabled=True, retrieval_router=leaky)
router = app.state.retrieval_router
assert isinstance(router, AclEgressVerifyingRouter)

_, refs = await router.route(_ctx(["eng"]), text="q", top_k=10)
assert {str(r.chunk_id) for r in refs} == {"c-eng"} # c-fin, c-none dropped at egress
Loading
Loading