officialCodeWork · officialCodeWork · May 24, 2026 · May 24, 2026
diff --git a/TRACKER.md b/TRACKER.md
@@ -10,7 +10,7 @@
 
 **Last updated:** 2026-05-24
 **Current phase:** Phase 1 — Ingestion + Knowledge Store
-**Next action:** Phase 1 Step 1.1c — PolicyEngine package (`rag-policy`): `PolicyEngine` SPI + noop impl as single PDP for ACL/PII/quotas/redaction
+**Next action:** Phase 1 Step 1.1d — Pipeline + Batcher primitives in `rag-core` (async DAG with bounded queues; DataLoader-pattern Batcher middleware coalescing concurrent SPI calls)
 
 > **Refactor window (Steps 1.1a–1.1f):** Before resuming the connectors framework (1.2), we insert a six-step refactor that locks in architecture + optimization decisions which are very expensive to retrofit later (PolicyEngine PDP, RequestContext-threaded SPIs, split Retrieval/Index backends, bulk + streaming + ID-only methods, Pipeline + Batcher primitives, three-way cache split, hot-path discipline). See [docs/adr/ADR-0005…0009] and [docs/architecture/policy-engine.md], [request-context.md], [caching.md], [performance.md].
 
@@ -32,14 +32,14 @@
 | Phase | Title | Steps | ✅ Done | Remaining |
 |-------|-------|------:|-------:|----------:|
 | 0 | Foundation | 13 | **13** | 0 |
-| 1 | Ingestion + Knowledge Store | 16 | **3** | 13 |
+| 1 | Ingestion + Knowledge Store | 16 | **4** | 12 |
 | 2 | Retrieval Engine | 11 | 0 | 11 |
 | 3 | Gateway & Agent Runtime | 11 | 0 | 11 |
 | 4 | Reliability | 6 | 0 | 6 |
 | 5 | Eval & Observability | 7 | 0 | 7 |
 | 6 | Governance & Tenancy | 10 | 0 | 10 |
 | 7 | Pilot, Harden, GA | 10 | 0 | 10 |
-| **Total** | | **84** | **16** | **68** |
+| **Total** | | **84** | **17** | **67** |
 
 ---
 
@@ -69,8 +69,8 @@
 |------|-------|--------|--------|----|-----------------|
 | 1.1 | Storage backends | ✅ | `build/phase-1/step-1.1-storage-backends` | [#40](https://github.com/officialCodeWork/AgentContextOS/pull/40) | `rag-backends` package: `PgVectorStore` (asyncpg + pgvector, ivfflat), `QdrantVectorStore` (query_points API), `RedisCache`, `S3Storage` (aioboto3, MinIO-compatible), `LocalFileStorage`; integration tests (skip-if-no-service); MinIO added to dev stack; `task test-integration` + `task test-backends`; ADR-0004 |
 | 1.1a | Core type & SPI refactor | ✅ | `build/phase-1/step-1.1a-core-type-spi-refactor` | _pending_ | `RequestContext` frozen model threaded through every SPI; `tenant_id` + `acl_labels` typed required on `Chunk`/`Embedding` (not metadata dict); `trust_level` on `Chunk` for prompt-injection defense; `dtype` on `Embedding` (float32/int8/binary); `BlobRef` for lazy chunk text; `QueryPlan` + `ChunkRef` + `Cost` + `PlanNode` types; typed `StageEvent`. `tests/contract/spi_signature.py` linter (RequestContext-first); rag-backends (`PgVectorStore`, `QdrantVectorStore`, `RedisCache`, `S3Storage`, `LocalFileStorage`) migrated; conformance + integration tests updated; `Budget.spend()` for agent-loop sub-turn budgets; schemas regenerated. ADR-0005 / ADR-0007 / ADR-0008 / ADR-0009 referenced. |
-| 1.1b | SPI split — Retrieval/Index, bulk + streaming + ID-only | ✅ | `build/phase-1/step-1.1b-spi-split-retrieval-index` | _pending_ | Split `VectorStore`/`KeywordStore`/`GraphStore` into `*RetrievalBackend` (read) + `*IndexBackend` (write) composite ABCs. `retrieve_ids` returns `list[ChunkRef]`; `hydrate` lives on the retrieval side (keyword full-Chunk; vector pass-through). Bulk: `bulk_index`/`bulk_delete` (+ graph bulk node/edge variants); streaming: `stream_index` async-iterator default that batches into `bulk_index`. `Embedder` split into single `embed` + canonical `bulk_embed`. New `IndexHint` + `WriteVolume` types passed to writes (per ADR-0009). Noop impls, `PgVectorStore`, `QdrantVectorStore` migrated; conformance + integration tests updated; `spi_signature.py` extended to enforce the split. Schemas regenerated (`IndexHint.json`). |
-| 1.1c | PolicyEngine package | ⏳ | 1.1a | — | New `packages/policy/` (`rag-policy`): `PolicyEngine` SPI + noop impl; single decision point consulted by every retrieval/ingest path for ACL, PII, quotas, redaction. Replaces scattered checks across Steps 1.7, 4.5, 6.3, 6.5. ADR-0005 finalized. |
+| 1.1b | SPI split — Retrieval/Index, bulk + streaming + ID-only | ✅ | `build/phase-1/step-1.1b-spi-split-retrieval-index` | [#45](https://github.com/officialCodeWork/AgentContextOS/pull/45) | Split `VectorStore`/`KeywordStore`/`GraphStore` into `*RetrievalBackend` (read) + `*IndexBackend` (write) composite ABCs. `retrieve_ids` returns `list[ChunkRef]`; `hydrate` lives on the retrieval side (keyword full-Chunk; vector pass-through). Bulk: `bulk_index`/`bulk_delete` (+ graph bulk node/edge variants); streaming: `stream_index` async-iterator default that batches into `bulk_index`. `Embedder` split into single `embed` + canonical `bulk_embed`. New `IndexHint` + `WriteVolume` types passed to writes (per ADR-0009). Noop impls, `PgVectorStore`, `QdrantVectorStore` migrated; conformance + integration tests updated; `spi_signature.py` extended to enforce the split. Schemas regenerated (`IndexHint.json`). |
+| 1.1c | PolicyEngine package | ✅ | `build/phase-1/step-1.1c-policy-engine-package` | _pending_ | New `packages/policy/` (`rag-policy` v0.1.0): `PolicyEngine` SPI + `NoopPolicyEngine` (always-ALLOW with tenant-scoped `filter_pushdown`); `PolicyDecision` enum (read_chunk / ingest_doc / egress_text / quota_check / rate_limit / execute_plan); `PolicyResult` (allow/deny/transform) with predicate helpers; `QuotaSubject` / `RateLimitSubject`; `FilterExpr` mini-language (Eq / AnyIn / And / Or / Not / TrueExpr) returned by `filter_pushdown`; `PolicyWriter` facade mirroring `AuditWriter` and emitting `policy.decision` structured logs via `rag-observability`. Coverage linter `tests/policy/coverage.py` greps for governance-relevant SPI calls without adjacent `PolicyEngine`/`PolicyWriter` consultation, with file allowlist that consumers shrink as they wire the PDP in. Workspace + pytest pythonpath updated. 20 conformance tests added. ADR-0005 finalized. |
 | 1.1d | Pipeline + Batcher primitives | ⏳ | 1.1a | — | `Pipeline` primitive in `rag-core`: async DAG with bounded queues, per-stage worker counts, backpressure (used by Step 1.10 write path). `Batcher[Req, Resp]` middleware (DataLoader pattern) coalescing concurrent SPI calls into batched provider calls; sits under Embedder/Reranker SPIs. |
 | 1.1e | Cache SPI split + perf discipline + async telemetry | ⏳ | 1.1a | — | Split `Cache` into `EmbeddingCache` (key model_id+text_hash), `RetrievalCache` (plan_hash+corpus_version), `AnswerCache` (plan_hash+corpus_version+policy_version). Each has distinct invalidation. Hot-path convention doc (Pydantic at SPI boundary, `model_construct`/msgspec inside). Async telemetry path with bounded buffer + drop-on-overflow counter. |
 | 1.1f | ADRs 0005–0009 + reviewer checklist | ⏳ | 1.1a–1.1e | — | ADR-0005 (PolicyEngine PDP), ADR-0006 (two-stage reranker default), ADR-0007 (tiered storage with BlobRef), ADR-0008 (cost-aware planner replacing reactive fallback), ADR-0009 (vector index strategy + quantization). Reviewer checklist in `docs/architecture/performance.md`. |
@@ -212,6 +212,8 @@
 | [#38](https://github.com/officialCodeWork/AgentContextOS/pull/38) | ci: wire RAG001 logging check into ci.yml on all OSes | `fix/ci-rag001-gate` | ✅ Merged | 2026-05-23 |
 | [#40](https://github.com/officialCodeWork/AgentContextOS/pull/40) | feat(backends): storage backend plugins — pgvector, Qdrant, Redis, S3 (Step 1.1) | `build/phase-1/step-1.1-storage-backends` | ✅ Merged | 2026-05-24 |
 | [#41](https://github.com/officialCodeWork/AgentContextOS/pull/41) | docs(planning): Phase 1 architecture-refactor window (Steps 1.1a–1.1f) + ADRs 0005–0009 | `planning/phase-1-architecture-refactor` | ✅ Merged | 2026-05-24 |
+| [#44](https://github.com/officialCodeWork/AgentContextOS/pull/44) | refactor(core): RequestContext + ctx-threaded SPI (Step 1.1a) | `build/phase-1/step-1.1a-core-type-spi-refactor` | ✅ Merged | 2026-05-24 |
+| [#45](https://github.com/officialCodeWork/AgentContextOS/pull/45) | refactor(core): SPI split — RetrievalBackend / IndexBackend (Step 1.1b) | `build/phase-1/step-1.1b-spi-split-retrieval-index` | ✅ Merged | 2026-05-24 |
 
 ---
 

diff --git a/docs/README.md b/docs/README.md
@@ -21,6 +21,7 @@
 | [ragctl.md](reference/ragctl.md) | Full `ragctl` command reference — public usage, internals, extension points |
 | [backends.md](reference/backends.md) | `rag-backends` reference — PgVectorStore, QdrantVectorStore, RedisCache, S3Storage, LocalFileStorage |
 | [rag-core.md](reference/rag-core.md) | `rag-core` type surface — `RequestContext`, `Budget`, `BlobRef`, `QueryPlan`, `ChunkRef`, `StageEvent` |
+| [rag-policy.md](reference/rag-policy.md) | `rag-policy` reference — `PolicyEngine`, `PolicyWriter`, `PolicyDecision`, `PolicyResult`, `FilterExpr` |
 
 ## guides/
 

diff --git a/docs/reference/rag-policy.md b/docs/reference/rag-policy.md
@@ -0,0 +1,204 @@
+# Reference — `rag-policy`
+
+The `rag-policy` package houses the **PolicyEngine SPI** — the single
+Policy Decision Point (PDP) for AgentContextOS governance.  See
+[ADR-0005](../adr/ADR-0005-policy-engine.md) for the decision and
+[docs/architecture/policy-engine.md](../architecture/policy-engine.md) for
+the design.
+
+This page focuses on the package's public surface as it lands in Step 1.1c.
+Consumers (gateway, ingest pipeline) will arrive in later phases; the
+coverage linter described below already guarantees that those consumers
+cannot land without consulting the PDP.
+
+---
+
+## Overview
+
+Five governance touchpoints in the V1 plan (PII at ingest, quotas, ACL
+push-down, ACL egress verifier, PII at egress) are consolidated behind one
+SPI:
+
+```python
+class PolicyEngine(HealthCheckMixin, ABC):
+    async def evaluate(ctx, decision, subject) -> PolicyResult: ...
+    async def filter_pushdown(ctx, decision) -> FilterExpr: ...
+```
+
+Every retrieval / ingest / egress code path consults an engine instance.  A
+coverage linter (`tests/policy/coverage.py`) fails CI when a known
+governance-relevant SPI call site is found without an adjacent
+`PolicyEngine` / `PolicyWriter` consultation.
+
+---
+
+## Usage
+
+```python
+from rag_policy import (
+    NoopPolicyEngine,
+    PolicyDecision,
+    PolicyResult,
+    PolicyWriter,
+)
+
+policy = PolicyWriter(NoopPolicyEngine())
+
+# 1. Push-down: PolicyEngine returns a FilterExpr injected into the backend.
+filter_expr = await policy.filter_pushdown(ctx, PolicyDecision.read_chunk)
+candidates = await backend.retrieve_ids(ctx, query_vec, top_k=200, corpus_ids=[])
+
+# 2. Per-item evaluation (rare — most filtering happens push-down).
+allowed = []
+for ref in candidates:
+    result = await policy.evaluate(ctx, PolicyDecision.read_chunk, ref)
+    if result.is_allow():
+        allowed.append(ref)
+    elif result.is_transform():
+        allowed.append(result.transformed)
+    # DENY: skipped, logged, audited via the writer's structured-log entry.
+```
+
+Reach for `PolicyWriter` rather than calling `PolicyEngine` directly — it
+emits the `policy.decision` structured log entry every governance decision
+should produce.
+
+### `PolicyDecision` values
+
+| Decision | Subject | Typical results |
+|---|---|---|
+| `read_chunk` | `Chunk` or `ChunkRef` | `allow` / `deny` (ACL mismatch) |
+| `ingest_doc` | `Document` | `allow` / `deny` (size, MIME) / `transform` (PII redaction) |
+| `egress_text` | `str` or `Chunk` | `allow` / `transform` (PII redaction) |
+| `quota_check` | `QuotaSubject(tenant_id, kind, amount)` | `allow` / `deny` (over-quota) |
+| `rate_limit` | `RateLimitSubject(tenant_id, endpoint)` | `allow` / `deny` (rate exceeded) |
+| `execute_plan` | `QueryPlan` | `allow` / `transform` (degraded plan under cost cap) |
+
+Adding a new decision requires updating `NoopPolicyEngine`, writing
+conformance tests, and amending this page.
+
+### `FilterExpr` mini-language
+
+Returned by `filter_pushdown`; consumed by retrieval backends.  Minimal
+shape at 1.1c — enough for tenant + ACL scoping:
+
+```python
+from rag_policy import and_, any_in, eq, FilterExpr
+
+expr: FilterExpr = and_(
+    eq("tenant_id", str(ctx.tenant_id)),
+    any_in("acl_labels", list(ctx.principal.acl_labels)),
+)
+```
+
+Nodes: `Eq`, `AnyIn`, `And`, `Or`, `Not`, `TrueExpr`.  Each is a frozen
+Pydantic model; the union is discriminated on the `kind` tag so backends
+switch over the shape rather than introspecting fields.
+
+---
+
+## Internals
+
+### Why a separate package
+
+Three reasons:
+
+1. **Lifecycle independence.** Policy will evolve (new decisions, new
+   backends like an OPA adapter) on a different cadence from the core types.
+2. **Dependency surface.** `rag-policy` depends only on `rag-core` and
+   `rag-observability`.  Production deployments that disable the noop and
+   load an external PDP do not need to drop in a heavier package.
+3. **CLAUDE.md graph.** The standing constraint records `policy → core`;
+   keeping it in its own package makes the boundary mechanical, not just
+   social.
+
+### `PolicyWriter` vs `PolicyEngine`
+
+`PolicyEngine` is the SPI authors implement.  `PolicyWriter` is the facade
+consumers should *use*.  The split mirrors `AuditStore` ↔ `AuditWriter` in
+`rag-core` — the writer adds the cross-cutting concern (structured-log
+emission) that every consumer needs but no SPI implementation should
+duplicate.
+
+Writer logs go through `rag_observability.logging.get_logger(__name__)` and
+land in the standard 7-field JSON envelope under the message
+`policy.decision`.  Fields: `rag_decision`, `rag_outcome`, `rag_tenant_id`,
+`rag_principal_id`, `rag_request_id`, `rag_reason`.
+
+### Coverage linter
+
+`tests/policy/coverage.py` greps for direct calls to governance-relevant
+SPI methods (`retrieve_ids`, `hydrate`, `bulk_index`, `stream_index`,
+`bulk_embed`, `complete`) in files outside an allowlist.  If a file
+contains such a call but no `PolicyEngine` / `PolicyWriter` marker, CI
+fails.
+
+The allowlist at the top of `coverage.py` contains:
+
+* SPI abstracts and noop reference impls (the call surface itself).
+* Real backend impls (governance happens at the call site, not inside the
+  backend).
+* Tests (fixtures exercise the SPI directly).
+* The `rag-policy` package (its writer/engine *are* the policed surface).
+
+As Steps 1.10, 3.1, etc. add consumers, their files come off the allowlist
+as `PolicyEngine` is wired in.  A future Step 1.1f or 3.x tightens the
+linter from file-allowlist to call-pattern matching.
+
+### Performance
+
+* `NoopPolicyEngine.evaluate` is essentially free; `filter_pushdown`
+  constructs one `And(Eq(...))` per call.
+* Production impls are expected to cache decisions keyed by
+  `(ctx.principal, decision, subject_hash)` for the lifetime of a
+  `RequestContext`.  See [docs/architecture/performance.md](../architecture/performance.md)
+  for the published p99 budget (`PolicyEngine.evaluate` noop ≤ 100 µs).
+
+---
+
+## Extension points
+
+Implement `rag_policy.PolicyEngine`:
+
+```python
+from rag_policy import (
+    FilterExpr,
+    PolicyDecision,
+    PolicyEngine,
+    PolicyResult,
+    and_,
+    any_in,
+    eq,
+)
+
+class MyOrgPolicyEngine(PolicyEngine):
+    async def evaluate(self, ctx, decision, subject) -> PolicyResult:
+        if decision is PolicyDecision.read_chunk:
+            if subject.acl_labels & ctx.principal.acl_labels:
+                return PolicyResult.allow()
+            return PolicyResult.deny("acl-mismatch")
+        return PolicyResult.allow()
+
+    async def filter_pushdown(self, ctx, decision) -> FilterExpr:
+        return and_(
+            eq("tenant_id", str(ctx.tenant_id)),
+            any_in("acl_labels", list(ctx.principal.acl_labels)),
+        )
+
+    async def health(self) -> bool:
+        return True
+```
+
+Register at the gateway composition root.  Future built-ins on the
+roadmap: `OpaPolicyEngine` (sidecar delegation), `CedarPolicyEngine`
+(in-process AWS Cedar).
+
+---
+
+## Related
+
+* [ADR-0005](../adr/ADR-0005-policy-engine.md) — the decision.
+* [docs/architecture/policy-engine.md](../architecture/policy-engine.md) — design.
+* [docs/architecture/request-context.md](../architecture/request-context.md) — the envelope `evaluate` receives.
+* [docs/reference/rag-core.md](rag-core.md) — types referenced by subjects (`Chunk`, `ChunkRef`, `Document`).
+* TRACKER.md Step 1.1c.
diff --git a/packages/policy/README.md b/packages/policy/README.md
@@ -0,0 +1,40 @@
+# rag-policy
+
+Central **Policy Decision Point (PDP)** for AgentContextOS — one auditable
+surface for ACL enforcement, PII handling, quotas, redaction, and rate
+limits.  Every retrieval / ingest / egress code path consults the
+`PolicyEngine`; this replaces the five scattered governance touchpoints
+described in [ADR-0005](../../docs/adr/ADR-0005-policy-engine.md).
+
+## Public surface
+
+```python
+from rag_policy import (
+    FilterExpr,
+    NoopPolicyEngine,
+    PolicyDecision,
+    PolicyEngine,
+    PolicyResult,
+    PolicyWriter,
+)
+```
+
+* `PolicyEngine` — SPI ABC.  Implement `evaluate(ctx, decision, subject)` and
+  `filter_pushdown(ctx, decision)`.
+* `NoopPolicyEngine` — always-ALLOW reference impl; tenant-scoped pushdown.
+* `PolicyWriter` — facade composing `evaluate()` with structured-log emission
+  (mirrors `AuditWriter`).
+* `PolicyDecision` — typed enum (`READ_CHUNK`, `INGEST_DOC`, `EGRESS_TEXT`,
+  `QUOTA_CHECK`, `RATE_LIMIT`, `EXECUTE_PLAN`).
+* `PolicyResult` — frozen union: `allow()`, `deny(reason)`,
+  `transform(subject)`.
+* `FilterExpr` — minimal filter mini-language returned by `filter_pushdown`
+  and injected into retrieval backends.
+
+See [docs/architecture/policy-engine.md](../../docs/architecture/policy-engine.md)
+for the full design and [docs/reference/rag-policy.md](../../docs/reference/rag-policy.md)
+for the API reference.
+
+## Step
+
+Introduced in Phase 1 Step 1.1c.
diff --git a/packages/policy/pyproject.toml b/packages/policy/pyproject.toml
@@ -0,0 +1,25 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "rag-policy"
+version = "0.1.0"
+description = "AgentContextOS — central Policy Decision Point (PolicyEngine, PolicyWriter)"
+readme = "README.md"
+requires-python = ">=3.12"
+dependencies = [
+    "pydantic>=2.7",
+    "rag-core",
+    "rag-observability",
+]
+
+[tool.uv.sources]
+rag-core = { workspace = true }
+rag-observability = { workspace = true }
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/rag_policy"]
+
+[tool.hatch.metadata]
+allow-direct-references = true
diff --git a/packages/policy/src/rag_policy/__init__.py b/packages/policy/src/rag_policy/__init__.py
@@ -0,0 +1,59 @@
+"""rag-policy — central Policy Decision Point for AgentContextOS.
+
+Public surface introduced in Phase 1 Step 1.1c.  See
+[docs/architecture/policy-engine.md](../../../docs/architecture/policy-engine.md)
+and [ADR-0005](../../../docs/adr/ADR-0005-policy-engine.md).
+"""
+
+from rag_policy.engine import PolicyEngine
+from rag_policy.filter import (
+    And,
+    AnyIn,
+    Eq,
+    FilterExpr,
+    Not,
+    Or,
+    TrueExpr,
+    and_,
+    any_in,
+    eq,
+    not_,
+    or_,
+    true,
+)
+from rag_policy.noop import NoopPolicyEngine
+from rag_policy.types import (
+    PolicyDecision,
+    PolicyResult,
+    QuotaSubject,
+    RateLimitSubject,
+)
+from rag_policy.writer import PolicyWriter
+
+__version__ = "0.1.0"
+
+__all__ = [
+    # engine
+    "PolicyEngine",
+    "NoopPolicyEngine",
+    "PolicyWriter",
+    # decisions + results
+    "PolicyDecision",
+    "PolicyResult",
+    "QuotaSubject",
+    "RateLimitSubject",
+    # filter mini-language
+    "FilterExpr",
+    "Eq",
+    "AnyIn",
+    "And",
+    "Or",
+    "Not",
+    "TrueExpr",
+    "eq",
+    "any_in",
+    "and_",
+    "or_",
+    "not_",
+    "true",
+]