officialCodeWork · officialCodeWork · May 23, 2026 · May 23, 2026
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -12,17 +12,19 @@ so every backend (vector store, embedder, LLM, etc.) is swappable without touchi
 what is in progress, and what is next. The memory index at
 `~/.claude/projects/.../memory/MEMORY.md` carries supporting facts between sessions.
 
-Current phase: **Phase 0 — Foundation** (10 of 13 steps complete as of 2026-05-23).
-Next step: **0.8 — Eval skeleton** (RAGAS spike, `ragctl eval`, golden-set schema).
+Current phase: **Phase 1 — Ingestion + Knowledge Store** (1 of 16 steps complete as of 2026-05-24).
+Next step: **1.1a — Core type & SPI refactor** (RequestContext threaded through every SPI; typed ACL on Chunk/Embedding; BlobRef/dtype/trust_level/QueryPlan/ChunkRef/StageEvent). Part of the Phase 1 refactor window (1.1a–1.1f) inserted ahead of Step 1.2 to lock in architecture + optimization decisions that are very expensive to retrofit. See ADRs 0005–0009 and `docs/architecture/{policy-engine,request-context,caching,performance}.md`.
 
 ## Repo layout
 
 ```
 AgentContextOS/
 ├── packages/
-│   ├── core/           # rag-core — domain types, errors, SPI interfaces, noop impls, AuditWriter
+│   ├── core/           # rag-core — domain types, errors, SPI interfaces, noop impls, AuditWriter, Pipeline, Batcher
 │   ├── config/         # rag-config — rag.yaml schema (Pydantic v2), loader, ragctl config CLI
-│   └── observability/  # rag-observability — structured JSON logger, set_log_context(), event registry
+│   ├── observability/  # rag-observability — structured JSON logger, set_log_context(), event registry, async exporter
+│   ├── policy/         # rag-policy — PolicyEngine (central PDP for ACL/PII/quotas/redaction); added in Step 1.1c
+│   └── backends/       # rag-backends — PgVectorStore, QdrantVectorStore, RedisCache, S3Storage, LocalFileStorage
 ├── apps/
 │   └── gateway/        # rag-gateway (FastAPI, future home of the REST + gRPC service)
 ├── tests/
@@ -43,9 +45,11 @@ AgentContextOS/
 
 | Package | Import root | Purpose |
 |---------|-------------|---------|
-| `rag-core` v0.2.0 | `rag_core` | Domain types (Pydantic v2 frozen models), 14-type error hierarchy, plugin SPI ABCs + noop impls, `AuditWriter` |
+| `rag-core` v0.2.0 | `rag_core` | Domain types (Pydantic v2 frozen models), 14-type error hierarchy, plugin SPI ABCs + noop impls, `AuditWriter`, `Pipeline` + `Batcher` primitives (added in Step 1.1d) |
 | `rag-config` v0.1.0 | `rag_config` | `rag.yaml` schema, env-var interpolation loader, `ConfigWatcher` hot-reload, `ragctl config` CLI |
-| `rag-observability` v0.1.0 | `rag_observability` | 7-field JSON structured logger, `set_log_context()` contextvar manager, event registry |
+| `rag-observability` v0.1.0 | `rag_observability` | 7-field JSON structured logger, `set_log_context()` contextvar manager, event registry, async exporter with drop-on-overflow |
+| `rag-policy` v0.1.0 (Step 1.1c) | `rag_policy` | `PolicyEngine` SPI + noop impl; central PDP for ACL / PII / quotas / redaction; `PolicyWriter` facade |
+| `rag-backends` v0.1.0 | `rag_backends` | Real backend implementations: PgVectorStore, QdrantVectorStore, RedisCache, S3Storage, LocalFileStorage |
 | `rag-gateway` v0.1.0 | `rag_gateway` | FastAPI gateway (scaffold only — real routes come in Phase 3) |
 
 ## Key architecture patterns
@@ -88,7 +92,36 @@ writer.write(AuditEvent(tenant_id=..., action="retrieval.query", ...))
 ### Domain types
 All in `rag_core.types`. All models are **frozen Pydantic v2** — create new instances,
 never mutate. Key types: `TraceContext`, `AuditEvent`, `Document`, `Chunk`, `Embedding`,
-`Query`, `RetrievalResult`, `Citation`.
+`Query`, `RetrievalResult`, `Citation`, `RequestContext`, `QueryPlan`, `ChunkRef`,
+`BlobRef`, `StageEvent`.
+
+### RequestContext threading (Step 1.1a)
+Every SPI method takes `ctx: RequestContext` as its first argument. The gateway
+constructs `ctx` once at the boundary; downstream code treats it as trusted and
+immutable. See [docs/architecture/request-context.md](docs/architecture/request-context.md).
+
+```python
+async def retrieve_ids(
+    self,
+    ctx: RequestContext,            # always first
+    query: Query,
+    filter_expr: FilterExpr | None = None,
+) -> list[ChunkRef]: ...
+```
+
+A signature linter in `tests/contract/spi_signature.py` fails CI on additions
+that omit `ctx`.
+
+### PolicyEngine (Step 1.1c)
+Every retrieval / ingest / egress path consults `PolicyEngine.evaluate()` or
+uses `PolicyWriter`. A coverage linter (`tests/policy/coverage.py`) catches
+bypasses. See [docs/architecture/policy-engine.md](docs/architecture/policy-engine.md).
+
+### Hot-path discipline (Step 1.1e)
+Pydantic at SPI boundaries only; inside hot loops use dataclasses, msgspec, or
+`Model.model_construct()`. Per-SPI p99 budgets enforced by conformance tests.
+Full convention + reviewer checklist in
+[docs/architecture/performance.md](docs/architecture/performance.md).
 
 ## Commands
 
@@ -195,5 +228,9 @@ Add a **Documentation** section listing the doc file(s) added or updated so revi
 - `mypy --strict` on all `packages/` and `apps/gateway/` source.
 - No `logging.getLogger()` outside the approved allowlist (RAG001).
 - All Pydantic domain models use `model_config = {"frozen": True}`.
-- No circular imports between packages: `rag-core` has no dependency on `rag-observability`
-  or `rag-config`. The dependency graph is: gateway → observability → core, config → core.
+- No circular imports between packages: `rag-core` has no dependency on `rag-observability`,
+  `rag-config`, `rag-policy`, or `rag-backends`. The dependency graph is:
+  gateway → observability → core; config → core; policy → core; backends → core.
+- Every SPI method takes `ctx: RequestContext` first (Step 1.1a onward).
+- Every retrieval / ingest / egress call site consults `PolicyEngine` (Step 1.1c onward).
+- Hot-path discipline: Pydantic at SPI boundaries only; `model_construct()` / msgspec inside loops.
diff --git a/TRACKER.md b/TRACKER.md
@@ -10,7 +10,9 @@
 
 **Last updated:** 2026-05-24
 **Current phase:** Phase 1 — Ingestion + Knowledge Store
-**Next action:** Phase 1 Step 1.2 — Connectors framework
+**Next action:** Phase 1 Step 1.1a — Core type & SPI refactor (RequestContext, typed ACL, dtype, BlobRef, QueryPlan, ChunkRef, StageEvent)
+
+> **Refactor window (Steps 1.1a–1.1f):** Before resuming the connectors framework (1.2), we insert a six-step refactor that locks in architecture + optimization decisions which are very expensive to retrofit later (PolicyEngine PDP, RequestContext-threaded SPIs, split Retrieval/Index backends, bulk + streaming + ID-only methods, Pipeline + Batcher primitives, three-way cache split, hot-path discipline). See [docs/adr/ADR-0005…0009] and [docs/architecture/policy-engine.md], [request-context.md], [caching.md], [performance.md].
 
 ---
 
@@ -30,14 +32,14 @@
 | Phase | Title | Steps | ✅ Done | Remaining |
 |-------|-------|------:|-------:|----------:|
 | 0 | Foundation | 13 | **13** | 0 |
-| 1 | Ingestion + Knowledge Store | 10 | **1** | 9 |
-| 2 | Retrieval Engine | 10 | 0 | 10 |
+| 1 | Ingestion + Knowledge Store | 16 | **1** | 15 |
+| 2 | Retrieval Engine | 11 | 0 | 11 |
 | 3 | Gateway & Agent Runtime | 11 | 0 | 11 |
 | 4 | Reliability | 6 | 0 | 6 |
 | 5 | Eval & Observability | 7 | 0 | 7 |
 | 6 | Governance & Tenancy | 10 | 0 | 10 |
 | 7 | Pilot, Harden, GA | 10 | 0 | 10 |
-| **Total** | | **77** | **14** | **63** |
+| **Total** | | **84** | **14** | **70** |
 
 ---
 
@@ -66,7 +68,13 @@
 | Step | Title | Status | Branch | PR | Key Deliverables |
 |------|-------|--------|--------|----|-----------------|
 | 1.1 | Storage backends | ✅ | `build/phase-1/step-1.1-storage-backends` | [#40](https://github.com/officialCodeWork/AgentContextOS/pull/40) | `rag-backends` package: `PgVectorStore` (asyncpg + pgvector, ivfflat), `QdrantVectorStore` (query_points API), `RedisCache`, `S3Storage` (aioboto3, MinIO-compatible), `LocalFileStorage`; integration tests (skip-if-no-service); MinIO added to dev stack; `task test-integration` + `task test-backends`; ADR-0004 |
-| 1.2 | Connectors framework | ⏳ | — | — | `Connector` SPI implementation; built-in: filesystem, S3, GCS; crawler base class |
+| 1.1a | Core type & SPI refactor | ⏳ | — | — | `RequestContext` frozen model threaded through every SPI; `tenant_id` + `acl_labels` typed required on `Chunk`/`Embedding` (not metadata dict); `trust_level` on `Chunk` for prompt-injection defense; `dtype` on `Embedding` (float32/int8/binary); `BlobRef` for lazy chunk text; `QueryPlan` + `ChunkRef` types; typed `StageEvent`. Conformance tests updated. ADR-0005 (PolicyEngine) + ADR-0008 (cost-aware planner) referenced. |
+| 1.1b | SPI split — Retrieval/Index, bulk + streaming + ID-only | ⏳ | 1.1a | — | Split `VectorStore`/`KeywordStore`/`GraphStore` into `RetrievalBackend` (read) + `IndexBackend` (write); add `bulk_embed`/`bulk_index`/`bulk_delete`; add async-iterator ingest variants; add `retrieve_ids` + `hydrate` pair; add `IndexHint` parameter for scale-tier index selection. ADR-0009 (vector index strategy) authored. |
+| 1.1c | PolicyEngine package | ⏳ | 1.1a | — | New `packages/policy/` (`rag-policy`): `PolicyEngine` SPI + noop impl; single decision point consulted by every retrieval/ingest path for ACL, PII, quotas, redaction. Replaces scattered checks across Steps 1.7, 4.5, 6.3, 6.5. ADR-0005 finalized. |
+| 1.1d | Pipeline + Batcher primitives | ⏳ | 1.1a | — | `Pipeline` primitive in `rag-core`: async DAG with bounded queues, per-stage worker counts, backpressure (used by Step 1.10 write path). `Batcher[Req, Resp]` middleware (DataLoader pattern) coalescing concurrent SPI calls into batched provider calls; sits under Embedder/Reranker SPIs. |
+| 1.1e | Cache SPI split + perf discipline + async telemetry | ⏳ | 1.1a | — | Split `Cache` into `EmbeddingCache` (key model_id+text_hash), `RetrievalCache` (plan_hash+corpus_version), `AnswerCache` (plan_hash+corpus_version+policy_version). Each has distinct invalidation. Hot-path convention doc (Pydantic at SPI boundary, `model_construct`/msgspec inside). Async telemetry path with bounded buffer + drop-on-overflow counter. |
+| 1.1f | ADRs 0005–0009 + reviewer checklist | ⏳ | 1.1a–1.1e | — | ADR-0005 (PolicyEngine PDP), ADR-0006 (two-stage reranker default), ADR-0007 (tiered storage with BlobRef), ADR-0008 (cost-aware planner replacing reactive fallback), ADR-0009 (vector index strategy + quantization). Reviewer checklist in `docs/architecture/performance.md`. |
+| 1.2 | Connectors framework | ⏳ | 1.1a–1.1f | — | `Connector` SPI implementation (now receives `RequestContext` and `ConnectorState` watermark); built-in: filesystem, S3, GCS; crawler base class |
 | 1.3 | Document parsers | ⏳ | — | — | PDF, DOCX, PPTX, XLSX, HTML, Markdown, plain text, JSON, CSV, YAML parsers; MIME detection |
 | 1.4 | OCR pipeline | ⏳ | — | — | Tesseract + PaddleOCR plugins; image region extraction; confidence scoring |
 | 1.5 | Structure-aware chunker | ⏳ | — | — | Heading-based chunking, parent-child `Chunk.parent_id` links, sentence-boundary, overlap, size normalization |
@@ -92,6 +100,7 @@
 | 2.8 | Context packer | ⏳ | — | — | Dedup, reordering, conflict detection, token budget enforcement; `pack` span |
 | 2.9 | GraphRAG | ⏳ | — | — | Community detection, sub-graph summarization, graph-aware retrieval |
 | 2.10 | Retrieval router | ⏳ | — | — | Per-query backend selection; degradation detection; fallback to BM25-only |
+| 2.11 | Agent-loop validation spike | ⏳ | 2.1–2.10 | — | Thin `agent_loop_v0` — iterative retrieve with budget enforcement (tokens/cost/wall/iter from `RequestContext.budget`); 50-query agent-style harness validates Phase 2 retrieval design against agent access patterns (shared embedding cache, partial plan reuse, sticky corpus routing) before committing to Phase 3. Output: gap list + any Phase 2 fixes before Phase 3.1 starts. |
 
 ---
 

diff --git a/docs/README.md b/docs/README.md
@@ -6,6 +6,10 @@
 |------|-------------|
 | [RAG-Platform-HLD.md](architecture/RAG-Platform-HLD.md) | High-Level Design: problem statement, layered architecture, pluggable backends, `rag.yaml` contract, deployment topologies, KPIs, risks, glossary |
 | [high-level-architecture.svg](architecture/high-level-architecture.svg) | Layered architecture diagram (SVG) |
+| [request-context.md](architecture/request-context.md) | `RequestContext` — the per-request envelope threaded through every SPI (tenant, principal, ACLs, PII policy, trace, budget) |
+| [policy-engine.md](architecture/policy-engine.md) | `PolicyEngine` (PDP) — single decision point for ACL, PII, quotas, redaction; replaces scattered governance checks |
+| [caching.md](architecture/caching.md) | Three-cache split: `EmbeddingCache`, `RetrievalCache`, `AnswerCache` — distinct invalidation rules |
+| [performance.md](architecture/performance.md) | Hot-path discipline, per-SPI p99 budgets, async telemetry, reviewer checklist |
 | [eval-skeleton.md](architecture/eval-skeleton.md) | Eval framework architecture: golden-set schema, metric functions, RAGAS spike, `ragctl eval` CLI, extension points |
 | [iac.md](architecture/iac.md) | IaC overview: Terraform module design, Helm chart structure, dev/prod environments, extension points |
 | [storage-backends.md](architecture/storage-backends.md) | Storage backend architecture: PgVector, Qdrant, Redis, S3/MinIO, tenant isolation, integration test strategy |
@@ -32,6 +36,11 @@
 | [ADR-0002-eval-framework.md](adr/ADR-0002-eval-framework.md) | Decision: pure-Python Tier 1 metrics always-on; RAGAS as optional Tier 2 |
 | [ADR-0003-iac-kubernetes-native.md](adr/ADR-0003-iac-kubernetes-native.md) | Decision: Kubernetes-native Terraform modules over cloud-provider-specific RDS/ElastiCache |
 | [ADR-0004-storage-backends.md](adr/ADR-0004-storage-backends.md) | Decision: single `rag-backends` package, MinIO for S3-compatible dev storage, graceful integration test skip |
+| [ADR-0005-policy-engine.md](adr/ADR-0005-policy-engine.md) | Decision: `rag-policy` package as central PDP for ACL/PII/quotas/redaction; supersedes Step 6.4 egress verifier |
+| [ADR-0006-two-stage-rerank.md](adr/ADR-0006-two-stage-rerank.md) | Decision: `Reranker` SPI ships as two-stage cascade (fast bi-encoder → cross-encoder) from day 1 |
+| [ADR-0007-tiered-storage.md](adr/ADR-0007-tiered-storage.md) | Decision: `Chunk.text: str \| BlobRef` enables lazy hydration and hot/warm/cold tiering |
+| [ADR-0008-cost-aware-planner.md](adr/ADR-0008-cost-aware-planner.md) | Decision: `QueryPlan` carries `estimated_cost`; fallback triggered planner-side before dispatch, not by post-hoc timeout |
+| [ADR-0009-vector-index-strategy.md](adr/ADR-0009-vector-index-strategy.md) | Decision: `IndexHint` selects index by scale tier (flat / ivfflat / HNSW / IVF-PQ / DiskANN); `Embedding.dtype` enables int8 / binary quantization |
 
 ## research/
 

diff --git a/docs/adr/ADR-0005-policy-engine.md b/docs/adr/ADR-0005-policy-engine.md
@@ -0,0 +1,80 @@
+# ADR-0005 — PolicyEngine as central PDP
+
+**Status:** Accepted
+**Date:** 2026-05-24
+**Deciders:** Core team
+**Step:** Phase 1 Step 1.1c
+
+---
+
+## Context
+
+Governance in the V1.0 plan was scattered across five steps:
+
+- 1.7 — PII detection at ingest
+- 4.5 — Quotas + rate limits
+- 6.3 — ACL push-down at retrieval
+- 6.4 — ACL egress verifier (defense-in-depth)
+- 6.5 — PII egress scanner
+
+Each is a separate code path with its own call sites. This is exactly how real-world governance leaks happen: a new feature adds a code path that forgets one of the checks. Step 6.4 (egress verifier) is a tacit admission that 6.3 alone is too fragile — the system needs *two* layers because the *first* one might be bypassed.
+
+The pattern is well-understood: a Policy Decision Point (PDP) consolidates these into one auditable surface. OPA / Cedar / AuthZed are existing implementations of this pattern in adjacent domains.
+
+## Decision
+
+### 1. Introduce a `rag-policy` workspace package
+
+New package `packages/policy/` (import root `rag_policy`). Sits between `rag-core` and consumers; depends only on `rag-core` types (`RequestContext`).
+
+### 2. `PolicyEngine` is the single SPI for all governance decisions
+
+```python
+class PolicyEngine(ABC):
+    async def evaluate(
+        self,
+        ctx: RequestContext,
+        decision: PolicyDecision,   # READ_CHUNK | INGEST_DOC | EGRESS_TEXT | QUOTA_CHECK | ...
+        subject: Any,
+    ) -> PolicyResult                # ALLOW | DENY(reason) | TRANSFORM(redacted_subject)
+
+    async def filter_pushdown(
+        self,
+        ctx: RequestContext,
+        decision: PolicyDecision,
+    ) -> FilterExpr                  # Injected into retrieval calls
+```
+
+Every retrieval, ingest, and egress call site **must** consult `PolicyEngine`. Enforced by a coverage linter (`tests/policy/coverage.py`) that fails CI on un-policed calls.
+
+### 3. `PolicyWriter` facade pattern (mirrors `AuditWriter`)
+
+Composes `PolicyEngine` evaluation with structured-log emission and audit-event write. Single import for consumers.
+
+### 4. Step 6.4 (ACL egress verifier) is marked redundant
+
+Because no retrieval call site bypasses the PDP, defense-in-depth is provided by the PDP design itself, not by a second checking layer. The TRACKER entry and planning file for Step 6.4 will be updated to "Superseded by ADR-0005" in the Phase 6 refactor pass.
+
+### 5. NoopPolicyEngine reference impl
+
+Always-ALLOW noop for OSS / dev. Production tenants override with their own implementation (or future built-ins).
+
+## Consequences
+
+- **Positive:** Governance decisions auditable from one surface. Adding a new governance rule (e.g., GDPR data-residency check) is one PolicyEngine extension, not five touchpoints. ACL pushdown + egress check unified. Coverage testable.
+- **Positive:** Pattern enables future plug-ins to OPA / Cedar without re-architecting consumer code.
+- **Negative:** Every existing retrieval / ingest test fixture must now construct a `RequestContext` and pass through PolicyEngine. One-time refactor cost.
+- **Negative:** Performance: a synchronous `evaluate` adds ~tens of microseconds per call. Mitigated by (a) noop impl is essentially free, (b) batch evaluation API, (c) plan-time evaluation cached per `RequestContext` lifetime.
+- **Neutral:** Step 6.4 removed from the critical path; team-week budget reallocated.
+
+## Alternatives considered
+
+- **Status quo (scattered checks):** Rejected. Predictable failure mode (one missed call site = leak).
+- **Embed a third-party PDP (OPA sidecar) from day 1:** Rejected for V1. Adds operational complexity (sidecar, language boundary) before the PDP-as-pattern is internalized. Future: ship an `OpaPolicyEngine` adapter.
+- **Rust-based in-process PDP:** Rejected for V1. Premature.
+
+## References
+
+- [docs/architecture/policy-engine.md](../architecture/policy-engine.md)
+- TRACKER.md Step 1.1c
+- Steps 1.7, 4.5, 6.3, 6.4, 6.5 (consumers / supersedes)