From ba8db8795814faa4b73903b4c902cf482d580914 Mon Sep 17 00:00:00 2001 From: Deep Kumar Singh Kushwah Date: Sun, 24 May 2026 02:20:18 +0530 Subject: [PATCH] =?UTF-8?q?docs(planning):=20insert=20Phase=201=20architec?= =?UTF-8?q?ture-refactor=20window=20(Steps=201.1a=E2=80=931.1f)=20+=20ADRs?= =?UTF-8?q?=200005=E2=80=930009?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Locks in architecture + optimization decisions before Phase 1.2 (connectors) that are very expensive to retrofit. Defers no implementation; updates all affected planning + architecture docs in one coherent pass. ## What changes ### TRACKER.md - Inserts Phase 1 Steps 1.1a–1.1f (refactor window) between 1.1 and 1.2. - Inserts Phase 2 Step 2.11 (agent-loop validation spike). - Milestone counts: Phase 1 10→16, Phase 2 10→11, Total 77→84. - Next action updated to 1.1a. ### Phase 1 plan (planning/phases/phase-1-ingestion.md) Six new detailed step plans with Goal / Inputs / Deliverables / Test plan / Acceptance criteria: - 1.1a — Core type & SPI refactor: RequestContext threaded through every SPI; typed tenant_id + acl_labels on Chunk/Embedding; trust_level for prompt-injection defense; Embedding.dtype (float32/int8/binary); BlobRef for lazy chunk text; QueryPlan + ChunkRef; typed StageEvent. - 1.1b — SPI split: RetrievalBackend / IndexBackend; bulk + streaming + retrieve_ids/hydrate + IndexHint. - 1.1c — PolicyEngine package (rag-policy): single PDP for ACL / PII / quotas / redaction. Supersedes scattered checks. Coverage linter blocks bypasses. Step 6.4 (ACL egress verifier) marked redundant. - 1.1d — Pipeline + Batcher primitives in rag-core. - 1.1e — Three-cache SPI split (EmbeddingCache / RetrievalCache / AnswerCache), hot-path discipline doc, async telemetry with drop-on-overflow. - 1.1f — ADRs 0005–0009 + reviewer checklist. ### Phase 2 plan (planning/phases/phase-2-retrieval.md) - Step 2.11 — Agent-loop validation spike at end of Phase 2 to validate retrieval design against agent access patterns before Phase 3. ### Architecture docs (new) - docs/architecture/request-context.md — the per-request envelope. - docs/architecture/policy-engine.md — central PDP design + usage + extension. - docs/architecture/caching.md — three caches, three invalidation rules. - docs/architecture/performance.md — hot-path discipline, per-SPI p99 budgets, reviewer checklist. ### ADRs (new) - ADR-0005 — PolicyEngine as central PDP. - ADR-0006 — Two-stage reranker as default SPI shape from day 1. - ADR-0007 — Chunk.text: str | BlobRef enables lazy hydration + tiering. - ADR-0008 — Cost-aware planner; fallback triggered planner-side before dispatch, not by post-hoc timeout. - ADR-0009 — Vector index strategy by scale tier + int8/binary quantization; resolves ADR-0004 §2 "ivfflat hardcoded" gap. ### HLD (docs/architecture/RAG-Platform-HLD.md) - v1.0 → v1.1. - §4.2 components table expanded: PolicyEngine added; Knowledge Store split into Retrieval + Index; runtime primitives table added (RequestContext, QueryPlan, Pipeline, Batcher, PolicyEngine, StageEvent, ChunkRef, BlobRef). - §10 governance rewritten around PolicyEngine PDP; trust_level documented as prompt-injection foundation; Step 6.4 (egress verifier) marked redundant. - §11 observability adds StageEvent stream + async telemetry path. ### EXECUTION-PLAN.md - §3 phase summary updated (Phase 1 + refactor window; Phase 2 + spike). - §9 change log: v1.2 entry. ### PROBLEM-TRACEABILITY.md - Cross-tenant, ACL, PII entries route through PolicyEngine PDP. - Latency entry adds 1.1b/1.1d/1.1e + ADRs 0006/0008/0009. - Step 6.4 marked superseded. - New entry: prompt injection → trust_level + LLM adapter isolation strategy. - 22 → 23 entries. ### RISK-REGISTER.md - R2, R4, R5, R11 mitigations hardened with explicit pointers to ADRs 0005, 0006, 0008, 0009 and the PolicyEngine coverage linter. ### CLAUDE.md - Current phase / next step updated to 1.1a + refactor window context. - Repo layout: rag-policy and rag-backends packages added. - Python packages table: rag-policy added; rag-core + rag-observability descriptions updated for Pipeline / Batcher / async exporter. - New sections: RequestContext threading, PolicyEngine, hot-path discipline. - Standing constraints expanded. ### docs/README.md - Indexes the four new architecture docs and five new ADRs. ## Documentation Every affected file is updated in this single PR per the standing per-PR-documentation rule. Cross-references between TRACKER, planning, architecture, and ADRs are bidirectional. ## Test plan No code changes in this PR — planning + docs only. Implementation work begins on a separate branch for Step 1.1a. Co-Authored-By: Claude Opus 4.7 (1M context) --- CLAUDE.md | 55 ++++- TRACKER.md | 19 +- docs/README.md | 9 + docs/adr/ADR-0005-policy-engine.md | 80 ++++++++ docs/adr/ADR-0006-two-stage-rerank.md | 64 ++++++ docs/adr/ADR-0007-tiered-storage.md | 65 ++++++ docs/adr/ADR-0008-cost-aware-planner.md | 56 ++++++ docs/adr/ADR-0009-vector-index-strategy.md | 91 +++++++++ docs/architecture/RAG-Platform-HLD.md | 61 ++++-- docs/architecture/caching.md | 134 +++++++++++++ docs/architecture/performance.md | 111 ++++++++++ docs/architecture/policy-engine.md | 116 +++++++++++ docs/architecture/request-context.md | 114 +++++++++++ planning/EXECUTION-PLAN.md | 5 +- planning/PROBLEM-TRACEABILITY.md | 26 ++- planning/RISK-REGISTER.md | 8 +- planning/phases/phase-1-ingestion.md | 223 +++++++++++++++++++++ planning/phases/phase-2-retrieval.md | 30 +++ 18 files changed, 1217 insertions(+), 50 deletions(-) create mode 100644 docs/adr/ADR-0005-policy-engine.md create mode 100644 docs/adr/ADR-0006-two-stage-rerank.md create mode 100644 docs/adr/ADR-0007-tiered-storage.md create mode 100644 docs/adr/ADR-0008-cost-aware-planner.md create mode 100644 docs/adr/ADR-0009-vector-index-strategy.md create mode 100644 docs/architecture/caching.md create mode 100644 docs/architecture/performance.md create mode 100644 docs/architecture/policy-engine.md create mode 100644 docs/architecture/request-context.md diff --git a/CLAUDE.md b/CLAUDE.md index c6e42d4..e09a8c1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,17 +12,19 @@ so every backend (vector store, embedder, LLM, etc.) is swappable without touchi what is in progress, and what is next. The memory index at `~/.claude/projects/.../memory/MEMORY.md` carries supporting facts between sessions. -Current phase: **Phase 0 — Foundation** (10 of 13 steps complete as of 2026-05-23). -Next step: **0.8 — Eval skeleton** (RAGAS spike, `ragctl eval`, golden-set schema). +Current phase: **Phase 1 — Ingestion + Knowledge Store** (1 of 16 steps complete as of 2026-05-24). +Next step: **1.1a — Core type & SPI refactor** (RequestContext threaded through every SPI; typed ACL on Chunk/Embedding; BlobRef/dtype/trust_level/QueryPlan/ChunkRef/StageEvent). Part of the Phase 1 refactor window (1.1a–1.1f) inserted ahead of Step 1.2 to lock in architecture + optimization decisions that are very expensive to retrofit. See ADRs 0005–0009 and `docs/architecture/{policy-engine,request-context,caching,performance}.md`. ## Repo layout ``` AgentContextOS/ ├── packages/ -│ ├── core/ # rag-core — domain types, errors, SPI interfaces, noop impls, AuditWriter +│ ├── core/ # rag-core — domain types, errors, SPI interfaces, noop impls, AuditWriter, Pipeline, Batcher │ ├── config/ # rag-config — rag.yaml schema (Pydantic v2), loader, ragctl config CLI -│ └── observability/ # rag-observability — structured JSON logger, set_log_context(), event registry +│ ├── observability/ # rag-observability — structured JSON logger, set_log_context(), event registry, async exporter +│ ├── policy/ # rag-policy — PolicyEngine (central PDP for ACL/PII/quotas/redaction); added in Step 1.1c +│ └── backends/ # rag-backends — PgVectorStore, QdrantVectorStore, RedisCache, S3Storage, LocalFileStorage ├── apps/ │ └── gateway/ # rag-gateway (FastAPI, future home of the REST + gRPC service) ├── tests/ @@ -43,9 +45,11 @@ AgentContextOS/ | Package | Import root | Purpose | |---------|-------------|---------| -| `rag-core` v0.2.0 | `rag_core` | Domain types (Pydantic v2 frozen models), 14-type error hierarchy, plugin SPI ABCs + noop impls, `AuditWriter` | +| `rag-core` v0.2.0 | `rag_core` | Domain types (Pydantic v2 frozen models), 14-type error hierarchy, plugin SPI ABCs + noop impls, `AuditWriter`, `Pipeline` + `Batcher` primitives (added in Step 1.1d) | | `rag-config` v0.1.0 | `rag_config` | `rag.yaml` schema, env-var interpolation loader, `ConfigWatcher` hot-reload, `ragctl config` CLI | -| `rag-observability` v0.1.0 | `rag_observability` | 7-field JSON structured logger, `set_log_context()` contextvar manager, event registry | +| `rag-observability` v0.1.0 | `rag_observability` | 7-field JSON structured logger, `set_log_context()` contextvar manager, event registry, async exporter with drop-on-overflow | +| `rag-policy` v0.1.0 (Step 1.1c) | `rag_policy` | `PolicyEngine` SPI + noop impl; central PDP for ACL / PII / quotas / redaction; `PolicyWriter` facade | +| `rag-backends` v0.1.0 | `rag_backends` | Real backend implementations: PgVectorStore, QdrantVectorStore, RedisCache, S3Storage, LocalFileStorage | | `rag-gateway` v0.1.0 | `rag_gateway` | FastAPI gateway (scaffold only — real routes come in Phase 3) | ## Key architecture patterns @@ -88,7 +92,36 @@ writer.write(AuditEvent(tenant_id=..., action="retrieval.query", ...)) ### Domain types All in `rag_core.types`. All models are **frozen Pydantic v2** — create new instances, never mutate. Key types: `TraceContext`, `AuditEvent`, `Document`, `Chunk`, `Embedding`, -`Query`, `RetrievalResult`, `Citation`. +`Query`, `RetrievalResult`, `Citation`, `RequestContext`, `QueryPlan`, `ChunkRef`, +`BlobRef`, `StageEvent`. + +### RequestContext threading (Step 1.1a) +Every SPI method takes `ctx: RequestContext` as its first argument. The gateway +constructs `ctx` once at the boundary; downstream code treats it as trusted and +immutable. See [docs/architecture/request-context.md](docs/architecture/request-context.md). + +```python +async def retrieve_ids( + self, + ctx: RequestContext, # always first + query: Query, + filter_expr: FilterExpr | None = None, +) -> list[ChunkRef]: ... +``` + +A signature linter in `tests/contract/spi_signature.py` fails CI on additions +that omit `ctx`. + +### PolicyEngine (Step 1.1c) +Every retrieval / ingest / egress path consults `PolicyEngine.evaluate()` or +uses `PolicyWriter`. A coverage linter (`tests/policy/coverage.py`) catches +bypasses. See [docs/architecture/policy-engine.md](docs/architecture/policy-engine.md). + +### Hot-path discipline (Step 1.1e) +Pydantic at SPI boundaries only; inside hot loops use dataclasses, msgspec, or +`Model.model_construct()`. Per-SPI p99 budgets enforced by conformance tests. +Full convention + reviewer checklist in +[docs/architecture/performance.md](docs/architecture/performance.md). ## Commands @@ -195,5 +228,9 @@ Add a **Documentation** section listing the doc file(s) added or updated so revi - `mypy --strict` on all `packages/` and `apps/gateway/` source. - No `logging.getLogger()` outside the approved allowlist (RAG001). - All Pydantic domain models use `model_config = {"frozen": True}`. -- No circular imports between packages: `rag-core` has no dependency on `rag-observability` - or `rag-config`. The dependency graph is: gateway → observability → core, config → core. +- No circular imports between packages: `rag-core` has no dependency on `rag-observability`, + `rag-config`, `rag-policy`, or `rag-backends`. The dependency graph is: + gateway → observability → core; config → core; policy → core; backends → core. +- Every SPI method takes `ctx: RequestContext` first (Step 1.1a onward). +- Every retrieval / ingest / egress call site consults `PolicyEngine` (Step 1.1c onward). +- Hot-path discipline: Pydantic at SPI boundaries only; `model_construct()` / msgspec inside loops. diff --git a/TRACKER.md b/TRACKER.md index 702f284..2e1e03b 100644 --- a/TRACKER.md +++ b/TRACKER.md @@ -10,7 +10,9 @@ **Last updated:** 2026-05-24 **Current phase:** Phase 1 — Ingestion + Knowledge Store -**Next action:** Phase 1 Step 1.2 — Connectors framework +**Next action:** Phase 1 Step 1.1a — Core type & SPI refactor (RequestContext, typed ACL, dtype, BlobRef, QueryPlan, ChunkRef, StageEvent) + +> **Refactor window (Steps 1.1a–1.1f):** Before resuming the connectors framework (1.2), we insert a six-step refactor that locks in architecture + optimization decisions which are very expensive to retrofit later (PolicyEngine PDP, RequestContext-threaded SPIs, split Retrieval/Index backends, bulk + streaming + ID-only methods, Pipeline + Batcher primitives, three-way cache split, hot-path discipline). See [docs/adr/ADR-0005…0009] and [docs/architecture/policy-engine.md], [request-context.md], [caching.md], [performance.md]. --- @@ -30,14 +32,14 @@ | Phase | Title | Steps | ✅ Done | Remaining | |-------|-------|------:|-------:|----------:| | 0 | Foundation | 13 | **13** | 0 | -| 1 | Ingestion + Knowledge Store | 10 | **1** | 9 | -| 2 | Retrieval Engine | 10 | 0 | 10 | +| 1 | Ingestion + Knowledge Store | 16 | **1** | 15 | +| 2 | Retrieval Engine | 11 | 0 | 11 | | 3 | Gateway & Agent Runtime | 11 | 0 | 11 | | 4 | Reliability | 6 | 0 | 6 | | 5 | Eval & Observability | 7 | 0 | 7 | | 6 | Governance & Tenancy | 10 | 0 | 10 | | 7 | Pilot, Harden, GA | 10 | 0 | 10 | -| **Total** | | **77** | **14** | **63** | +| **Total** | | **84** | **14** | **70** | --- @@ -66,7 +68,13 @@ | Step | Title | Status | Branch | PR | Key Deliverables | |------|-------|--------|--------|----|-----------------| | 1.1 | Storage backends | ✅ | `build/phase-1/step-1.1-storage-backends` | [#40](https://github.com/officialCodeWork/AgentContextOS/pull/40) | `rag-backends` package: `PgVectorStore` (asyncpg + pgvector, ivfflat), `QdrantVectorStore` (query_points API), `RedisCache`, `S3Storage` (aioboto3, MinIO-compatible), `LocalFileStorage`; integration tests (skip-if-no-service); MinIO added to dev stack; `task test-integration` + `task test-backends`; ADR-0004 | -| 1.2 | Connectors framework | ⏳ | — | — | `Connector` SPI implementation; built-in: filesystem, S3, GCS; crawler base class | +| 1.1a | Core type & SPI refactor | ⏳ | — | — | `RequestContext` frozen model threaded through every SPI; `tenant_id` + `acl_labels` typed required on `Chunk`/`Embedding` (not metadata dict); `trust_level` on `Chunk` for prompt-injection defense; `dtype` on `Embedding` (float32/int8/binary); `BlobRef` for lazy chunk text; `QueryPlan` + `ChunkRef` types; typed `StageEvent`. Conformance tests updated. ADR-0005 (PolicyEngine) + ADR-0008 (cost-aware planner) referenced. | +| 1.1b | SPI split — Retrieval/Index, bulk + streaming + ID-only | ⏳ | 1.1a | — | Split `VectorStore`/`KeywordStore`/`GraphStore` into `RetrievalBackend` (read) + `IndexBackend` (write); add `bulk_embed`/`bulk_index`/`bulk_delete`; add async-iterator ingest variants; add `retrieve_ids` + `hydrate` pair; add `IndexHint` parameter for scale-tier index selection. ADR-0009 (vector index strategy) authored. | +| 1.1c | PolicyEngine package | ⏳ | 1.1a | — | New `packages/policy/` (`rag-policy`): `PolicyEngine` SPI + noop impl; single decision point consulted by every retrieval/ingest path for ACL, PII, quotas, redaction. Replaces scattered checks across Steps 1.7, 4.5, 6.3, 6.5. ADR-0005 finalized. | +| 1.1d | Pipeline + Batcher primitives | ⏳ | 1.1a | — | `Pipeline` primitive in `rag-core`: async DAG with bounded queues, per-stage worker counts, backpressure (used by Step 1.10 write path). `Batcher[Req, Resp]` middleware (DataLoader pattern) coalescing concurrent SPI calls into batched provider calls; sits under Embedder/Reranker SPIs. | +| 1.1e | Cache SPI split + perf discipline + async telemetry | ⏳ | 1.1a | — | Split `Cache` into `EmbeddingCache` (key model_id+text_hash), `RetrievalCache` (plan_hash+corpus_version), `AnswerCache` (plan_hash+corpus_version+policy_version). Each has distinct invalidation. Hot-path convention doc (Pydantic at SPI boundary, `model_construct`/msgspec inside). Async telemetry path with bounded buffer + drop-on-overflow counter. | +| 1.1f | ADRs 0005–0009 + reviewer checklist | ⏳ | 1.1a–1.1e | — | ADR-0005 (PolicyEngine PDP), ADR-0006 (two-stage reranker default), ADR-0007 (tiered storage with BlobRef), ADR-0008 (cost-aware planner replacing reactive fallback), ADR-0009 (vector index strategy + quantization). Reviewer checklist in `docs/architecture/performance.md`. | +| 1.2 | Connectors framework | ⏳ | 1.1a–1.1f | — | `Connector` SPI implementation (now receives `RequestContext` and `ConnectorState` watermark); built-in: filesystem, S3, GCS; crawler base class | | 1.3 | Document parsers | ⏳ | — | — | PDF, DOCX, PPTX, XLSX, HTML, Markdown, plain text, JSON, CSV, YAML parsers; MIME detection | | 1.4 | OCR pipeline | ⏳ | — | — | Tesseract + PaddleOCR plugins; image region extraction; confidence scoring | | 1.5 | Structure-aware chunker | ⏳ | — | — | Heading-based chunking, parent-child `Chunk.parent_id` links, sentence-boundary, overlap, size normalization | @@ -92,6 +100,7 @@ | 2.8 | Context packer | ⏳ | — | — | Dedup, reordering, conflict detection, token budget enforcement; `pack` span | | 2.9 | GraphRAG | ⏳ | — | — | Community detection, sub-graph summarization, graph-aware retrieval | | 2.10 | Retrieval router | ⏳ | — | — | Per-query backend selection; degradation detection; fallback to BM25-only | +| 2.11 | Agent-loop validation spike | ⏳ | 2.1–2.10 | — | Thin `agent_loop_v0` — iterative retrieve with budget enforcement (tokens/cost/wall/iter from `RequestContext.budget`); 50-query agent-style harness validates Phase 2 retrieval design against agent access patterns (shared embedding cache, partial plan reuse, sticky corpus routing) before committing to Phase 3. Output: gap list + any Phase 2 fixes before Phase 3.1 starts. | --- diff --git a/docs/README.md b/docs/README.md index e3aca9b..c004557 100644 --- a/docs/README.md +++ b/docs/README.md @@ -6,6 +6,10 @@ |------|-------------| | [RAG-Platform-HLD.md](architecture/RAG-Platform-HLD.md) | High-Level Design: problem statement, layered architecture, pluggable backends, `rag.yaml` contract, deployment topologies, KPIs, risks, glossary | | [high-level-architecture.svg](architecture/high-level-architecture.svg) | Layered architecture diagram (SVG) | +| [request-context.md](architecture/request-context.md) | `RequestContext` — the per-request envelope threaded through every SPI (tenant, principal, ACLs, PII policy, trace, budget) | +| [policy-engine.md](architecture/policy-engine.md) | `PolicyEngine` (PDP) — single decision point for ACL, PII, quotas, redaction; replaces scattered governance checks | +| [caching.md](architecture/caching.md) | Three-cache split: `EmbeddingCache`, `RetrievalCache`, `AnswerCache` — distinct invalidation rules | +| [performance.md](architecture/performance.md) | Hot-path discipline, per-SPI p99 budgets, async telemetry, reviewer checklist | | [eval-skeleton.md](architecture/eval-skeleton.md) | Eval framework architecture: golden-set schema, metric functions, RAGAS spike, `ragctl eval` CLI, extension points | | [iac.md](architecture/iac.md) | IaC overview: Terraform module design, Helm chart structure, dev/prod environments, extension points | | [storage-backends.md](architecture/storage-backends.md) | Storage backend architecture: PgVector, Qdrant, Redis, S3/MinIO, tenant isolation, integration test strategy | @@ -32,6 +36,11 @@ | [ADR-0002-eval-framework.md](adr/ADR-0002-eval-framework.md) | Decision: pure-Python Tier 1 metrics always-on; RAGAS as optional Tier 2 | | [ADR-0003-iac-kubernetes-native.md](adr/ADR-0003-iac-kubernetes-native.md) | Decision: Kubernetes-native Terraform modules over cloud-provider-specific RDS/ElastiCache | | [ADR-0004-storage-backends.md](adr/ADR-0004-storage-backends.md) | Decision: single `rag-backends` package, MinIO for S3-compatible dev storage, graceful integration test skip | +| [ADR-0005-policy-engine.md](adr/ADR-0005-policy-engine.md) | Decision: `rag-policy` package as central PDP for ACL/PII/quotas/redaction; supersedes Step 6.4 egress verifier | +| [ADR-0006-two-stage-rerank.md](adr/ADR-0006-two-stage-rerank.md) | Decision: `Reranker` SPI ships as two-stage cascade (fast bi-encoder → cross-encoder) from day 1 | +| [ADR-0007-tiered-storage.md](adr/ADR-0007-tiered-storage.md) | Decision: `Chunk.text: str \| BlobRef` enables lazy hydration and hot/warm/cold tiering | +| [ADR-0008-cost-aware-planner.md](adr/ADR-0008-cost-aware-planner.md) | Decision: `QueryPlan` carries `estimated_cost`; fallback triggered planner-side before dispatch, not by post-hoc timeout | +| [ADR-0009-vector-index-strategy.md](adr/ADR-0009-vector-index-strategy.md) | Decision: `IndexHint` selects index by scale tier (flat / ivfflat / HNSW / IVF-PQ / DiskANN); `Embedding.dtype` enables int8 / binary quantization | ## research/ diff --git a/docs/adr/ADR-0005-policy-engine.md b/docs/adr/ADR-0005-policy-engine.md new file mode 100644 index 0000000..3364b37 --- /dev/null +++ b/docs/adr/ADR-0005-policy-engine.md @@ -0,0 +1,80 @@ +# ADR-0005 — PolicyEngine as central PDP + +**Status:** Accepted +**Date:** 2026-05-24 +**Deciders:** Core team +**Step:** Phase 1 Step 1.1c + +--- + +## Context + +Governance in the V1.0 plan was scattered across five steps: + +- 1.7 — PII detection at ingest +- 4.5 — Quotas + rate limits +- 6.3 — ACL push-down at retrieval +- 6.4 — ACL egress verifier (defense-in-depth) +- 6.5 — PII egress scanner + +Each is a separate code path with its own call sites. This is exactly how real-world governance leaks happen: a new feature adds a code path that forgets one of the checks. Step 6.4 (egress verifier) is a tacit admission that 6.3 alone is too fragile — the system needs *two* layers because the *first* one might be bypassed. + +The pattern is well-understood: a Policy Decision Point (PDP) consolidates these into one auditable surface. OPA / Cedar / AuthZed are existing implementations of this pattern in adjacent domains. + +## Decision + +### 1. Introduce a `rag-policy` workspace package + +New package `packages/policy/` (import root `rag_policy`). Sits between `rag-core` and consumers; depends only on `rag-core` types (`RequestContext`). + +### 2. `PolicyEngine` is the single SPI for all governance decisions + +```python +class PolicyEngine(ABC): + async def evaluate( + self, + ctx: RequestContext, + decision: PolicyDecision, # READ_CHUNK | INGEST_DOC | EGRESS_TEXT | QUOTA_CHECK | ... + subject: Any, + ) -> PolicyResult # ALLOW | DENY(reason) | TRANSFORM(redacted_subject) + + async def filter_pushdown( + self, + ctx: RequestContext, + decision: PolicyDecision, + ) -> FilterExpr # Injected into retrieval calls +``` + +Every retrieval, ingest, and egress call site **must** consult `PolicyEngine`. Enforced by a coverage linter (`tests/policy/coverage.py`) that fails CI on un-policed calls. + +### 3. `PolicyWriter` facade pattern (mirrors `AuditWriter`) + +Composes `PolicyEngine` evaluation with structured-log emission and audit-event write. Single import for consumers. + +### 4. Step 6.4 (ACL egress verifier) is marked redundant + +Because no retrieval call site bypasses the PDP, defense-in-depth is provided by the PDP design itself, not by a second checking layer. The TRACKER entry and planning file for Step 6.4 will be updated to "Superseded by ADR-0005" in the Phase 6 refactor pass. + +### 5. NoopPolicyEngine reference impl + +Always-ALLOW noop for OSS / dev. Production tenants override with their own implementation (or future built-ins). + +## Consequences + +- **Positive:** Governance decisions auditable from one surface. Adding a new governance rule (e.g., GDPR data-residency check) is one PolicyEngine extension, not five touchpoints. ACL pushdown + egress check unified. Coverage testable. +- **Positive:** Pattern enables future plug-ins to OPA / Cedar without re-architecting consumer code. +- **Negative:** Every existing retrieval / ingest test fixture must now construct a `RequestContext` and pass through PolicyEngine. One-time refactor cost. +- **Negative:** Performance: a synchronous `evaluate` adds ~tens of microseconds per call. Mitigated by (a) noop impl is essentially free, (b) batch evaluation API, (c) plan-time evaluation cached per `RequestContext` lifetime. +- **Neutral:** Step 6.4 removed from the critical path; team-week budget reallocated. + +## Alternatives considered + +- **Status quo (scattered checks):** Rejected. Predictable failure mode (one missed call site = leak). +- **Embed a third-party PDP (OPA sidecar) from day 1:** Rejected for V1. Adds operational complexity (sidecar, language boundary) before the PDP-as-pattern is internalized. Future: ship an `OpaPolicyEngine` adapter. +- **Rust-based in-process PDP:** Rejected for V1. Premature. + +## References + +- [docs/architecture/policy-engine.md](../architecture/policy-engine.md) +- TRACKER.md Step 1.1c +- Steps 1.7, 4.5, 6.3, 6.4, 6.5 (consumers / supersedes) diff --git a/docs/adr/ADR-0006-two-stage-rerank.md b/docs/adr/ADR-0006-two-stage-rerank.md new file mode 100644 index 0000000..8251f67 --- /dev/null +++ b/docs/adr/ADR-0006-two-stage-rerank.md @@ -0,0 +1,64 @@ +# ADR-0006 — Two-stage reranker as default SPI shape + +**Status:** Accepted +**Date:** 2026-05-24 +**Deciders:** Core team, ML / Retrieval lead +**Step:** Phase 1 Step 1.1f (decision); Phase 2 Step 2.7 (implementation) + +--- + +## Context + +A single-stage cross-encoder reranker over ~100 hybrid-retrieval candidates costs ~10 ms per pair on CPU, so ~1 s per query. That blows the Phase 2 p95 budget (≤ 500 ms dev, ≤ 250 ms SaaS) and the Phase 4 latency tuning gate (gateway p99 ≤ 30 ms excluding retrieval). + +The risk register's R2 ("Reranker latency overshadows quality gains") already notes the mitigation: two-stage rerank with early-exit. But the V1.0 plan treats this as a Step-4.6 optimization layered on top of a single-stage Step-2.7 reranker. The pattern of "ship single-stage, then bolt on two-stage during latency tuning" is the slow path — it forces an SPI refactor mid-build. + +## Decision + +The `Reranker` SPI ships as a two-stage cascade *from day 1* in Step 2.7. Single-stage rerankers are implemented as `precise_rerank` only, with `fast_rerank` defaulting to a pass-through bi-encoder scoring. + +```python +class Reranker(ABC): + async def fast_rerank( + self, + ctx: RequestContext, + query: Query, + candidates: list[ChunkRef], # ~200 ChunkRefs from hybrid retrieval + top_n: int = 50, + ) -> list[ChunkRef] # Stage 1: cheap, p99 < 5 ms + + async def precise_rerank( + self, + ctx: RequestContext, + query: Query, + candidates: list[Chunk], # hydrated top-50 + top_n: int = 10, + ) -> list[Chunk] # Stage 2: cross-encoder, p99 < 100 ms + + async def should_early_exit( + self, + stage_1_scores: list[float], + ) -> bool # Skip stage 2 if scores tightly clustered above threshold +``` + +Default implementations: +- `LocalBgeReranker` — bi-encoder for stage 1, `bge-reranker-v2-m3` for stage 2. +- `CohereReranker` — Cohere's API, called only at stage 2 (stage 1 still local). + +## Consequences + +- **Positive:** Single SPI shape covers cheap, expensive, hybrid local-then-hosted, and single-stage variants without an SPI break later. Latency budget achievable from the first retrieval-engine ship. +- **Positive:** Composes with `retrieve_ids` + `hydrate` (Step 1.1b): stage 1 operates on `ChunkRef` (no full text), stage 2 hydrates only the top-50. +- **Negative:** Reranker SPI surface is wider than a single-method `rerank(query, candidates)`. Implementers must understand the two-stage contract. +- **Neutral:** Conformance tests must cover both stages and the early-exit predicate. + +## Alternatives considered + +- **Single-stage SPI + late refactor:** Rejected. The Risk Register already names this exact failure mode (R2). Refactoring an SPI after backends ship is the costly path. +- **Per-tenant policy selection between single and two-stage:** Deferred to V2. The two-stage default already supports single-stage usage by trivial `fast_rerank`. + +## References + +- [planning/RISK-REGISTER.md](../../planning/RISK-REGISTER.md) R2 +- TRACKER.md Step 2.7 +- ADR-0009 (vector index strategy — companion scale decision) diff --git a/docs/adr/ADR-0007-tiered-storage.md b/docs/adr/ADR-0007-tiered-storage.md new file mode 100644 index 0000000..abaab9f --- /dev/null +++ b/docs/adr/ADR-0007-tiered-storage.md @@ -0,0 +1,65 @@ +# ADR-0007 — Tiered storage with `BlobRef` + +**Status:** Accepted +**Date:** 2026-05-24 +**Deciders:** Core team, SRE +**Step:** Phase 1 Step 1.1a (`BlobRef` introduction); Phase 6 (incremental adoption) + +--- + +## Context + +At million-document scale, holding every `Chunk.text` in memory through the ingest and retrieval paths is infeasible: + +- A 1000-page PDF parses into ~5 MB of chunk text. 1000 such docs in flight = 5 GB. +- Many chunks are read rarely (long-tail); paying RAM cost for them is wasteful. +- Cold storage on S3 / GCS is 20× cheaper than Redis or pgvector row storage. + +Two distinct needs: +1. **Memory bounding during ingest** — large-doc workflows must not OOM. +2. **Tiered cost** — hot chunks in fast store, warm in primary store, cold in object storage. + +Both need the same addressing primitive: a reference to chunk text that can be resolved lazily. + +## Decision + +### 1. `Chunk.text: str | BlobRef` + +`BlobRef = (storage_provider, key, byte_range)`. A chunk may carry its text inline (small chunks, hot path) or as a reference (large chunks, cold path). Resolved on-demand by the context packer (Step 2.8). + +### 2. Resolution uses the existing `Storage` SPI + +No new SPI. `BlobRef` carries a `storage_provider` discriminator (`"s3"`, `"local_fs"`, etc.); resolution dispatches to the configured `Storage` implementation. Resolved bytes cached within `RequestContext` lifetime. + +### 3. Three tiers (operational, not type-level) + +| Tier | Backend | Policy | +|---|---|---| +| Hot | Redis | Recently accessed chunks (LRU); resolved `BlobRef` cache | +| Warm | pgvector / Qdrant payload | Default storage for chunks ≤ 2 KB text | +| Cold | S3 / GCS via `BlobRef` | Chunks > 2 KB or accessed < N times in 30d; archived chunks | + +Tier promotion / demotion is a Phase-6 background job; the SPI is invariant. + +### 4. Conformance: every store handles `BlobRef`-backed chunks identically + +Round-trip test: packing a `BlobRef`-backed `Chunk` produces the same output as packing the inline equivalent. + +## Consequences + +- **Positive:** Memory-bounded ingest. Storage cost drops materially at scale. Air-gapped deployments still work (local_fs as cold tier). +- **Positive:** No SPI break later when tiering is added — `BlobRef` is in the type from day 1. +- **Negative:** Two text-resolution paths to test. Resolution latency on cold tier (S3 GET) can be ~10–50 ms — mitigated by `BlobRef` cache and lazy hydration only for chunks that survive rerank. +- **Negative:** Object storage adds eventual-consistency considerations (GET after PUT) — acceptable because chunks are immutable post-ingest. + +## Alternatives considered + +- **Inline text only + add tiering later via Storage SPI:** Rejected. Adding `Chunk.text: str | BlobRef` post-ship is a breaking change across every backend and every consumer. +- **Separate `ChunkBody` type holding either text or ref:** Rejected. Adds indirection at every consumer for no semantic benefit. +- **Always store chunk text in object storage, never inline:** Rejected. Pays cold-tier latency for hot chunks; bad default. + +## References + +- TRACKER.md Step 1.1a +- [docs/architecture/storage-backends.md](../architecture/storage-backends.md) +- ADR-0004 (storage backends) diff --git a/docs/adr/ADR-0008-cost-aware-planner.md b/docs/adr/ADR-0008-cost-aware-planner.md new file mode 100644 index 0000000..b25b7a9 --- /dev/null +++ b/docs/adr/ADR-0008-cost-aware-planner.md @@ -0,0 +1,56 @@ +# ADR-0008 — Cost-aware planner replacing reactive fallback + +**Status:** Accepted +**Date:** 2026-05-24 +**Deciders:** Core team, ML / Retrieval lead +**Step:** Phase 1 Step 1.1a (`QueryPlan` type); Phase 4 Step 4.2 (fallback) refactored + +--- + +## Context + +The V1.0 plan for Step 4.2 ("Fallback chain") describes a reactive pattern: hybrid → BM25-only → keyword → "no answer", triggered by errors or timeouts after the fact. This works but has three drawbacks: + +1. **Latency:** an expensive call must time out (often hundreds of ms) before fallback fires. The user pays the cost of the failed attempt. +2. **Cost:** an expensive embedding / LLM call may be billed even when the result is discarded. +3. **Observability:** "why did fallback fire?" requires correlating the failed call's timeout with the fallback trigger — non-trivial. + +A planner that knows the *cost envelope* of each plan node can decide *before* dispatch whether to take the expensive path. + +## Decision + +### 1. `QueryPlan` is a first-class type (introduced in Step 1.1a) + +Output of query understanding (Step 2.6), input to retrieval (Steps 2.1–2.5). Each plan node carries an `estimated_cost: Cost(ms, tokens, dollars)`. + +### 2. Planner-side budget check before dispatch + +Before executing a plan, the planner sums `estimated_cost` across nodes and compares to `RequestContext.budget`. If exceeded, it **mutates the plan** (drops graph hop, skips precise rerank, narrows top-K, falls back to BM25-only) rather than executing-then-failing. + +### 3. Step 4.2 "Fallback chain" is refactored + +It remains a Phase 4 step but is now about (a) defining the budget-overrun mutation rules and (b) handling unexpected runtime failures (backend down, network partition). Reactive fallback is the exceptional path, not the primary one. + +### 4. Cost estimation is best-effort + learned + +Initial estimates come from rolling-window observed latency / token / cost histograms per backend per `IndexHint`. Estimates update online as `StageEvent`s flow in. + +## Consequences + +- **Positive:** Latency on budget-constrained queries is bounded before dispatch — no expensive timeout. Costs are predictable per tenant. +- **Positive:** Composes with the agent loop (Step 3.6) which mutates plans across turns under a shared session budget. +- **Positive:** `/v1/query/explain` becomes trivially supportable — the plan and its cost estimate are first-class. +- **Negative:** Cost estimation accuracy matters. Bad estimates cause unnecessary fallback (false-positive degraded mode). Mitigated by online learning + bounded confidence intervals. +- **Negative:** Conceptually heavier than "try then fall back." Engineers need to understand plan-mutation rules. + +## Alternatives considered + +- **Pure reactive fallback (V1.0 plan):** Rejected. Wastes cost and latency on failed attempts. +- **Static budgets per tenant tier:** Insufficient — same tenant has different budgets per query (background re-eval vs. interactive). +- **External planner service:** Rejected for V1. The planner is in-process, called from the gateway. + +## References + +- TRACKER.md Steps 1.1a, 4.2 +- ADR-0005 (PolicyEngine — runs at planner-time too) +- [docs/architecture/performance.md](../architecture/performance.md) diff --git a/docs/adr/ADR-0009-vector-index-strategy.md b/docs/adr/ADR-0009-vector-index-strategy.md new file mode 100644 index 0000000..dfbca36 --- /dev/null +++ b/docs/adr/ADR-0009-vector-index-strategy.md @@ -0,0 +1,91 @@ +# ADR-0009 — Vector index strategy by scale tier + quantization + +**Status:** Accepted +**Date:** 2026-05-24 +**Deciders:** Core team, ML / Retrieval lead +**Step:** Phase 1 Step 1.1b (SPI); Phase 2 Step 2.2 (backend implementation) + +--- + +## Context + +ADR-0004 §2 hardcodes `PgVectorStore` to `ivfflat` cosine index. This is correct for the Phase 1 milestone (small corpora, smoke tests) but does not scale: + +| Corpus size | Suitable index | Notes | +|---|---|---| +| < 100K vectors | flat / ivfflat | Brute-force is fine | +| 100K – 10M | HNSW | Standard choice; high memory | +| 10M – 100M | IVF-PQ | Quantized, lower memory, slight recall loss | +| > 100M | DiskANN / sharded HNSW | Disk-resident | + +ivfflat performance degrades materially above ~1M vectors. A second axis is **embedding quantization**: Cohere v3 supports int8 and binary embeddings, cutting memory 4× and 32× respectively with < 2% recall loss for most workloads. + +The Phase-1 backends were written before `RequestContext`, `IndexHint`, and `Embedding.dtype` existed. Step 1.1b adds them; this ADR defines the strategy backends apply. + +## Decision + +### 1. `IndexHint` parameter on the `IndexBackend` SPI + +```python +class IndexHint(BaseModel): + estimated_size: int # current + projected vector count + recall_target: float # 0..1, default 0.95 + latency_target_ms: float # p99 query latency target + write_volume: WriteVolume # LOW | MEDIUM | HIGH (affects index rebuild cost tolerance) +``` + +Backends choose the index implementation. Default mapping (PgVector): + +| size | recall_target | choice | +|---|---|---| +| < 100K | any | flat | +| < 1M | ≥ 0.99 | ivfflat | +| < 10M | ≥ 0.95 | HNSW | +| ≥ 10M | ≥ 0.90 | IVF-PQ (with quantization) | + +### 2. `Embedding.dtype` (introduced in Step 1.1a) + +`dtype: Literal["float32", "int8", "binary"]`. Backends accept all three; storage and query paths use the native dtype. + +### 3. ANN tuning utility (Phase 2 Step 2.2) + +`ragctl ann tune --corpus --target-recall 0.95` measures recall-vs-latency curves against a brute-force baseline and writes recommended index parameters into `rag.yaml`. + +### 4. Quantization is opt-in per corpus + +`rag.yaml`: + +```yaml +corpora: + legal: + embedder: + provider: cohere + model: embed-v3 + dtype: int8 # 4× memory cut; ~1% recall loss +``` + +Default remains `float32` for predictable behavior. Quantization advertised in docs as a memory / cost lever for large corpora. + +### 5. ADR-0004 §2 amended + +The "ivfflat hardcoded" gap noted in ADR-0004 is resolved by this ADR. `PgVectorStore` now picks index type at `initialize()` time based on `IndexHint`. + +## Consequences + +- **Positive:** Backends scale from laptop (flat) to production (HNSW / IVF-PQ) without consumer code change. +- **Positive:** Quantization unlocks 4–32× memory savings for large-corpus tenants at their option. +- **Positive:** Removes the foot-gun of "Phase 1 demo worked, Phase 7 load test failed at 5M vectors." +- **Negative:** Index switching at backend boot adds first-time-setup complexity. Mitigated by: defaults are explicit per size bucket; misconfigurations surface at `initialize()`, not at first query. +- **Negative:** Conformance suite must cover all three dtypes per backend. Test-matrix expansion. + +## Alternatives considered + +- **Keep ivfflat hardcoded; advise external sharding:** Rejected. Pushes scale problem to operators. +- **Per-backend explicit config (no hint abstraction):** Rejected. Forces tenants to understand backend internals. +- **Ship int8/binary as a future ADR:** Rejected. Adding `dtype` later is a breaking change to `Embedding`. + +## References + +- ADR-0004 (storage backends — superseded §2) +- TRACKER.md Steps 1.1a, 1.1b, 2.2 +- [docs/architecture/performance.md](../architecture/performance.md) diff --git a/docs/architecture/RAG-Platform-HLD.md b/docs/architecture/RAG-Platform-HLD.md index f9a7110..c6ad731 100644 --- a/docs/architecture/RAG-Platform-HLD.md +++ b/docs/architecture/RAG-Platform-HLD.md @@ -1,10 +1,12 @@ # Enterprise RAG Platform — High-Level Design (HLD) -**Document version:** 1.0 -**Date:** 22 May 2026 +**Document version:** 1.1 +**Date:** 24 May 2026 (revision) **Status:** Draft for review **Owner:** Platform Engineering +**v1.1 changes:** Promoted `RequestContext` to a first-class architectural primitive threaded through every SPI; added a central `PolicyEngine` (PDP) component that subsumes ACL push-down + ACL egress verifier + PII gates + quota gates; split the "Knowledge Store" SPI into separate Retrieval and Index surfaces with bulk + streaming + ID-only methods; added `Pipeline` + `Batcher` runtime primitives; split the semantic cache into three caches with distinct invalidation rules (`EmbeddingCache`, `RetrievalCache`, `AnswerCache`); typed `trust_level` on chunks as the foundation for prompt-injection defense; introduced `BlobRef` for lazy chunk text. See ADRs 0005–0009 and the Phase 1 refactor window (Steps 1.1a–1.1f) in [TRACKER.md](../../TRACKER.md). + --- ## 1. Executive Summary @@ -102,18 +104,31 @@ This platform addresses all of these as first-class capabilities. | Component | Responsibility | |---|---| -| API Gateway | AuthN/Z, routing, rate limit, OpenAPI + gRPC + MCP surface | -| Ingestion Service | Connectors, parsing, structure-aware chunking, enrichment | -| Embedder Service | Pluggable embedding providers (local & hosted) | -| Knowledge Store | Pluggable vector / BM25 / graph / relational backends | -| Retrieval Engine | Hybrid retrieve → rerank → pack | -| Query Understanding | Rewrite, decompose, HyDE, synonym expansion | -| Agent Runtime | Iterative retrieval loops, corpus routing | -| Reliability Layer | Semantic cache, fallback chain, hallucination guard | -| Eval Service | Offline + online metrics, drift detection, feedback ingestion | -| Governance | ACL enforcement, PII pipeline, audit log, tenancy isolation | +| API Gateway | AuthN/Z, routing, rate limit, OpenAPI + gRPC + MCP surface; constructs `RequestContext` (tenant, principal, ACLs, PII policy, trace, budget) once at the boundary | +| Ingestion Service | Connectors, parsing, structure-aware chunking, enrichment; runs as a `Pipeline` DAG with bounded queues and per-stage parallelism | +| Embedder Service | Pluggable embedding providers (local & hosted); concurrent calls auto-coalesced by `Batcher` middleware | +| Knowledge Store | Pluggable backends, split into **`RetrievalBackend`** (read-side: vector / BM25 / graph) and **`IndexBackend`** (write-side: bulk + streaming + soft-delete); `IndexHint` selects index strategy by scale tier | +| Retrieval Engine | Hybrid retrieve → rerank (two-stage by default) → pack; uses `retrieve_ids` + `hydrate` to avoid over-fetching full chunks; produces / consumes a typed `QueryPlan` | +| Query Understanding | Rewrite, decompose, HyDE, synonym expansion; output is a `QueryPlan` with `estimated_cost(ms, tokens, $)` consumed by the cost-aware planner | +| Agent Runtime | Iterative retrieval loops, corpus routing; budget enforcement (tokens / cost / wall / iter) read from `RequestContext.budget`; reuses `QueryPlan` across turns | +| Reliability Layer | Three caches (`EmbeddingCache`, `RetrievalCache`, `AnswerCache` — distinct invalidation rules), fallback chain (planner-triggered, not post-hoc timeout), hallucination guard, circuit breakers | +| Eval Service | Offline + online metrics, drift detection, feedback ingestion; consumes typed `StageEvent` stream emitted by every SPI call | +| **PolicyEngine (PDP)** | **Single decision point for ACL, PII, quotas, redaction. Every retrieval / ingest / egress path calls `evaluate(ctx, decision, subject) → ALLOW \| DENY \| TRANSFORM`. Replaces scattered governance checks; makes ACL egress verifier (formerly Step 6.4) redundant by construction.** | +| Governance | PII pipeline (Presidio plugins), immutable hash-chained audit log, tenancy isolation (logical + physical tier) — all invoked **through PolicyEngine**, not directly | | Control Plane | Tenant mgmt, billing, license, config | +#### 4.2.1 Cross-cutting runtime primitives (introduced in Step 1.1a–1.1d) + +| Primitive | Purpose | Lives in | +|---|---|---| +| `RequestContext` | Frozen per-request envelope: tenant, principal, ACLs, PII policy, trace, budget, feature flags. Threaded through every SPI call. | `rag_core.types` | +| `QueryPlan` | First-class plan object: subqueries, chosen backends + weights, filter pushdowns, rerank policy, packer policy, `estimated_cost`. Enables explain, plan caching, A/B, agent-loop reuse. | `rag_core.types` | +| `Pipeline` | Async DAG with bounded queues, per-stage workers, backpressure. Substrate for ingestion (1.10) and connectors. | `rag_core.pipeline` | +| `Batcher[Req, Resp]` | DataLoader-pattern coalescer sitting under Embedder / Reranker SPIs; turns N concurrent calls into 1 batched provider call. | `rag_core.batcher` | +| `PolicyEngine` | PDP for all governance decisions (see above). | `rag_policy` | +| `StageEvent` | Typed event emitted alongside every SPI result, captured by pluggable `EvalRecorder` sink. Substrate for online eval / drift / shadow mode / hallucination guard. | `rag_core.events` | +| `ChunkRef` / `BlobRef` | Lightweight chunk handle (ID + score) + lazy text reference; enables ID-only retrieval and memory-bounded ingest. | `rag_core.types` | + --- ## 5. Core Capabilities (Problem → Capability Mapping) @@ -242,12 +257,14 @@ retriever = RagPlatformRetriever(endpoint="https://rag.mycorp.com", api_key="... ## 10. Security & Governance -- **Tenancy:** logical (namespace + row-level) at SMB tier; physical (dedicated index/keys) at Enterprise tier. -- **ACL-aware retrieval:** chunks tagged with principals; filters pushed to vector DB; egress double-check. -- **PII:** Presidio / Azure AI Language at ingest; policy-driven redact/mask/encrypt; egress scanner. -- **Prompt-injection defense:** retrieved doc sanitization, system prompt hardening, content firewall. -- **Audit:** immutable log (user, query, retrieved IDs, scores, ACL decisions, model, answer, citations). -- **AuthN:** OIDC/SAML SSO, SCIM. **AuthZ:** RBAC with custom roles. +All security and governance decisions are routed through a single **PolicyEngine** (PDP) component — see ADR-0005 and [docs/architecture/policy-engine.md](policy-engine.md). Scattered checks across retrieval, ingest, and egress are replaced by `PolicyEngine.evaluate(ctx, decision, subject)`. + +- **Tenancy:** logical (namespace + row-level via typed `Chunk.tenant_id`) at SMB tier; physical (dedicated index/keys, per-tenant connection pools) at Enterprise tier. `tenant_id` is a typed required field on `Chunk` and `Embedding`, indexed at every backend — no JSON-filter cost. +- **ACL-aware retrieval:** chunks tagged with typed `acl_labels: tuple[str, ...]`; PolicyEngine returns a `FilterExpr` pushed into every retrieval call. Egress verification is *unnecessary* because no path bypasses the PDP — what was Step 6.4 (egress verifier) is therefore redundant. +- **Prompt-injection defense:** every `Chunk` carries a typed `trust_level: Literal["system", "tenant_curated", "tenant_user", "external"]`. The context packer + LLM adapter use this to choose isolation strategy (XML-tagged section, separate turn, refusal). Not a runtime ad-hoc check. +- **PII:** Presidio / Azure AI Language at ingest; egress redaction policy enforced through PolicyEngine (`TRANSFORM` decisions); per-tenant `block / redact / mask / encrypt / tag-only`. +- **Audit:** immutable hash-chained log (`AuditWriter` facade over `AuditStore`); records `RequestContext`, query, `QueryPlan`, retrieved IDs, scores, policy decisions, model, answer, citations. +- **AuthN:** OIDC/SAML SSO, SCIM. **AuthZ:** RBAC with custom roles, evaluated by PolicyEngine. - **Encryption:** TLS 1.3 in transit; AES-256 at rest; BYOK via Key Vault/KMS. - **Compliance posture:** SOC2 Type 2, ISO 27001, HIPAA, GDPR readiness. @@ -255,11 +272,13 @@ retriever = RagPlatformRetriever(endpoint="https://rag.mycorp.com", api_key="... ## 11. Observability & Evaluation -- **Per-query trace:** query → router → retrievers → reranker → packer → LLM → guard, with stage-level latency, tokens, cost, quality. -- **Offline eval harness:** RAGAS metrics + custom domain evals, gating CI on regressions. -- **Online metrics:** thumbs-up/down, edit-distance, hallucination guard blocks. +- **Per-query trace:** query → policy → router → retrievers → reranker → packer → LLM → guard, with stage-level latency, tokens, cost, quality. +- **Typed `StageEvent` stream:** every SPI call optionally returns a typed event (`stage_name`, `inputs_hash`, `outputs_hash`, `latency_ms`, `cost`) captured by a pluggable `EvalRecorder` sink (noop in OSS, ClickHouse / BigQuery in Enterprise). This is the substrate for online eval, drift detection, shadow mode, and hallucination guard — they don't each reinvent instrumentation. +- **Offline eval harness:** Tier-1 pure-Python metrics + optional RAGAS (per ADR-0002), gating CI on regressions. +- **Online metrics:** thumbs-up/down, edit-distance, hallucination guard blocks, all keyed on `RequestContext.request_id` for round-trip with the trace. - **Drift monitors:** embedding distribution, query topic, retrieval score distribution → alerts. - **Per-tenant dashboards:** quality, latency, cost, usage. +- **Async telemetry path:** bounded buffer + non-blocking OTel exporter + drop-on-overflow counter. The request path never blocks on exporter back-pressure. --- diff --git a/docs/architecture/caching.md b/docs/architecture/caching.md new file mode 100644 index 0000000..daa0a9b --- /dev/null +++ b/docs/architecture/caching.md @@ -0,0 +1,134 @@ +# Caching — three caches, three invalidation rules + +## Overview + +AgentContextOS ships **three distinct cache SPIs**, each with a different cache key and a different invalidation contract. They commonly share a backend (Redis) but the SPIs themselves are separate because conflating them produces either stale answers or low hit rates. + +| Cache | Key | Invalidation trigger | +|---|---|---| +| `EmbeddingCache` | `(model_id, model_version, text_hash)` | Embedder model or version change | +| `RetrievalCache` | `(plan_hash, corpus_version)` | Document add / update / delete in queried corpus | +| `AnswerCache` | `(plan_hash, corpus_version, policy_version)` | Either of the above + any policy / PII rule change | + +A single "semantic cache" that ignores these distinctions produces stale results (when corpus changes invalidate at the wrong granularity) or a < 5% hit rate (when it invalidates too aggressively). + +## Why it exists + +The V1.0 plan described a single "L0/L1/L2 semantic cache" in Step 4.1. Three observations forced the split: + +1. **Embedding cache reuse across re-indexes.** Re-chunking a corpus (e.g., new chunker version) should keep the embedding cache hot — text hasn't changed. A single cache that keys on the query plan would miss. +2. **Retrieval cache invalidates on corpus mutation.** Adding one document to a 10M-doc corpus shouldn't dump the entire answer cache. A single cache that keyed on policy would. +3. **Answer cache invalidates on policy changes.** A tenant rotating their PII policy must invalidate cached answers (which may contain text affected by the new policy) but should preserve retrieval and embedding caches. + +## Usage + +### EmbeddingCache + +```python +from rag_core.spi import EmbeddingCache + +cache: EmbeddingCache = ... + +cached = await cache.get(model_id="bge-large-en-v1.5", model_version="1.5.0", text_hash=h) +if cached is None: + emb = await embedder.embed(ctx, text) + await cache.put(model_id="bge-large-en-v1.5", model_version="1.5.0", text_hash=h, value=emb) +``` + +Hit rate target: ≥ 99% on re-ingest of identical corpora. + +### RetrievalCache + +```python +from rag_core.spi import RetrievalCache + +cache: RetrievalCache = ... + +cached = await cache.get(plan_hash=plan.hash(), corpus_version=corpus.version) +if cached is None: + refs = await retrieve_pipeline(ctx, plan) + await cache.put(plan_hash=plan.hash(), corpus_version=corpus.version, value=refs) +``` + +Hit rate target: ≥ 30% on production query mix (Pareto-distributed). + +### AnswerCache + +```python +from rag_core.spi import AnswerCache + +cache: AnswerCache = ... + +cached = await cache.get( + plan_hash=plan.hash(), + corpus_version=corpus.version, + policy_version=ctx.principal.policy_version, +) +if cached is None: + answer = await llm_pipeline(ctx, plan) + await cache.put(..., value=answer) +``` + +Hit rate target: ≥ 15% on production query mix. + +## Internals + +### Single Redis backend, three keyspaces + +The default implementation puts all three in one Redis instance under prefixes `rag:emb:`, `rag:ret:`, `rag:ans:`. Memory budgets are tunable per cache. Eviction is LRU within each prefix. + +### `corpus_version` + +A monotonically incrementing integer per corpus, bumped by the write path (`IndexBackend.bulk_index`) on any successful upsert / delete. Read by the retrieval pipeline as part of the cache key. + +### `policy_version` + +A hash of the active `PolicyEngine` configuration for the tenant. Bumped by the control plane on any policy change. Stale answers are invalidated transparently on the next lookup. + +### Cross-tenant safety + +Every cache key is implicitly tenant-scoped via `ctx.tenant_id` baked into `plan_hash`. A red-team probe in `tests/redteam/cross_tenant_cache.py` asserts no key from tenant A can resolve to tenant B's value. + +### Sizing + +- Embedding cache: largest. ~1 KB per entry × millions of entries. +- Retrieval cache: medium. ~10 KB per entry (top-K ChunkRefs). +- Answer cache: smallest. ~5 KB per entry (text + citations). + +Default sizing in `rag.yaml`: + +```yaml +cache: + embedding: + provider: redis + max_memory_mb: 4096 + ttl_seconds: 0 # never expire (invalidated on model change) + retrieval: + provider: redis + max_memory_mb: 1024 + ttl_seconds: 3600 # belt-and-braces + answer: + provider: redis + max_memory_mb: 512 + ttl_seconds: 900 +``` + +## Extension points + +Implement any of: + +```python +class EmbeddingCache(ABC): + async def get(self, *, model_id: str, model_version: str, text_hash: str) -> Embedding | None: ... + async def put(self, *, model_id: str, model_version: str, text_hash: str, value: Embedding) -> None: ... +``` + +(Similar shapes for `RetrievalCache`, `AnswerCache`.) + +Plug in alternative backends (Valkey, Memcached, in-memory LRU) without changing consumers. + +## Related + +- [ADR-0008](../adr/ADR-0008-cost-aware-planner.md) — `plan_hash` is the cache key for retrieval + answer caches. +- TRACKER.md Steps 1.1e (SPI split) + 4.1 (production implementation). +- [performance.md](performance.md) — hit-rate targets and observability. diff --git a/docs/architecture/performance.md b/docs/architecture/performance.md new file mode 100644 index 0000000..0f085a7 --- /dev/null +++ b/docs/architecture/performance.md @@ -0,0 +1,111 @@ +# Performance — discipline, budgets, and reviewer checklist + +## Overview + +This document captures the cross-cutting performance disciplines every contributor follows. It exists because performance at million-document scale is an *architectural* concern, not a tuning pass — the choices that determine whether you can hit p99 budgets are made at PR-review time, not during the Phase 4 latency sweep. + +Three sections: hot-path discipline (how code is written), per-SPI budgets (what's measured), reviewer checklist (what's enforced at PR time). + +## Hot-path discipline + +### Pydantic at SPI boundaries only + +Pydantic v2 validation is fast (~µs per model) but it runs every time. In a hot loop running 10k times per query, that's tens of milliseconds of pure overhead. Convention: + +| Layer | Type | Validation | +|---|---|---| +| Network boundary (gateway HTTP/gRPC) | Pydantic | Validate once | +| Public SPI surface | Pydantic | Validate once | +| Inside a single SPI call (loops) | `dataclass`, `msgspec`, or `Model.model_construct()` | None | +| Process-internal cross-stage envelopes (`RequestContext`, `QueryPlan`) | Pydantic | Validate at construction; treat as trusted | + +When trusted internal construction is needed for a Pydantic model, use `Model.model_construct(**fields)` — this skips validation entirely. Reserve for code paths where inputs are known-good (e.g., reconstructing a `RequestContext` from a cache). + +### Avoid per-call `getLogger` + +`logging.getLogger(__name__)` is fast on modern Python but still has overhead in tight loops. Always store at module level: + +```python +from rag_observability.logging import get_logger +_log = get_logger(__name__) # module level — once + +async def hot_loop(): + for item in items: + _log.debug("processing", extra={"id": item.id}) +``` + +(This is also enforced by the RAG001 pre-commit hook for `logging.getLogger`.) + +### Async all the way + +Any sync I/O call in the request path will tank p99 under load. Default to async clients for every backend. If a library only ships sync, wrap with `asyncio.to_thread()` at the SPI boundary, not in the hot path. + +### Bounded queues, never unbounded + +`asyncio.Queue()` defaults to unbounded. That's a foot-gun: a slow consumer + fast producer = OOM. Always pass `maxsize=`. The `Pipeline` primitive (Step 1.1d) enforces this. + +### Don't hydrate what you'll throw away + +Retrieve `ChunkRef` (IDs + scores), rerank, then `hydrate` only the survivors. See [Step 1.1b](../../planning/phases/phase-1-ingestion.md#step-11b--spi-split-retrieval--index-bulk--streaming--id-only) for the SPI shape. + +### Batch concurrent provider calls + +Use the `Batcher` middleware (Step 1.1d) for any SPI backed by a billed external API (embedders, rerankers, LLMs). Coalesces N concurrent calls into 1 provider call. Cuts cost ~10× and increases throughput ~3×. + +## Per-SPI p99 budgets + +Published in `tests/contract/budgets.py`; enforced by the conformance harness for every backend implementation. + +| SPI | Method | p99 budget (dev profile) | p99 budget (SaaS profile) | +|---|---|---:|---:| +| `Embedder` | `embed` (single) | 50 ms | 30 ms | +| `Embedder` | `bulk_embed` (batch 100) | 500 ms | 200 ms | +| `VectorRetrievalBackend` | `retrieve_ids` (top 100) | 50 ms | 20 ms | +| `VectorRetrievalBackend` | `hydrate` (50 chunks) | 30 ms | 15 ms | +| `KeywordRetrievalBackend` | `retrieve_ids` | 30 ms | 15 ms | +| `Reranker` | `fast_rerank` (200 → 50) | 10 ms | 5 ms | +| `Reranker` | `precise_rerank` (50 → 10) | 200 ms | 100 ms | +| `PolicyEngine` (noop) | `evaluate` | 100 µs | 100 µs | +| `PolicyEngine` (filter_pushdown) | — | 500 µs | 500 µs | +| `EmbeddingCache` | `get` | 2 ms | 1 ms | +| `RetrievalCache` | `get` | 2 ms | 1 ms | +| `AnswerCache` | `get` | 2 ms | 1 ms | +| `AuditWriter` | `write` (async) | 5 ms | 2 ms | + +Gateway end-to-end (cache miss): p99 ≤ 500 ms dev, ≤ 250 ms SaaS — the Phase 2 exit gate. + +When a budget is missed, the conformance suite fails. The fix is either (a) tune the implementation or (b) raise the budget with an ADR explaining why. + +## Async telemetry path + +OTel exporter back-pressure cannot block the request path. The shipped `rag_observability` configuration uses: + +- **Bounded buffer:** default 10,000 records. +- **Non-blocking exporter:** drops on overflow. +- **Drop counter:** `telemetry.dropped_total{kind}` metric, alertable. + +This is a Step 1.1e deliverable, tested by simulating exporter back-pressure for 60s and asserting the request path's p99 is unchanged. + +## Reviewer checklist + +Apply to every PR that touches an SPI, a hot path, or an SPI consumer: + +- [ ] **RequestContext threading.** Every new SPI method takes `ctx: RequestContext` as the first arg. Every call to an existing SPI passes `ctx` (no re-construction from globals). +- [ ] **Typed ACL access.** Code reads `chunk.tenant_id` and `chunk.acl_labels` directly (typed fields), not `chunk.metadata["tenant_id"]`. +- [ ] **No raw `logging.getLogger`.** All loggers come from `rag_observability.logging.get_logger(__name__)`. RAG001 pre-commit catches this; if it didn't run, check it's installed. +- [ ] **No unbounded queues.** Every `asyncio.Queue(...)` has `maxsize=`. +- [ ] **Pydantic discipline.** No `Model(...)` construction inside a tight loop on trusted data — use `Model.model_construct(...)` or a dataclass. +- [ ] **PolicyEngine call site.** Any new code path that reads chunks, ingests data, or emits text consults `PolicyEngine` / `PolicyWriter`. The coverage linter (`tests/policy/coverage.py`) catches misses. +- [ ] **Hydration only on survivors.** Retrieval code paths use `retrieve_ids` → rerank → `hydrate`, not full-`Chunk` retrieval up front. +- [ ] **Batcher for external APIs.** New Embedder / Reranker / LLM adapters sit under the `Batcher` middleware (or document why not). +- [ ] **Budgets honored.** If the change affects a method with a published p99 budget, the conformance suite still passes. +- [ ] **Cache key correctness.** New caching code uses the right SPI of the three (Embedding / Retrieval / Answer) for its invalidation rule. +- [ ] **BlobRef-safe.** Code that consumes `Chunk.text` handles both `str` and `BlobRef` (or hydrates explicitly). + +## Related + +- [request-context.md](request-context.md) — the per-request envelope. +- [policy-engine.md](policy-engine.md) — governance PDP. +- [caching.md](caching.md) — three caches. +- [ADR-0006](../adr/ADR-0006-two-stage-rerank.md), [ADR-0008](../adr/ADR-0008-cost-aware-planner.md), [ADR-0009](../adr/ADR-0009-vector-index-strategy.md) — performance-relevant decisions. +- TRACKER.md Steps 1.1a–1.1f (refactor window), 4.6 (latency tuning), 7.1 (load testing). diff --git a/docs/architecture/policy-engine.md b/docs/architecture/policy-engine.md new file mode 100644 index 0000000..ff6cbcf --- /dev/null +++ b/docs/architecture/policy-engine.md @@ -0,0 +1,116 @@ +# PolicyEngine — central governance decision point + +## Overview + +`PolicyEngine` is AgentContextOS's single source of truth for governance decisions: ACL enforcement, PII handling, quotas, redaction, and rate limits. Every retrieval, ingest, and egress code path consults it via `evaluate(ctx, decision, subject) → ALLOW | DENY | TRANSFORM`. + +It replaces five scattered governance touchpoints (PII at ingest, quotas, ACL push-down, ACL egress verifier, PII at egress) with one auditable surface. See [ADR-0005](../adr/ADR-0005-policy-engine.md). + +## Why it exists + +Real-world governance leaks come from **path coverage failures**: a new code path adds a feature, forgets to call one of the governance checks, and a sensitive document slips out. The mitigation in V1.0 (Step 6.4 — ACL egress verifier as a second layer) is a tacit acknowledgment that the first layer alone is unreliable. + +A Policy Decision Point (PDP) consolidates these decisions into a single, lint-enforceable surface. There is one method to forget to call (and a CI gate that fails if you do). + +## Usage + +```python +from rag_core.types import RequestContext +from rag_policy import PolicyEngine, PolicyDecision + +async def retrieve(ctx: RequestContext, query: Query) -> list[Chunk]: + # 1. Push-down: PolicyEngine returns a FilterExpr injected into the backend. + filter_expr = await policy.filter_pushdown(ctx, PolicyDecision.READ_CHUNK) + candidates = await backend.retrieve_ids(ctx, query, filter_expr) + + # 2. Per-chunk evaluation (rare — most filtering happens push-down). + allowed = [] + for ref in candidates: + result = await policy.evaluate(ctx, PolicyDecision.READ_CHUNK, ref) + if result.is_allow(): + allowed.append(ref) + elif result.is_transform(): + allowed.append(result.transformed) + # DENY → skipped, logged, audited. + + chunks = await backend.hydrate(ctx, allowed) + + # 3. Egress evaluation on text content. + return [ + (await policy.evaluate(ctx, PolicyDecision.EGRESS_TEXT, c)).transformed_or_self() + for c in chunks + ] +``` + +The `PolicyWriter` facade composes evaluation + structured log + audit-event write — most consumers should use it instead of calling `PolicyEngine` directly. + +## Decisions + +| `PolicyDecision` | Subject type | Common results | +|---|---|---| +| `READ_CHUNK` | `Chunk` or `ChunkRef` | ALLOW / DENY (ACL mismatch) | +| `INGEST_DOC` | `Document` | ALLOW / DENY (size, content-type) / TRANSFORM (PII redaction) | +| `EGRESS_TEXT` | `str` or `Chunk` | ALLOW / TRANSFORM (PII redaction) | +| `QUOTA_CHECK` | `QuotaSubject(tenant, kind, amount)` | ALLOW / DENY (over-quota) | +| `RATE_LIMIT` | `RateLimitSubject(tenant, endpoint)` | ALLOW / DENY (rate exceeded) | +| `EXECUTE_PLAN` | `QueryPlan` | ALLOW / TRANSFORM (degraded plan under cost cap) | + +`PolicyDecision` is a typed enum in `rag_policy.types`. Adding a new decision requires updating the noop impl + writing conformance tests + amending this page. + +## Internals + +### Filter push-down + +`filter_pushdown(ctx, decision)` returns a `FilterExpr` (the same mini-language used by retrieval backends — see [Step 2.1](../../planning/phases/phase-2-retrieval.md#step-21--knowledge-store-abstraction-read-path)). Backends apply it natively (pgvector WHERE clause, Qdrant payload filter, ES filter clause). This is the *primary* enforcement path — chunks the principal cannot read never leave the backend. + +### Per-chunk evaluation + +A fallback for cases where push-down cannot express the rule (rare). Consumers should prefer push-down; per-chunk evaluation in a hot loop is a perf smell and triggers a CI warning. + +### Coverage linter + +`tests/policy/coverage.py` greps for direct calls to `RetrievalBackend.retrieve`, `IndexBackend.index`, and `LLM.complete` without an adjacent `PolicyEngine` / `PolicyWriter` call within the same function scope. Failures block CI. The allowlist lives at the top of the file; additions require ADR sign-off. + +### Performance + +- Noop impl `evaluate` p99 < 100 µs (target). +- Production impls cache decisions keyed by `(ctx.principal, decision, subject_hash)` within `RequestContext` lifetime. +- `filter_pushdown` results cached per `RequestContext` — typically called once per query. + +## Extension points + +Implement `rag_policy.spi.PolicyEngine`: + +```python +class MyOrgPolicyEngine(PolicyEngine): + async def evaluate(self, ctx, decision, subject) -> PolicyResult: + if decision == PolicyDecision.READ_CHUNK: + if subject.acl_labels & ctx.principal.acls: + return PolicyResult.allow() + return PolicyResult.deny(reason="ACL mismatch") + ... + + async def filter_pushdown(self, ctx, decision) -> FilterExpr: + return FilterExpr.and_( + FilterExpr.eq("tenant_id", ctx.tenant_id), + FilterExpr.any_in("acl_labels", list(ctx.principal.acls)), + ) +``` + +Register at the composition root (`apps/gateway/`): + +```python +from rag_policy import PolicyEngine +from myorg.policy import MyOrgPolicyEngine + +policy: PolicyEngine = MyOrgPolicyEngine(...) +``` + +Future built-ins on the roadmap: `OpaPolicyEngine` (delegates to an OPA sidecar), `CedarPolicyEngine` (AWS Cedar in-process). + +## Related + +- [ADR-0005](../adr/ADR-0005-policy-engine.md) — the decision. +- [request-context.md](request-context.md) — the envelope every policy call receives. +- TRACKER.md Step 1.1c — implementation step. +- Steps 1.7, 4.5, 6.3, 6.5 — consumers; Step 6.4 superseded. diff --git a/docs/architecture/request-context.md b/docs/architecture/request-context.md new file mode 100644 index 0000000..69fcc54 --- /dev/null +++ b/docs/architecture/request-context.md @@ -0,0 +1,114 @@ +# RequestContext — the per-request envelope + +## Overview + +`RequestContext` is a frozen Pydantic v2 model that travels with every SPI call. It carries everything downstream layers need to make correct, governed, budgeted decisions without reaching back to global state or guessing. + +It is constructed exactly once — at the API gateway boundary — and passed through every subsequent call as the first argument. Downstream code treats it as a trusted, immutable object. + +## Why it exists + +Before `RequestContext`, downstream layers had to retrieve tenant / principal / budget / trace state from a mix of contextvars, thread-locals, and parameter lists. Three failure modes followed: + +1. **Coverage gaps** — a new SPI method forgets to look up the principal; ACL checks silently no-op. +2. **Test friction** — every fixture mocks contextvars in different ways. +3. **Concurrency hazards** — contextvar propagation across `asyncio.gather` is subtle. + +A single typed envelope on every call signature eliminates all three. + +## Shape + +```python +class RequestContext(BaseModel): + model_config = {"frozen": True} + + request_id: str # ULID; round-trips with the trace + tenant_id: str # required; non-empty + principal: Principal # user/service identity + ACL set + pii_policy: PiiPolicy # per-tenant: redact | mask | encrypt | tag-only | block + trace: TraceContext # OTel span + correlation IDs + budget: Budget # tokens, cost (dollars), wall_ms, max_iter + feature_flags: frozenset[str] # for shadow mode, A/B, killswitches + corpus_routing_hint: str | None # optional sticky routing across agent-loop turns +``` + +`Principal`, `PiiPolicy`, `TraceContext`, `Budget` are all frozen Pydantic models in `rag_core.types`. + +## Usage + +### At the boundary (gateway) + +```python +async def query_endpoint(req: QueryRequest, auth: AuthInfo) -> QueryResponse: + ctx = RequestContext( + request_id=ulid.new(), + tenant_id=auth.tenant_id, + principal=auth.principal, + pii_policy=tenant_config.pii_policy, + trace=TraceContext.from_otel(current_span()), + budget=req.budget or tenant_config.default_budget, + feature_flags=tenant_config.feature_flags, + corpus_routing_hint=req.session.last_corpus, + ) + return await pipeline.execute(ctx, req.query) +``` + +### Inside an SPI + +```python +class VectorRetrievalBackend(ABC): + async def retrieve_ids( + self, + ctx: RequestContext, # always first + query: Query, + filter_expr: FilterExpr | None = None, + top_k: int = 100, + ) -> list[ChunkRef]: ... +``` + +### Across an `asyncio.gather` + +Pass `ctx` explicitly to every awaitable. Do not rely on contextvars. + +```python +vec, bm25, graph = await asyncio.gather( + vector.retrieve_ids(ctx, query), + keyword.retrieve_ids(ctx, query), + graph.retrieve_ids(ctx, query), +) +``` + +## Internals + +### Immutability + +`frozen=True`. To produce a derived context (e.g., for an agent-loop sub-turn with a reduced budget), use `ctx.model_copy(update={"budget": ctx.budget.spend(used_tokens=...)})`. + +### Validation + +Validated exactly once at construction. Downstream SPIs accept it as trusted — they MUST NOT re-validate (cost is non-trivial in hot paths). See [performance.md](performance.md). + +### Type-level enforcement + +A `tests/contract/spi_signature.py` linter asserts every public SPI method's first argument is `ctx: RequestContext`. CI fails on additions that omit it. + +### Lifetime caches + +`PolicyEngine` and `Storage` (for `BlobRef` resolution) cache results keyed on `ctx.request_id` for the lifetime of a single request. The cache is held in a `ContextLocal` attached to the context, not a process-wide dict. + +## Extension points + +Adding a field is a minor-version change. Process: + +1. Add the field to `RequestContext` with a default (so existing call sites don't break). +2. Document it here (Shape table). +3. Wire it through the gateway's `RequestContext` construction. +4. Update downstream consumers that need it. + +Removing or renaming a field is a major change and requires an ADR. + +## Related + +- [policy-engine.md](policy-engine.md) — primary consumer. +- [performance.md](performance.md) — hot-path discipline; do not re-validate. +- TRACKER.md Step 1.1a — introduces the type. diff --git a/planning/EXECUTION-PLAN.md b/planning/EXECUTION-PLAN.md index 5ed04eb..b73756c 100644 --- a/planning/EXECUTION-PLAN.md +++ b/planning/EXECUTION-PLAN.md @@ -128,8 +128,8 @@ Language choices: | # | Phase | Wk | Key outputs | Eval gate at exit | |---|---|---|---|---| | 0 | Foundation | 0–4 | Monorepo, SPI, `rag.yaml`, CI/CD, IaC, eval skeleton, local stack | Build green; schema valid; sample plugin passes contract tests | -| 1 | Ingestion + Store | 4–10 | Parsers, chunker, enricher, embedder, CDC connectors, soft delete | 95% parser pass rate on 200-doc fixture; chunker preserves tables/lists | -| 2 | Retrieval Engine | 8–14 | Hybrid retrieval, QU, reranker, packer, GraphRAG | Recall@10 ≥ 0.85 on golden; MRR ≥ 0.70; nDCG ≥ 0.80 | +| 1 | Ingestion + Store | 4–10 | **Core type + SPI refactor (RequestContext, typed ACL, BlobRef, QueryPlan, StageEvent), PolicyEngine package, Pipeline + Batcher primitives, three-way cache split (Steps 1.1a–1.1f)**, parsers, chunker, enricher, embedder, CDC connectors, soft delete | 95% parser pass rate on 200-doc fixture; chunker preserves tables/lists; PolicyEngine consulted on every governance-relevant call site | +| 2 | Retrieval Engine | 8–14 | Hybrid retrieval, QU, reranker (two-stage from day 1 per ADR-0006), packer, GraphRAG, **agent-loop validation spike (Step 2.11)** | Recall@10 ≥ 0.85 on golden; MRR ≥ 0.70; nDCG ≥ 0.80; agent-loop gap report clear | | 3 | Gateway + Agent | 12–18 | REST/gRPC/MCP, corpus router, agent loop, SDKs, adapters | 5-min `pip` quickstart passes; MCP works in Claude Desktop | | 4 | Reliability | 16–22 | Cache, fallback, hallucination guard, breakers, quotas | Cache hit ≥ 30% on repeat-set; fallback triggers in chaos test | | 5 | Eval & Obs | 18–24 | Tracing, RAGAS, drift, dashboards, CI gate | CI fails on planted regression; drift alert fires on synthetic shift | @@ -246,5 +246,6 @@ Beyond the HLD glossary, these terms recur in step files: |---|---|---|---| | 1.0 | 2026-05-22 | Platform Engineering | Initial draft on `planning/V1/execution-plan` | | 1.1 | 2026-05-22 | Platform Engineering | Elevated logging and the operator GUI to first-class principles; added [LOGGING-STANDARD.md](LOGGING-STANDARD.md) and [GUI-SPECIFICATION.md](GUI-SPECIFICATION.md); expanded DoD; added Phase 0 logging step and Phase 3 GUI step. | +| 1.2 | 2026-05-24 | Platform Engineering | Inserted Phase 1 architecture-refactor window (Steps 1.1a–1.1f): RequestContext threading, typed ACL on Chunk/Embedding, BlobRef/dtype/trust_level/QueryPlan/ChunkRef/StageEvent types, PolicyEngine package as central PDP, split Retrieval/Index SPIs with bulk + streaming + ID-only methods, Pipeline + Batcher primitives, three-way cache split (Embedding/Retrieval/Answer), hot-path discipline + async telemetry. Added Phase 2 Step 2.11 agent-loop validation spike. Added ADRs 0005–0009. Step 6.4 (ACL egress verifier) marked redundant — replaced by PolicyEngine. | *End of master plan. Open the phase files to drill in.* diff --git a/planning/PROBLEM-TRACEABILITY.md b/planning/PROBLEM-TRACEABILITY.md index b7532a8..d1cd836 100644 --- a/planning/PROBLEM-TRACEABILITY.md +++ b/planning/PROBLEM-TRACEABILITY.md @@ -72,8 +72,9 @@ Legend: 🔴 Critical · 🟠 High · 🟡 Medium · ✅ Closed gate test exists ## Latency & Reliability ### 🔴 Retrieval Latency Kills Agents -- **Primary:** [4.1 Semantic cache](phases/phase-4-reliability.md#step-41--semantic-cache), [4.6 Latency tuning sweep](phases/phase-4-reliability.md#step-46--latency-tuning-sweep), [2.5.4 Concurrency budgets](phases/phase-2-retrieval.md#step-25--hybrid-retrieval-orchestrator), [2.7.3 Early-exit rerank](phases/phase-2-retrieval.md#step-27--reranker). -- **Gate test:** profile SLOs met; nightly latency regression < 10%. +- **Primary:** [1.1b Bulk + ID-only SPI methods](phases/phase-1-ingestion.md#step-11b--spi-split-retrieval--index-bulk--streaming--id-only), [1.1d Batcher middleware](phases/phase-1-ingestion.md#step-11d--pipeline--batcher-primitives), [1.1e three-cache split](phases/phase-1-ingestion.md#step-11e--cache-spi-split--perf-discipline--async-telemetry), [4.1 Semantic cache (implementation)](phases/phase-4-reliability.md#step-41--semantic-cache), [4.6 Latency tuning sweep](phases/phase-4-reliability.md#step-46--latency-tuning-sweep), [2.5.4 Concurrency budgets](phases/phase-2-retrieval.md#step-25--hybrid-retrieval-orchestrator), [2.7 Two-stage reranker (ADR-0006)](phases/phase-2-retrieval.md#step-27--reranker). +- **Supporting:** ADR-0008 (cost-aware planner — fallback before dispatch, not after timeout), ADR-0009 (vector index strategy + quantization). +- **Gate test:** profile SLOs met; nightly latency regression < 10%; per-SPI p99 budgets enforced by conformance suite. ### 🟠 No Graceful Fallback - **Primary:** [4.2 Tiered fallback chain](phases/phase-4-reliability.md#step-42--fallback-chain). @@ -104,21 +105,28 @@ Legend: 🔴 Critical · 🟠 High · 🟡 Medium · ✅ Closed gate test exists ## Enterprise & Multi-Tenancy ### 🔴 Data Leakage Across Tenants -- **Primary:** [6.1 Logical tenancy](phases/phase-6-governance.md#step-61--logical-multi-tenancy), [6.2 Physical isolation tier](phases/phase-6-governance.md#step-62--physical-isolation-tier). -- **Gate test:** 10k cross-tenant probe set returns zero leaks. +- **Primary:** [1.1a Typed `tenant_id` on Chunk/Embedding](phases/phase-1-ingestion.md#step-11a--core-type--spi-refactor-architecture-lock-in), [1.1c PolicyEngine PDP](phases/phase-1-ingestion.md#step-11c--policyengine-package-rag-policy), [6.1 Logical tenancy](phases/phase-6-governance.md#step-61--logical-multi-tenancy), [6.2 Physical isolation tier](phases/phase-6-governance.md#step-62--physical-isolation-tier). +- **Supporting:** Cross-tenant cache safety test (`tests/redteam/cross_tenant_cache.py`). +- **Gate test:** 10k cross-tenant probe set returns zero leaks; PolicyEngine coverage linter green on every CI run. ### 🔴 No Document-Level ACLs -- **Primary:** [6.3 ACL push-down](phases/phase-6-governance.md#step-63--acl-push-down-at-retrieval), [6.4 ACL egress verifier](phases/phase-6-governance.md#step-64--acl-egress-verifier). -- **Gate test:** pushdown + egress two layers verified on every store backend; defense-in-depth chaos probe. +- **Primary:** [1.1a Typed `acl_labels` on Chunk](phases/phase-1-ingestion.md#step-11a--core-type--spi-refactor-architecture-lock-in), [1.1c PolicyEngine PDP](phases/phase-1-ingestion.md#step-11c--policyengine-package-rag-policy), [6.3 ACL push-down via PolicyEngine.filter_pushdown](phases/phase-6-governance.md#step-63--acl-push-down-at-retrieval). +- **Supporting:** [ADR-0005](../docs/adr/ADR-0005-policy-engine.md). Step 6.4 (ACL egress verifier) **superseded** — defense-in-depth provided by PDP design. +- **Gate test:** filter pushdown verified on every backend; coverage linter asserts no retrieval / ingest / egress call bypasses PolicyEngine. ### 🟠 PII Leaking Through Retrieval -- **Primary:** [1.7 PII detector](phases/phase-1-ingestion.md#step-17--pii-detection-ingest-side), [6.5 PII policies + egress scanner](phases/phase-6-governance.md#step-65--pii-policies--egress-scanner). -- **Gate test:** ≥ 99% PII recall; zero unredacted egress on probe corpus. +- **Primary:** [1.7 PII detector](phases/phase-1-ingestion.md#step-17--pii-detection-ingest-side) (Presidio plugin), [1.1c PolicyEngine `EGRESS_TEXT` decision](phases/phase-1-ingestion.md#step-11c--policyengine-package-rag-policy), [6.5 PII policies](phases/phase-6-governance.md#step-65--pii-policies--egress-scanner) (per-tenant policy applied through PDP). +- **Gate test:** ≥ 99% PII recall; zero unredacted egress on probe corpus; PolicyEngine `EGRESS_TEXT` called on every text return path. ### 🟠 No Audit Trail - **Primary:** [6.6 Immutable audit log w/ hash chain + WORM export](phases/phase-6-governance.md#step-66--immutable-audit-log). - **Gate test:** tamper attempt detected nightly; chain replay reconstructs sessions. +### 🟠 Prompt Injection via Retrieved Content +- **Primary:** [1.1a `trust_level` typed field on Chunk](phases/phase-1-ingestion.md#step-11a--core-type--spi-refactor-architecture-lock-in), consumed by [2.8 Context packer](phases/phase-2-retrieval.md#step-28--context-packer) and [3.1 Gateway LLM adapter](phases/phase-3-gateway-agent.md#step-31--gateway-service-rest--openapi) to choose isolation strategy. +- **Supporting:** [4.3 Hallucination guard](phases/phase-4-reliability.md#step-43--hallucination-guard) (downstream defense-in-depth), [7.3 Red-team prompt-injection corpus](phases/phase-7-pilot-ga.md#step-73--red-team). +- **Gate test:** prompt-injection red-team corpus achieves ≥ 95% block rate at LLM-adapter level; no `trust_level=external` chunk is serialized into a system-trust position in any test case. + --- ## Agentic-Specific Failures @@ -139,4 +147,4 @@ Legend: 🔴 Critical · 🟠 High · 🟡 Medium · ✅ Closed gate test exists ## Summary -22 distinct problem entries; every one has a primary step and a named acceptance gate test. Each phase file restates these so a reviewer can verify that a phase actually closes the problems it claims to. +23 distinct problem entries; every one has a primary step and a named acceptance gate test. Each phase file restates these so a reviewer can verify that a phase actually closes the problems it claims to. The Phase 1 refactor window (Steps 1.1a–1.1f, ADRs 0005–0009) tightens the *primary* step for governance and latency problems by introducing typed ACL fields, the PolicyEngine PDP, the two-stage reranker default, the cost-aware planner, and the vector index strategy. diff --git a/planning/RISK-REGISTER.md b/planning/RISK-REGISTER.md index 5dc68a6..7854833 100644 --- a/planning/RISK-REGISTER.md +++ b/planning/RISK-REGISTER.md @@ -15,7 +15,7 @@ Tracked risks for the V1 build. Each has an owner, a mitigation, a detector (how - **Owner:** ML / Retrieval lead. - **Detector:** Phase 2 eval shows MRR lift but P95 latency target missed. -- **Mitigation:** Two-stage rerank with early-exit (Step 2.7); ONNX-optimized local models; GPU autoscaling option. +- **Mitigation:** Two-stage rerank with early-exit shipped *as the default SPI shape* from day 1 per [ADR-0006](../docs/adr/ADR-0006-two-stage-rerank.md) — not bolted on during latency tuning. ONNX-optimized local models; GPU autoscaling option. Combined with `retrieve_ids` + `hydrate` (Step 1.1b) so stage-1 reranking operates on `ChunkRef` without full text. - **Kill criteria:** If quality lift < 0.05 MRR after tuning, switch primary path to retrieval-only with reranker opt-in only. ## R3 — Eval set quality degrades @@ -29,14 +29,14 @@ Tracked risks for the V1 build. Each has an owner, a mitigation, a detector (how - **Owner:** Platform lead. - **Detector:** Backend-specific quirks leaking into core (e.g., Qdrant-isms in retriever code). -- **Mitigation:** Plugin SPI conformance suite from day 1 (0.3); cross-backend nightly conformance run; ADR required to introduce backend-specific code paths. +- **Mitigation:** Plugin SPI conformance suite from day 1 (0.3); SPI split into `RetrievalBackend` + `IndexBackend` (Step 1.1b) so read and write concerns can't conflate; `IndexHint` per [ADR-0009](../docs/adr/ADR-0009-vector-index-strategy.md) lets backends pick index strategy without leaking choices into consumers; cross-backend nightly conformance run; ADR required to introduce backend-specific code paths. - **Kill criteria:** If conformance suite cannot be passed by ≥ 3 backends within Phase 2, pause backend additions and refactor SPI. ## R5 — Multi-tenant noisy neighbor - **Owner:** SRE. - **Detector:** Latency or error budgets shared tenants impacted by one heavy tenant. -- **Mitigation:** Quotas (4.5); bulkheads (4.4); physical tier (6.2) for top customers. +- **Mitigation:** Quotas via PolicyEngine `QUOTA_CHECK` decision (Step 1.1c + 4.5); bulkheads (4.4); per-tenant connection pools + fairness scheduler designed by ADR (Step 1.1f, built in 6.1); physical tier (6.2) for top customers; cost-aware planner (ADR-0008) caps per-query budget before dispatch. - **Kill criteria:** If quota/bulkhead alone cannot keep P95 within SLO for ≥ 95% of tenants, default Enterprise tier to physical isolation. ## R6 — On-prem demand outpaces SaaS @@ -78,7 +78,7 @@ Tracked risks for the V1 build. Each has an owner, a mitigation, a detector (how - **Owner:** Security + Governance lead. - **Detector:** Red-team probe surfaces a leak path; or audit reveals PII in unexpected place (logs, traces, error reports). -- **Mitigation:** Default-deny PII in spans; secret scanning in logs; structured logging review checklist; nightly red-team (Phase 6 onward). +- **Mitigation:** Default-deny PII in spans; secret scanning in logs; structured logging review checklist; PolicyEngine coverage linter (`tests/policy/coverage.py`, Step 1.1c) blocks CI if any retrieval / ingest / egress path bypasses the PDP; cross-tenant cache safety probe (`tests/redteam/cross_tenant_cache.py`); nightly red-team (Phase 6 onward). - **Kill criteria:** Any cross-tenant leak in production triggers immediate incident response per Step 7.8 runbook. ## R12 — Hallucination guard over-blocks diff --git a/planning/phases/phase-1-ingestion.md b/planning/phases/phase-1-ingestion.md index cba0e1b..e1047d4 100644 --- a/planning/phases/phase-1-ingestion.md +++ b/planning/phases/phase-1-ingestion.md @@ -43,6 +43,229 @@ --- +## Step 1.1a — Core type & SPI refactor (architecture lock-in) + +**Solves (preventatively):** ACL push-down at retrieval (6.3) hitting JSON-filter perf wall, embedder unable to batch concurrent callers, reranker latency budget blown, prompt-injection defense having no place to live in the type system, every later SPI written without RequestContext threading. + +**Inputs:** 0.3 (SPI), 1.1 (storage backends), prior-art noop implementations. + +**Deliverables (in `packages/core/src/rag_core/`):** + +### 1.1a.1 `RequestContext` frozen model (`types.py`) +Fields: `tenant_id`, `principal`, `acls`, `pii_policy`, `trace` (`TraceContext`), `budget` (tokens / cost / wall / iter), `request_id`, `feature_flags`, `corpus_routing_hint`. Required, immutable, threaded through every SPI method call. + +### 1.1a.2 Typed ACL on `Chunk` / `Embedding` +Move `tenant_id` and `acl_labels: tuple[str, ...]` from `Chunk.metadata` (dict) to typed fields. Index these columns at every backend (PgVector: `CREATE INDEX … on (tenant_id, acl_labels)`; Qdrant: payload-index; ES: keyword field). Removes JSON-filter cost from retrieval hot path. + +### 1.1a.3 `trust_level` on `Chunk` +`trust_level: Literal["system", "tenant_curated", "tenant_user", "external"]`. Consumed by context packer (Step 2.8) and LLM adapter (Step 3.1) to choose isolation strategy (XML-tagged section, separate turn, refusal threshold). Foundation for prompt-injection defense referenced in HLD §10. + +### 1.1a.4 `dtype` on `Embedding` +`dtype: Literal["float32", "int8", "binary"]`. Unblocks Cohere-v3 int8 (4× memory cut) and binary (32× cut) without an `Embedding` schema break later. + +### 1.1a.5 `BlobRef` for lazy chunk text +`Chunk.text: str | BlobRef` where `BlobRef = (storage_provider, key, byte_range)`. Context packer hydrates on demand. Required before 1.10 (write path) ships, otherwise large-doc ingest holds the world in memory. + +### 1.1a.6 `QueryPlan` first-class type +Frozen model: `subqueries`, `chosen_backends`, `backend_weights`, `filter_pushdowns`, `rerank_policy`, `packer_policy`, `estimated_cost: Cost(ms, tokens, dollars)`. Produced by query-understanding (Step 2.6), consumed by retrieval (Steps 2.1–2.5), enables `/v1/query/explain`, plan caching, shadow A/B (5.7), agent-loop turn reuse (3.6). + +### 1.1a.7 `ChunkRef` for projection +Lightweight record: `chunk_id`, `score`, `tenant_id`, `acl_labels`. Returned by `retrieve_ids`; expanded to full `Chunk` only for top-N post-rerank via `hydrate`. + +### 1.1a.8 Typed `StageEvent` +Promote the event registry (Step 0.7b) to typed `StageEvent(stage_name, inputs_hash, outputs_hash, latency_ms, cost, attributes)`. Returned alongside SPI results, captured by pluggable `EvalRecorder` sink (noop in OSS, ClickHouse in Enterprise). Substrate for online eval (5.4), drift (5.5), shadow mode (5.7), hallucination guard (4.3). + +**Implementation notes:** +- `RequestContext` validates exactly once at the gateway boundary; downstream SPIs accept it as a trusted object — see `docs/architecture/performance.md` for the hot-path convention. +- ACL label cardinality is bounded per tenant (config cap, default 200); above that, fall back to a hash-bucket scheme so backend indexes don't bloat. +- `BlobRef` resolution uses `Storage` SPI; default implementations cache resolved bytes within `RequestContext` lifetime. + +**Test plan:** +- Type test: every public SPI method's signature includes `ctx: RequestContext` as first arg (linter rule in `tests/contract/`). +- Property test: `Chunk.acl_labels` is required and typed; constructing a chunk without it fails at validation time. +- Round-trip: `BlobRef`-backed `Chunk` packs and unpacks identically to inline-text `Chunk` in the packer. +- Conformance: existing 79+ SPI conformance tests pass after `RequestContext` threading. + +**Acceptance criteria:** +- [ ] Every SPI method in `packages/core/src/rag_core/spi/` accepts `ctx: RequestContext`. +- [ ] `Chunk` and `Embedding` carry typed `tenant_id` + `acl_labels`; metadata-dict access for these fields removed. +- [ ] `trust_level`, `dtype`, `BlobRef`, `QueryPlan`, `ChunkRef`, `StageEvent` exported from `rag_core.types`. +- [ ] Backwards-compat shims for old call sites are NOT added — this is a clean break enforced by mypy. +- [ ] `docs/architecture/request-context.md` written. + +--- + +## Step 1.1b — SPI split: Retrieval / Index, bulk + streaming + ID-only + +**Solves (preventatively):** CDC (1.9) pain from mixed read/write SPI; ingest OOMs at 100K+ docs due to list-shaped APIs; vector retrieval over-fetching full chunks when rerank keeps 10; vector index strategy hardcoded at ivfflat (per ADR-0004 §2) and unable to scale past ~1M vectors. + +**Inputs:** 1.1a. + +**Deliverables (in `packages/core/src/rag_core/spi/`):** + +### 1.1b.1 Split read/write SPIs +`VectorStore` → `VectorRetrievalBackend` + `VectorIndexBackend`. Same for `KeywordStore`, `GraphStore`. Backends commonly implement both, but the SPI surface is split because their consistency, transaction, and update semantics differ. Required before CDC (1.9) gets ugly. + +### 1.1b.2 Bulk methods +`bulk_embed(ctx, texts) -> list[Embedding]`, `bulk_index(ctx, chunks) -> None`, `bulk_delete(ctx, chunk_ids) -> None`, `bulk_retrieve(ctx, queries) -> list[list[ChunkRef]]` for re-eval. Adding now: 0.5d per SPI. Adding after 6 backends ship: rewrite every backend. + +### 1.1b.3 Streaming ingest variants +`async def ingest(ctx, source) -> AsyncIterator[Chunk]` on `Connector` and `Parser` SPIs. No more list-of-100K-chunks held in memory. + +### 1.1b.4 ID-only retrieval + hydrate pair +`retrieve_ids(ctx, query) -> list[ChunkRef]` and `hydrate(ctx, ids) -> list[Chunk]`. Hybrid retrieval at the gateway calls `retrieve_ids` 3× (vector + BM25 + graph), unions, reranks ChunkRefs, then `hydrate` only the surviving top-N. Cuts retrieval-stage latency 3–5× at scale. + +### 1.1b.5 `IndexHint` parameter +`bulk_index(ctx, chunks, hint: IndexHint)` where `IndexHint = (estimated_size, recall_target, latency_target, write_volume)`. Backend chooses: flat / ivfflat / HNSW / IVF-PQ / DiskANN. Backed by ADR-0009. + +**Test plan:** +- Conformance: every backend implements both read and write SPIs; contract suite split into `retrieval/` and `index/` directories. +- Memory: ingesting a 1 GB synthetic corpus uses bounded memory (peak < 200 MB) — enforced by pytest-memray. +- ID-only round-trip: `hydrate(retrieve_ids(q))` returns the same chunks as a single-call `retrieve(q)`. +- `IndexHint` honored: PgVectorStore picks HNSW for >1M-row hint, ivfflat for <100K hint. + +**Acceptance criteria:** +- [ ] All Phase 1.1 backends (PgVector, Qdrant, Redis, S3) refactored to the split SPI. +- [ ] Bulk + streaming variants ship with default implementations that fall back to single-call when backend has no native bulk. +- [ ] `retrieve_ids` / `hydrate` pair in conformance suite. +- [ ] ADR-0009 (vector index strategy) authored and merged. + +--- + +## Step 1.1c — PolicyEngine package (`rag-policy`) + +**Solves (preventatively):** Governance bypass-by-omission. Today ACL (6.3), PII (1.7/6.5), quotas (4.5), egress redaction (6.5), audit gating are five scattered call sites. A new code path that forgets one of them is exactly how real systems leak. + +**Inputs:** 1.1a (RequestContext). + +**Deliverables:** + +- New workspace package `packages/policy/` (`rag-policy`), import root `rag_policy`. +- `PolicyEngine` SPI in `rag_policy.spi`: + ```python + async def evaluate( + ctx: RequestContext, + decision: PolicyDecision, # READ_CHUNK | INGEST_DOC | EGRESS_TEXT | QUOTA_CHECK | … + subject: Any, + ) -> PolicyResult # ALLOW | DENY(reason) | TRANSFORM(redacted_subject) + ``` +- `NoopPolicyEngine` always-allow reference impl (parity with `NoopAuditStore` pattern). +- `PolicyEngine` consulted by every retrieval, ingest, and egress path. Replaces ad-hoc checks in Steps 1.7 (PII at ingest), 4.5 (quotas), 6.3 (ACL push-down), 6.5 (PII egress). +- ACL push-down (6.3) becomes: `PolicyEngine.filter_pushdown(ctx)` returns a `FilterExpr` injected into retrieval — no separate egress verifier needed (6.4 becomes redundant by construction). +- `PolicyWriter` facade pattern (mirrors `AuditWriter`) — wraps `PolicyEngine` + structured log. + +**Test plan:** +- 20+ conformance tests covering ALLOW/DENY/TRANSFORM paths for every `PolicyDecision`. +- Coverage gate: any new SPI call site that touches user data must call `PolicyEngine.evaluate` — enforced by a `tests/policy/coverage.py` linter that greps for `RetrievalBackend.retrieve` / `IndexBackend.index` calls without an adjacent policy call. +- Performance: noop `evaluate` p99 < 100 µs. + +**Acceptance criteria:** +- [ ] `packages/policy/` ships with noop impl + 20+ conformance tests. +- [ ] No retrieval / ingest / egress call site bypasses `PolicyEngine`. +- [ ] ADR-0005 (PolicyEngine PDP) merged. +- [ ] `docs/architecture/policy-engine.md` written. +- [ ] Step 6.4 (ACL egress verifier) marked redundant in TRACKER + planning docs; replaced by a single policy decision point. + +--- + +## Step 1.1d — Pipeline + Batcher primitives + +**Solves (preventatively):** Ingest at 1M+ docs requires a fan-out DAG with backpressure (parse → chunk → enrich → PII → embed → store, parallel per stage, bounded queues between stages). Without a Pipeline primitive, 6 connectors will each invent their own orchestration. Concurrent embedder/reranker calls cost 10× more without auto-batching. + +**Inputs:** 1.1a. + +**Deliverables:** + +### 1.1d.1 `Pipeline` primitive (`packages/core/src/rag_core/pipeline.py`) +Async DAG with: +- Typed stages (`Stage[In, Out]`) with configurable worker count per stage. +- Bounded `asyncio.Queue` between stages (configurable size; default 1000). +- Backpressure: a slow downstream stage causes upstream to block, not OOM. +- Lifecycle: graceful drain on shutdown, cancellation propagation. +- Telemetry: per-stage queue depth, throughput, latency metrics. +- ~200 lines; not Temporal / Dagster — those are V2 if scale demands. + +Used by Step 1.10 (knowledge-store write path) and every Phase 1 connector. + +### 1.1d.2 `Batcher[Req, Resp]` middleware (`packages/core/src/rag_core/batcher.py`) +DataLoader pattern. Coalesces concurrent SPI calls within a configurable window (default 50 ms) into a single batched provider call. Sits *under* the Embedder and Reranker SPIs so every adapter benefits transparently. Cuts OpenAI/Cohere bills ~10× and increases throughput ~3×. + +**Test plan:** +- Pipeline: 1M-element synthetic stream completes with peak memory < 200 MB and zero drops. +- Pipeline: kill a worker mid-stream; remaining workers drain the queue; no data loss. +- Batcher: 100 concurrent `embed` calls within window result in 1 provider call with 100 inputs. +- Batcher: latency overhead at zero contention < 1 ms. + +**Acceptance criteria:** +- [ ] `Pipeline` documented in `docs/architecture/performance.md` with a Step-1.10 worked example. +- [ ] `Batcher` integrated into Embedder + Reranker SPI default base class. +- [ ] Both have observability metrics (queue depth, batch size histogram, coalescing rate). + +--- + +## Step 1.1e — Cache SPI split + perf discipline + async telemetry + +**Solves (preventatively):** A single "semantic cache" (current Step 4.1 phrasing) produces either stale answers (when corpus changes invalidate at the wrong granularity) or a 5% hit rate. The three caches have fundamentally different invalidation triggers. Pydantic in the hot loop adds 30 µs × thousands of calls and quietly destroys p99. OTel exporter back-pressure blocks the request path. + +**Inputs:** 1.1a. + +**Deliverables:** + +### 1.1e.1 Three cache SPIs (`packages/core/src/rag_core/spi/cache.py`) +- `EmbeddingCache` — key: `(model_id, model_version, text_hash)`. Invalidate on embedder model change. +- `RetrievalCache` — key: `(plan_hash, corpus_version)`. Invalidate on doc change in queried corpus. +- `AnswerCache` — key: `(plan_hash, corpus_version, policy_version)`. Invalidate on any of the above + policy change. + +Single underlying backend (Redis) is fine; the SPIs are about *invalidation contracts*, not storage. The general-purpose `Cache` SPI remains for misc use. + +### 1.1e.2 Hot-path convention (`docs/architecture/performance.md`) +- Pydantic frozen models **at SPI boundaries only**. +- Inside hot loops: dataclasses, `msgspec`, or `Model.model_construct()` (no validation) on trusted internal data. +- Per-SPI p99 budgets published in `tests/contract/budgets.py` and enforced by conformance harness. + +### 1.1e.3 Async telemetry path (`packages/observability/`) +- Bounded buffer (default 10k records) in front of OTel exporter. +- Non-blocking exporter (drop-on-overflow), with a `telemetry.dropped` counter the alerting layer watches. +- No request-path span emission blocks on exporter back-pressure. + +**Test plan:** +- Cache: distinct invalidation tests for each of the three caches (force a model bump, a corpus version bump, a policy version bump — verify the correct cache misses). +- Perf budget: conformance suite enforces published p99 budgets per SPI (embedder, vector retrieve, rerank). +- Telemetry: simulate exporter back-pressure for 60s; request-path latency unchanged; `telemetry.dropped` counter advances. + +**Acceptance criteria:** +- [ ] Three cache SPIs in `rag_core.spi`; noop impls in `rag_core.spi.noop`. +- [ ] `docs/architecture/caching.md` describes invalidation rules per cache. +- [ ] `docs/architecture/performance.md` written and linked from CLAUDE.md. +- [ ] OTel exporter back-pressure cannot block the request path (tested). + +--- + +## Step 1.1f — ADRs 0005–0009 + reviewer checklist + +**Solves:** Locking in the design decisions for Phases 2/4/6 so they don't get re-litigated mid-build. + +**Inputs:** 1.1a–1.1e (the changes they capture). + +**Deliverables:** + +- **ADR-0005** — PolicyEngine as central PDP (replaces scattered governance checks; Step 6.4 egress verifier marked redundant). +- **ADR-0006** — Two-stage reranker as the default SPI shape (fast bi-encoder → top-50 → cross-encoder → top-10), making Step 2.7 implement it from day one rather than as a perf rescue. +- **ADR-0007** — Tiered storage (hot/warm/cold) with `BlobRef` as the addressing mechanism; Phase 6 storage decisions inherit. +- **ADR-0008** — Cost-aware planner: `QueryPlan` carries `estimated_cost`; fallback (Step 4.2) is triggered planner-side *before* an expensive call is made, not by post-hoc timeout. +- **ADR-0009** — Vector index strategy by scale tier (flat → ivfflat → HNSW → IVF-PQ → DiskANN) + quantization (int8 / binary); resolves the ADR-0004 §2 "ivfflat hardcoded" gap. + +Plus: a one-page reviewer checklist in `docs/architecture/performance.md` covering: RequestContext threading, typed ACL access, BlobRef usage, hot-path Pydantic discipline, PolicyEngine call site coverage. + +**Test plan:** All ADRs reviewed by ≥ 2 maintainers; cross-linked from affected step pages; PR comment template updated. + +**Acceptance criteria:** +- [ ] Five ADRs merged in `docs/adr/`. +- [ ] Reviewer checklist appended to PR template (`.github/PULL_REQUEST_TEMPLATE.md` if present, else `docs/architecture/performance.md` linked from CLAUDE.md). +- [ ] PROBLEM-TRACEABILITY.md and RISK-REGISTER.md updated to reflect changes (Steps 6.4 redundancy; R2/R4/R5 mitigations hardened). + +--- + ## Step 1.2 — Connector framework **Solves:** "Unstructured format chaos" (sourcing half of it), "No incremental sync" (CDC entry-points). diff --git a/planning/phases/phase-2-retrieval.md b/planning/phases/phase-2-retrieval.md index 8953b7f..dc03753 100644 --- a/planning/phases/phase-2-retrieval.md +++ b/planning/phases/phase-2-retrieval.md @@ -293,9 +293,39 @@ Returns a parallel `citations: list[Citation]` matching pack order, each with `c --- +## Step 2.11 — Agent-loop validation spike + +**Solves (preventatively):** Agent loops change retrieval access patterns — many small queries, shared embedding cache, iterative refinement, budget enforcement across turns, partial plan reuse, sticky corpus routing per session. If Phase 2 is tuned only against single-shot queries, you'll redesign Phase 2 during Phase 3. + +**Inputs:** 2.1–2.10, 1.1a (`RequestContext.budget`, `QueryPlan`). + +**Deliverables:** + +- A thin `agent_loop_v0` in `apps/gateway/` (not the full Step 3.6 loop — minimal). Behavior: + - Iterative `retrieve_ids` + `hydrate` + LLM call cycle. + - Budget enforced from `RequestContext.budget` (tokens, cost, wall-clock, iter count). + - Reuses `QueryPlan` across turns (mutation, not regeneration). + - Shares `EmbeddingCache` and `RetrievalCache` across turns. + - Sticky `corpus_routing_hint` per session. +- Harness `tests/agent_loop_spike/` with 50 agent-style queries covering: 2-hop, 3-hop, follow-up clarification, contradictory-evidence resolution, "I don't know" graceful exit on budget exhaustion. +- Output: a gap report `docs/architecture/agent-loop-gap-report.md` listing any Phase 2 design choices that need correction before Phase 3.1 starts. + +**Test plan:** +- All 50 queries complete without OOM, without budget runaway, without cross-tenant leakage. +- Cache reuse: turn N+1 of a session has ≥ 50% cache hit on embeddings issued in turn N. +- Budget enforcement: a query with `iter=1` cannot run a second retrieval. + +**Acceptance criteria:** +- [ ] Spike completes; 50-query harness passes. +- [ ] Gap report merged; any blocking findings opened as Phase 2 follow-up issues before Phase 3.1. +- [ ] No new public SPIs introduced — spike consumes existing surfaces only. + +--- + ## Phase 2 exit gate - [ ] Eval gate satisfied (Recall@10 ≥ 0.85, MRR ≥ 0.70, nDCG ≥ 0.80, multi-hop recall ≥ 0.70). - [ ] Latency: p95 ≤ 500 ms on dev profile, ≤ 250 ms on SaaS profile. - [ ] Demo: one query each for (exact-ID lookup, vocabulary-mismatch, multi-hop, conflicting docs) all return correct citations. +- [ ] Agent-loop spike (2.11) gap report has zero blocking findings open for Phase 3.1. - [ ] Phase 2 retrospective lists every retrieval failure type the team saw and the test now guarding against it.