feat(core): RequestContext + ctx-threaded SPI (Step 1.1a) - #44
Merged
officialCodeWork merged 1 commit intoMay 23, 2026
Merged
Conversation
Phase 1 Step 1.1a — locks in the per-request envelope architecture before
resuming the connectors framework. This is the first of the 1.1a–1.1f
refactor window; ADR-0005 / 0007 / 0008 / 0009 are referenced consumers.
Types (new in rag_core.types):
- RequestContext — frozen per-request envelope; gateway constructs once,
every SPI takes ctx as first arg. Validates tenant_id == principal.tenant_id.
- Principal.acl_labels — typed frozenset[str] (was metadata dict).
- PiiPolicy + PiiAction — per-tenant PII contract on ctx.
- Budget — tokens / dollars / wall_ms / max_iter, with .spend() for sub-turns.
- BlobRef — lazy chunk-text pointer (ADR-0007).
- ChunkRef — ID-only retrieval result (Step 1.1b consumer).
- QueryPlan / PlanNode / Cost — planner output with estimated_cost (ADR-0008).
- StageEvent + StageEventKind — typed cross-stage observation feeding the
online cost estimator.
- TrustLevel — chunk provenance for prompt-injection defense.
- EmbeddingDtype — float32 / int8 / binary (ADR-0009 quantization).
Types changed:
- Chunk: typed `acl_labels`, `trust_level`, `content_ref: BlobRef | None`;
`content` now optional (either content or content_ref must be set).
- Embedding: typed required `tenant_id` + `acl_labels`; new `dtype` field.
SPIs (every public method now takes ctx: RequestContext first):
- VectorStore, KeywordStore, GraphStore, Embedder, Reranker, LLM, Cache,
Queue, Storage, Secrets, Connector, Parser, OCR, PIIDetector.
- Documented exceptions (in tests/contract/spi_signature.py):
Auth (runs pre-ctx at the gateway boundary), Telemetry (cross-cutting
infra owning TraceContext), AuditStore (consumed via AuditWriter facade),
HealthCheckMixin.health, Embedder.{model,dimension}, Parser.supports.
Signature linter:
- New `tests/contract/spi_signature.py` reflects every SPI ABC and asserts
the first non-self parameter is `ctx: RequestContext`. Wired into pytest
via the `python_files` override in pyproject.toml. Fails CI on additions
that omit ctx.
Backends (rag_backends):
- PgVectorStore / QdrantVectorStore: drop redundant tenant_id arg; derive
from ctx.tenant_id.
- RedisCache: keys are now namespaced as `<prefix><tenant_id>:<key>` so
tenants can't accidentally collide.
- LocalFileStorage / S3Storage: per-tenant key prefix derived from ctx;
path-traversal check tightened to the per-tenant root.
Tests:
- tests/contract/: conftest exposes `ctx` and `other_ctx` fixtures + a
`make_ctx()` helper; every conformance test updated to pass ctx.
- packages/core/tests/test_types.py: extensive coverage for new types
(RequestContext invariant, Budget.spend, Chunk BlobRef variant, etc.).
- tests/integration/: pgvector / qdrant / redis / s3 / local-storage suites
migrated to the new SPI signatures plus a tenant-isolation test where
applicable.
Result: 508 passed, 33 skipped (integration tests requiring services);
mypy --strict clean; ruff clean; schemas regenerated under dist/schemas/.
Docs:
- docs/reference/rag-core.md (new) — public type surface, usage examples,
internals (why acl_labels typed, why trust_level lives on Chunk).
- docs/architecture/request-context.md updated to mark Step 1.1a as
implemented; shape table reflects RequestId / created_at / typed enums.
- docs/README.md indexes the new reference page.
- TRACKER.md: Step 1.1a → ✅; Phase 1 done-count 1 → 2; Next action → 1.1b.
Bumps rag-core 0.2.0 → 0.3.0, rag_core.__version__ 0.4.0 → 0.5.0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
5 tasks
officialCodeWork
pushed a commit
that referenced
this pull request
May 24, 2026
… (Step 1.1c)
New workspace package packages/policy/ (import root rag_policy v0.1.0).
Establishes the single Policy Decision Point ADR-0005 calls for: every
retrieval / ingest / egress code path consults PolicyEngine, and a CI
linter fails when a governance-relevant SPI call lands without an
adjacent consultation.
Surface
- PolicyEngine ABC: async evaluate(ctx, decision, subject) -> PolicyResult
and async filter_pushdown(ctx, decision) -> FilterExpr.
- NoopPolicyEngine: always-ALLOW; filter_pushdown still emits a
tenant-scoped And(Eq("tenant_id", ...)) so backends never cross-tenant
leak even with the noop loaded.
- PolicyWriter facade (mirrors AuditWriter): delegates to engine, emits
policy.decision structured log via rag-observability.
- PolicyDecision enum: read_chunk / ingest_doc / egress_text /
quota_check / rate_limit / execute_plan.
- PolicyResult: frozen union (allow / deny(reason) / transform(subject))
with is_allow / is_deny / is_transform / transformed_or helpers.
- QuotaSubject / RateLimitSubject for the non-Chunk decision subjects.
- FilterExpr mini-language (Eq, AnyIn, And, Or, Not, TrueExpr):
discriminated-union of frozen Pydantic models; backends translate to
native filter languages.
Coverage linter
- tests/policy/coverage.py greps for retrieve_ids / hydrate / bulk_index
/ stream_index / bulk_embed / .complete call sites without an adjacent
PolicyEngine / PolicyWriter marker. File-allowlist at top; failures
block CI. Consumers (gateway, ingest) shrink the allowlist as they
wire the PDP in. Future Step 1.1f tightens to call-pattern matching.
- Collected via pytest python_files extended to include coverage.py.
Wiring
- packages/policy added to [tool.uv.workspace].members and to pytest
pythonpath in root pyproject.toml.
- Dependencies: rag-core + rag-observability only (matches backends
precedent; CLAUDE.md graph: policy -> core).
Tests + gates
- 20 conformance tests under tests/contract/test_policy_engine.py.
- 1 coverage-linter run under tests/policy/coverage.py.
- ruff + mypy --strict (83 source files, +7 from 1.1b) + RAG001 logging
check all green; full non-integration suite 528 tests pass.
Docs
- New docs/reference/rag-policy.md (Overview / Usage / Internals /
Extension points), docs/README.md index updated. ADR-0005 and
docs/architecture/policy-engine.md already match the shipped shape.
TRACKER housekeeping
- Step 1.1c flipped to done; Next action set to 1.1d (Pipeline +
Batcher primitives).
- PR history table: added rows for #44 (1.1a) and #45 (1.1b) which
were missed when those steps merged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
officialCodeWork
pushed a commit
that referenced
this pull request
May 24, 2026
… (Step 1.1c)
New workspace package packages/policy/ (import root rag_policy v0.1.0).
Establishes the single Policy Decision Point ADR-0005 calls for: every
retrieval / ingest / egress code path consults PolicyEngine, and a CI
linter fails when a governance-relevant SPI call lands without an
adjacent consultation.
Surface
- PolicyEngine ABC: async evaluate(ctx, decision, subject) -> PolicyResult
and async filter_pushdown(ctx, decision) -> FilterExpr.
- NoopPolicyEngine: always-ALLOW; filter_pushdown still emits a
tenant-scoped And(Eq("tenant_id", ...)) so backends never cross-tenant
leak even with the noop loaded.
- PolicyWriter facade (mirrors AuditWriter): delegates to engine, emits
policy.decision structured log via rag-observability.
- PolicyDecision enum: read_chunk / ingest_doc / egress_text /
quota_check / rate_limit / execute_plan.
- PolicyResult: frozen union (allow / deny(reason) / transform(subject))
with is_allow / is_deny / is_transform / transformed_or helpers.
- QuotaSubject / RateLimitSubject for the non-Chunk decision subjects.
- FilterExpr mini-language (Eq, AnyIn, And, Or, Not, TrueExpr):
discriminated-union of frozen Pydantic models; backends translate to
native filter languages.
Coverage linter
- tests/policy/coverage.py greps for retrieve_ids / hydrate / bulk_index
/ stream_index / bulk_embed / .complete call sites without an adjacent
PolicyEngine / PolicyWriter marker. File-allowlist at top; failures
block CI. Consumers (gateway, ingest) shrink the allowlist as they
wire the PDP in. Future Step 1.1f tightens to call-pattern matching.
- Collected via pytest python_files extended to include coverage.py.
Wiring
- packages/policy added to [tool.uv.workspace].members and to pytest
pythonpath in root pyproject.toml.
- Dependencies: rag-core + rag-observability only (matches backends
precedent; CLAUDE.md graph: policy -> core).
Tests + gates
- 20 conformance tests under tests/contract/test_policy_engine.py.
- 1 coverage-linter run under tests/policy/coverage.py.
- ruff + mypy --strict (83 source files, +7 from 1.1b) + RAG001 logging
check all green; full non-integration suite 528 tests pass.
Docs
- New docs/reference/rag-policy.md (Overview / Usage / Internals /
Extension points), docs/README.md index updated. ADR-0005 and
docs/architecture/policy-engine.md already match the shipped shape.
TRACKER housekeeping
- Step 1.1c flipped to done; Next action set to 1.1d (Pipeline +
Batcher primitives).
- PR history table: added rows for #44 (1.1a) and #45 (1.1b) which
were missed when those steps merged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced May 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 1 Step 1.1a — locks in the per-request envelope architecture before resuming the connectors framework. First of the 1.1a–1.1f refactor window; ADR-0005 / 0007 / 0008 / 0009 are referenced consumers.
rag_core.types:RequestContext(frozen, gateway-constructed, validatestenant_id == principal.tenant_id),PiiPolicy+PiiAction,Budget(with.spend()for agent-loop sub-turns),BlobRef(lazy chunk text — ADR-0007),ChunkRef(ID-only retrieval — Step 1.1b),QueryPlan/PlanNode/Cost(ADR-0008),StageEvent+StageEventKind,TrustLevel,EmbeddingDtype(ADR-0009).Chunkgains typedacl_labels/trust_level/content_ref: BlobRef | None(content now optional, either-or invariant).Embeddinggains requiredtenant_id+acl_labelsanddtype.Principalgains typedacl_labels.VectorStore,KeywordStore,GraphStore,Embedder,Reranker,LLM,Cache,Queue,Storage,Secrets,Connector,Parser,OCR,PIIDetectornow takesctx: RequestContextfirst. Documented exceptions:Auth(pre-ctx boundary),Telemetry(cross-cutting infra),AuditStore(consumed viaAuditWriter),HealthCheckMixin.health,Embedder.{model,dimension},Parser.supports.tests/contract/spi_signature.pyreflects every SPI ABC and asserts the first non-selfparameter isctx: RequestContext. Wired into pytest viapython_filesoverride; fails CI on additions that omitctx.PgVectorStore,QdrantVectorStore,RedisCache(now namespaces keys as<prefix><tenant_id>:<key>),LocalFileStorage/S3Storage(per-tenant key prefix from ctx; path-traversal check scoped to per-tenant root).tests/contract/conftest.pyexposesctx/other_ctxfixtures and amake_ctx()helper; every contract test updated;packages/core/tests/test_types.pycovers all new types includingRequestContextinvariants andBudget.spend(); integration suites (pgvector / qdrant / redis / s3 / local-storage) migrated with a new tenant-isolation assertion where applicable.mypy --strictclean;ruffclean; schemas regenerated underdist/schemas/.rag-core0.2.0 → 0.3.0 andrag_core.__version__0.4.0 → 0.5.0.Documentation
docs/reference/rag-core.md— public type surface with usage examples (gateway boundary, BlobRef hydration, quantized embeddings, planner) plus internals (whyacl_labelsis now typed, whytrust_levellives onChunk) and extension points.docs/architecture/request-context.mdmarked as Status: Implemented in Step 1.1a; Shape table now matches the as-built model (RequestId,created_at, typedPiiPolicy/Budget).docs/README.mdindexes the new reference page.TRACKER.md— Step 1.1a → ✅; Phase 1 done-count 1 → 2; Next action → 1.1b (SPI split / bulk + streaming + ID-only).Test plan
task lint/uv run ruff check . && uv run ruff format --check .— cleantask test/uv run pytest— 508 passed, 33 skipped (integration)uv run mypy packages/ apps/gateway/— Success: no issues in 76 source filesuv run python -m rag_core.gen_schemas dist/schemas/— regenerates 20 schemas including new typesuv run pytest tests/contract/spi_signature.py— linter passes against current SPIsubuntu-22.04,macos-14,windows-latesttask devstack (Postgres + Qdrant + Redis + MinIO) to confirm new tenant-prefix behavior🤖 Generated with Claude Code