feat(crypto): BYOK envelope encryption library + local KMS (Step 6.7a)

TRACKER.md

@@ -16,10 +16,11 @@
 | **Last updated** | 2026-06-08 |
 | **Current phase** | Phase 6 — Governance & Tenancy (**6 / 10 steps**) |
 | **Overall** | **70 / 84 steps** — Phases 0–5 complete |
-| **Next action** | **Step 6.7 — BYOK (Bring Your Own Key)**: KMS integration (AWS KMS / GCP KMS / HashiCorp Vault) + envelope encryption for embeddings; per-tenant keys. New ground (no existing crypto seam beyond the HMAC signers from 5.1 / 6.6b). |
+| **Next action** | **Step 6.7b — Cloud KMS providers + storage wiring**: AWS / GCP / Azure / Vault `EnvelopeKeyManager` subclasses behind `[kms-*]` extras (lazy SDK, wrap/unwrap via KMS) + wire `EncryptingStorage` into the ingest/storage path + per-tenant key config (`cfg.kms` + `TenantConfig` key ref). Completes Step 6.7 with 6.7c (key rotation). 6.7a — the encryption library + local KMS — shipped. |
 
 **Recently shipped**
 
+- **6.7a** 🚧 BYOK envelope encryption library — `KeyManager` SPI + `NoopKeyManager` + `EncryptingStorage` (rag-core); `EnvelopeKeyManager` (AES-256-GCM DEK + `tenant_id` AAD) + `LocalKeyManager` (in-process per-tenant KEK) (rag-backends); per-tenant isolation + **sealing** (`KeyUnavailableError`) + tamper detection; `ragctl kms`; first slice of Step 6.7 (cloud KMS + wiring is 6.7b) — [#155](https://github.com/officialCodeWork/AgentContextOS/pull/155)
 - **6.6** ✅ Immutable audit log — **6.6a** shared `AuditWriter`/store on `app.state` + `GET /v1/audit` (tenant-scoped, `chain_verified`) + `GET /v1/audit/verify` (whole-log) + `cfg.audit.enabled` ([#153](https://github.com/officialCodeWork/AgentContextOS/pull/153)); **6.6b** `AuditExporter` self-verifying WORM bundle (SHA-256 `content_hash` + HMAC, offline `verify()`), `POST /v1/audit/export` (tenant-scoped) + `ragctl audit`, `cfg.audit.export_secret` ([#154](https://github.com/officialCodeWork/AgentContextOS/pull/154))
 - **6.5** ✅ PII egress policies — `PiiPolicyEngine` (`rag-pii`) answers `egress_text` over the context (`list[Chunk]`) + agent answer (`str`) the gateway already passes, applying the per-tenant `pii_policy` (allow / redact / mask / block); reuses the Step 1.7 detector + rewriters; opt-in `cfg.pii.enabled`; emits `pii.egress_blocked` — [#152](https://github.com/officialCodeWork/AgentContextOS/pull/152)
 - **6.4** ✅ ACL egress verifier — `AclEgressVerifier` re-checks returned `ChunkRef`s against the principal's labels at the gateway router boundary (defense-in-depth, same overlap semantics, independent of the PDP); `acl.egress_violation` event; red-team zero-violation-rate gate — [#151](https://github.com/officialCodeWork/AgentContextOS/pull/151)
@@ -655,7 +656,7 @@
 | 6.4 | ACL egress verifier | ✅ | [#151](https://github.com/officialCodeWork/AgentContextOS/pull/151) — `AclEgressVerifier` re-checks returned chunks at the gateway router boundary (defense-in-depth above the 6.3 push-down); `acl.egress_violation`; zero-violation-rate red-team gate |
 | 6.5 | PII policies | ✅ | [#152](https://github.com/officialCodeWork/AgentContextOS/pull/152) — `PiiPolicyEngine` egress_text decorator (allow / redact / mask / block per tenant) over answer + context; reuses Step 1.7 detector; `pii.egress_blocked` |
 | 6.6 | Immutable audit log | ✅ | **6.6a** [#153](https://github.com/officialCodeWork/AgentContextOS/pull/153) — read API `GET /v1/audit` + `GET /v1/audit/verify` + shared store + `cfg.audit`.  **6.6b** [#154](https://github.com/officialCodeWork/AgentContextOS/pull/154) — `AuditExporter` signed WORM bundle + `POST /v1/audit/export` + `ragctl audit` |
-| 6.7 | BYOK (Bring Your Own Key) | ⏳ | KMS integration (AWS KMS, GCP KMS, HashiCorp Vault); envelope encryption for embeddings |
+| 6.7 | BYOK (Bring Your Own Key) | 🚧 | **6.7a** ✅ [#155](https://github.com/officialCodeWork/AgentContextOS/pull/155) — `KeyManager` SPI + envelope encryption (`LocalKeyManager`) + `EncryptingStorage` + sealing/isolation/tamper + `ragctl kms`.  **6.7b** ⏳ — cloud KMS providers + storage wiring + config; **6.7c** ⏳ — key rotation |
 | 6.8 | SSO / SCIM | ⏳ | OIDC + SAML IdP federation; SCIM 2.0 user provisioning; per-tenant IdP config |
 | 6.9 | Air-gapped install bundle | ⏳ | Signed tarball with all images + Helm chart; offline bootstrap; cosign verification |
 | 6.10 | Compliance posture | ⏳ | SOC 2 Type II control mapping; GDPR data-residency config; data-retention policies |
@@ -727,6 +728,18 @@ Step 0.7c shipped the foundation — an append-only `AuditStore` SPI, a `NoopAud
 - New `AuditExport` / `AuditExportSignature` / `AuditExportVerification` core types (`dist/schemas` regenerated); `POST /v1/audit/export` → `dist/openapi`; **`ragctl audit`** (seed → export → verify round-trip, `--out` writes the bundle, `--verify FILE` checks one offline).  ~23 new tests (exporter unit: content_hash determinism, sign→verify, unsigned, no_secret, wrong-secret, **content tamper → content_mismatch**, **signature tamper → signature_mismatch**, empty; gateway: tenant-scoped + signed + **offline-verifies** + cross-tenant isolation + unsigned + disabled→404 + no-auth→401; ragctl round-trip + file verify).  All gates green (ruff, mypy --strict 299 files, RAG001, schema/openapi-drift, policy-coverage, log-schema).  [ADR-0038](docs/adr/ADR-0038-immutable-audit-log.md), [architecture/audit-log.md](docs/architecture/audit-log.md), [reference/audit.md](docs/reference/audit.md)
 - **Deferred:** a durable live-store `AuditStore` backend (Postgres / append-only ledger) and expanding what gets audited beyond `corpus.route`.
 
+### 6.7 — BYOK (Bring Your Own Key) 🚧 (sliced 6.7a + 6.7b + 6.7c)
+
+New ground — the only prior crypto was HMAC signing.  The V1 plan calls for envelope encryption with per-tenant data keys wrapped by a customer-controlled KMS KEK, zero-downtime rotation, and **sealing** (KEK unavailable → tenant data unreadable).  Delivered in slices.
+
+#### 6.7a — Envelope encryption library + local KMS ✅ [#155](https://github.com/officialCodeWork/AgentContextOS/pull/155)
+
+- **What's encrypted:** **chunk content / blobs at rest** — *not* embedding vectors (ANN search reads vectors directly, so encrypting them would break retrieval; vectors carry no raw text).  A deliberate, documented constraint
+- **`KeyManager` SPI** (`rag-core`) over opaque bytes — `encrypt(ctx, plaintext) → bytes` / `decrypt(ctx, ciphertext) → bytes`; ciphertext is a self-describing envelope.  `NoopKeyManager` (passthrough) for wiring tests.  **`EncryptingStorage`** (`rag-core`, crypto-free) is a `Storage` decorator that encrypts on `put` / decrypts on `get` — the application seam over any backend.  `EncryptionError` + `KeyUnavailableError` (sealing)
+- **`EnvelopeKeyManager`** base (`rag-backends`, `cryptography`) does the data-key half once for every provider: a fresh AES-256-GCM **DEK** per payload (with `ctx.tenant_id` bound as **AAD**), the small DEK handed to a subclass to wrap/unwrap with the **KEK**.  **`LocalKeyManager`** wraps with an in-process per-tenant KEK (dev / tests / air-gapped); cloud providers (6.7b) subclass and wrap/unwrap via the KMS API so the KEK never leaves the customer
+- **Per-tenant isolation** via KEK-per-tenant **and** tenant-bound AAD (even a shared KEK can't read another tenant's blob); **sealing** via `KeyUnavailableError` (a tenant with no KEK is denied, others unaffected); **tamper-evidence** via the GCM auth tag.  `cryptography` in `rag-backends`; `rag-core` stays crypto-free (SPI + decorator + noop)
+- **Scope:** library + local KMS only.  **Deferred:** cloud KMS providers behind `[kms-*]` extras + wiring `EncryptingStorage` into the ingest/storage path + per-tenant key config (6.7b); zero-downtime key rotation (6.7c); vector encryption is out of scope by design.  ~19 new tests (KeyManager contract over noop + local; envelope round-trip / non-determinism / tamper / malformed / cross-tenant / shared-KEK-still-blocked / sealing on encrypt+decrypt / wrong-size KEK / per-tenant non-impact / EncryptingStorage stores ciphertext + delegates key ops) + `ragctl kms` smoke; all gates green (ruff, mypy --strict 305 files, RAG001, policy-coverage, log-schema; no dist drift).  [ADR-0039](docs/adr/ADR-0039-byok-envelope-encryption.md), [architecture/byok.md](docs/architecture/byok.md), [reference/encryption.md](docs/reference/encryption.md)
+
 ---
 
 ## Phase 7 — Pilot, Harden, GA (Weeks 34–40) ⏳
@@ -885,6 +898,7 @@ Complete log of every PR. Routine Dependabot bumps are grouped; everything else
 | [#152](https://github.com/officialCodeWork/AgentContextOS/pull/152) | 2026-06-08 | feat(pii): PII egress policies — PiiPolicyEngine egress_text decorator (Step 6.5) |
 | [#153](https://github.com/officialCodeWork/AgentContextOS/pull/153) | 2026-06-08 | feat(gateway): audit read API + chain verification (Step 6.6a) |
 | [#154](https://github.com/officialCodeWork/AgentContextOS/pull/154) | 2026-06-08 | feat(audit): WORM signed export — AuditExporter + POST /v1/audit/export (Step 6.6b) |
+| [#155](https://github.com/officialCodeWork/AgentContextOS/pull/155) | 2026-06-08 | feat(crypto): BYOK envelope encryption library + local KMS (Step 6.7a) |
 | #78–#80, #116–#118 | Open | Dependabot bumps — awaiting merge |
 | #81 | Closed | Dependabot bump — superseded |
 

docs/README.md

@@ -10,6 +10,7 @@
 | [multi-tenancy.md](architecture/multi-tenancy.md) | Logical multi-tenancy (Step 6.1): make per-tenant `rag.yaml` config drive requests. `TenantResolver` (`rag_config.tenancy`) maps a tenant id → frozen `TenantSettings` (namespace / pii_policy / acl_labels), applied **once** at the gateway boundary (namespace + pii_policy onto the `RequestContext`, acl_labels unioned into the principal); unknown tenants resolve to safe defaults (namespace = id, default PII, no labels — isolated not privileged); `RequestContext.namespace` defaults to `tenant_id` (Pinecone partitions on it; `filter_pushdown` unchanged); resolves+threads only — ACL push-down is 6.3, PII egress 6.5, physical tenancy 6.2; `GET /v1/status/tenant`; inert in `build_app` |
 | [policy-engine.md](architecture/policy-engine.md) | `PolicyEngine` (PDP) — single decision point for ACL, PII, quotas, redaction; replaces scattered governance checks |
 | [audit-log.md](architecture/audit-log.md) | Immutable audit log (Step 6.6): tamper-evidence (SHA-256 hash chain) vs immutability-at-rest (WORM export, 6.6b); one shared `AuditWriter`/store on `app.state`; read-path tenant scoping (tenant-scoped list vs whole-log verify); why the read API defaults on |
+| [byok.md](architecture/byok.md) | BYOK / envelope encryption (Step 6.7): what's encrypted (chunk content at rest) vs not (embedding vectors — search needs plaintext); DEK+KEK envelope (client-side AES-GCM DEK, provider wraps the DEK); per-tenant isolation via KEK + `tenant_id` AAD; sealing as a typed error; rag-core/rag-backends split; slicing (6.7a library, 6.7b cloud KMS + wiring, 6.7c rotation) |
 | [caching.md](architecture/caching.md) | Three-cache split: `EmbeddingCache`, `RetrievalCache`, `AnswerCache` — distinct invalidation rules |
 | [performance.md](architecture/performance.md) | Hot-path discipline, per-SPI p99 budgets, async telemetry, reviewer checklist |
 | [pipeline-batcher.md](architecture/pipeline-batcher.md) | `Pipeline` (async DAG, bounded queues, per-stage workers) + `Batcher` (DataLoader-pattern coalescing) primitives — Step 1.1d |
@@ -106,6 +107,7 @@
 | [admin-ui.md](reference/admin-ui.md) | Admin console (Step 3.10) — Next.js 14 operator GUI (`apps/admin-ui`); 9 pages (dashboard, corpora, connectors, glossary, webhooks, audit, API keys, tenants, config), live-vs-seed hybrid + `NEXT_PUBLIC_GATEWAY_URL`, header identity, running it, internals (shell/primitives/data layer), extension points |
 | [tenancy.md](reference/tenancy.md) | Logical multi-tenancy (Step 6.1) — per-tenant `rag.yaml` config (`namespace` / `acl_labels` / `pii_policy` / `quota`); `TenantResolver.resolve(id) → TenantSettings`; `RequestContext.namespace`; `GET /v1/status/tenant`; `ragctl tenant list` / `resolve`; config table + scope/boundaries (6.2/6.3/6.5) + extension points; physical tenancy (6.2), ACL push-down (6.3) + egress verifier (6.4 — `cfg.acl.verify_egress`) sections |
 | [audit.md](reference/audit.md) | Audit log (Step 6.6) — `AuditEvent` / `AuditStore` (append / events / verify_chain) / `NoopAuditStore` SHA-256 hash chain / `AuditWriter` (+ `.store`); read API `GET /v1/audit` (tenant-scoped, `chain_verified`) + `GET /v1/audit/verify` (whole-log); WORM signed export (6.6b) — `AuditExporter` (content_hash + HMAC), `POST /v1/audit/export`, offline `verify()`, `ragctl audit`; `cfg.audit.enabled` / `export_secret`; durable-store extension points |
+| [encryption.md](reference/encryption.md) | BYOK envelope encryption (Step 6.7a) — `KeyManager` SPI (encrypt/decrypt opaque bytes) + `NoopKeyManager`; `EnvelopeKeyManager` (AES-256-GCM DEK + `tenant_id` AAD) + `LocalKeyManager` (in-process per-tenant KEK); `EncryptingStorage` decorator; `EncryptionError` / `KeyUnavailableError` (sealing); guarantees table (confidentiality / tamper / isolation / sealing); `ragctl kms`; cloud-provider extension points (6.7b) |
 | [webhooks.md](reference/webhooks.md) | Outbound webhooks (Step 3.9) — event catalogue (`ingest.completed` / `audit.policy_violation` / `drift.detected` / `eval.regression`), event envelope, HMAC signing + `verify()`, at-least-once delivery, `/v1/webhooks/subscriptions` CRUD + test, `rag.yaml` block, `ragctl webhooks demo`, internals + extension points |
 | [integrations.md](reference/integrations.md) | Framework adapters (Step 3.8) — `agentcontextos.integrations.*` for LangChain / LlamaIndex / Haystack / DSPy / LangGraph / CrewAI / AutoGen / Semantic Kernel; per-framework extras, shared config + chunk metadata, usage per framework, internals + extension points |
 | [status-api.md](reference/status-api.md) | Status & Metrics API (Step 3.11) — `/v1/status/health` / `metrics` / `logs` (+ SSE `logs/stream`), `WS /v1/status/ws`, `/v1/connectors/status`; metric catalogue + request-timing middleware, the `MetricsCollector` / `LogTail` read-side, CORS + query-param identity for browser streams, extension points |

docs/adr/ADR-0039-byok-envelope-encryption.md

@@ -0,0 +1,84 @@
+# ADR-0039 — BYOK envelope encryption
+
+**Status:** Accepted
+**Date:** 2026-06-08
+**Step:** 6.7 — BYOK (Bring Your Own Key) (Phase 6 — Governance & Tenancy)
+**Related:** [ADR-0007](ADR-0007-tiered-storage.md) (BlobRef / tiered storage), [ADR-0026](ADR-0026-per-query-tracing-provenance.md) (HMAC signing), [architecture/byok.md](../architecture/byok.md), [reference/encryption.md](../reference/encryption.md)
+
+## Context
+
+The V1 plan (Step 6.7) calls for **envelope encryption** with **per-tenant data
+keys** wrapped by a **customer-controlled KEK** in a KMS (AWS KMS / GCP KMS /
+Azure Key Vault / HashiCorp Vault — "Bring Your Own Key"), plus zero-downtime key
+rotation and **sealing** (if the KEK becomes unavailable, that tenant's data
+becomes unreadable, with no cross-tenant impact). This is new ground — the only
+crypto in the codebase was HMAC signing (provenance / webhooks / audit export).
+
+This ADR is delivered in slices: **6.7a** (this) — the encryption library + a
+local KMS; **6.7b** — cloud KMS providers + wiring the encryption into the storage
+path + per-tenant key config; **6.7c** — key rotation.
+
+## Decision
+
+**1. Encrypt content at rest, not embedding vectors.** Embedding *vectors* must
+stay plaintext — approximate-nearest-neighbour search reads them directly, so
+encrypting them would break retrieval (and homomorphic search is out of scope).
+The encryption target is the **sensitive chunk content / blobs at rest** (the
+text), stored behind the `Storage` SPI. Vectors carry no raw text; the text is
+what BYOK protects.
+
+**2. A `KeyManager` SPI over opaque bytes.** `encrypt(ctx, plaintext) -> bytes` /
+`decrypt(ctx, ciphertext) -> bytes`. The ciphertext is a self-describing envelope;
+callers treat it as opaque and store it wherever they'd store the plaintext. Keeps
+the SPI tiny and provider-agnostic. `NoopKeyManager` (passthrough, no crypto) is
+the test/dev wiring stub.
+
+**3. Envelope encryption in a shared base.** `EnvelopeKeyManager` (rag-backends)
+does the data-key half once for every provider: a fresh AES-256-GCM **DEK** per
+payload encrypts the plaintext (with `ctx.tenant_id` bound in as **AAD**, so a
+ciphertext can't be replayed under another tenant), and the small DEK is handed to
+a subclass to **wrap / unwrap** with the **KEK**. Subclasses implement only
+`_wrap_dek` / `_unwrap_dek`: `LocalKeyManager` wraps with an in-process KEK (6.7a);
+cloud providers wrap via the KMS API (6.7b) — so the KEK never leaves the
+customer's control.
+
+**4. `EncryptingStorage` is the application seam.** A crypto-free `Storage`
+decorator (rag-core) that runs blobs through the `KeyManager` on `put` / `get`. It
+composes with any storage backend and any provider, so wiring BYOK = wrapping the
+production `Storage` when enabled.
+
+**5. Per-tenant isolation + sealing are first-class.** The KEK is resolved from
+`ctx.tenant_id`; a tenant with no KEK is **sealed** — `encrypt` / `decrypt` raise
+`KeyUnavailableError` (a typed error, distinct from `EncryptionError` for
+tampering), with no cross-tenant impact. The tenant-bound AAD means even a shared
+KEK can't read another tenant's blob.
+
+**6. `cryptography` in rag-backends.** AES-GCM comes from the well-maintained
+`cryptography` library (don't roll your own crypto), declared in rag-backends
+where the real provider impls live. rag-core stays crypto-free (SPI + decorator +
+`NoopKeyManager` only).
+
+## Consequences
+
+**Positive**
+- BYOK control: a customer holds the KEK; revoking it seals their data
+  (demonstrated by the sealing test) — independent of platform operators.
+- Tamper-evident (AES-GCM auth tag) + tenant-bound (AAD), verified by tests.
+- The encryption library is complete + tested now; wiring + cloud providers layer
+  on without changing the SPI.
+
+**Negative / deferred**
+- **Embedding vectors are not encrypted** — a deliberate constraint (search needs
+  plaintext vectors). Deployments needing vector confidentiality rely on
+  infrastructure-level disk encryption for the vector store.
+- Data is decrypted server-side for use, so plaintext exists in memory during a
+  request — BYOK protects data **at rest** + gates access via the KEK, not against
+  a compromised running server.
+- **6.7b** (cloud KMS providers behind `[kms-*]` extras + wiring `EncryptingStorage`
+  into the ingest/storage path + per-tenant key config) and **6.7c** (zero-downtime
+  rotation, decrypt-only retention of old keys) are deferred.
+
+## See also
+- [architecture/byok.md](../architecture/byok.md) — envelope design, AAD, sealing, slicing
+- [reference/encryption.md](../reference/encryption.md) — `KeyManager` / `LocalKeyManager` / `EncryptingStorage`
+- [ADR-0007](ADR-0007-tiered-storage.md) — the `BlobRef` / `Storage` path this encrypts