feat(crypto): BYOK envelope encryption library + local KMS (Step 6.7a)

[officialCodeWork]

Summary

First slice of Step 6.7 — BYOK (Bring Your Own Key): the envelope-encryption library + a local KMS. New ground — the only prior crypto was HMAC signing.

• What's encrypted: chunk content / blobs at rest — not embedding vectors. ANN search reads vectors directly, so encrypting them would break retrieval; vectors carry no raw text. A deliberate, documented constraint (see ADR-0039).
• KeyManager SPI (rag-core) over opaque bytes (encrypt/decrypt) + NoopKeyManager (passthrough). EncryptingStorage (rag-core, crypto-free) — a Storage decorator that encrypts on put / decrypts on get: the application seam over any backend. EncryptionError + KeyUnavailableError.
• EnvelopeKeyManager base (rag-backends, cryptography) does the data-key half once for every provider: a fresh AES-256-GCM DEK per payload (with ctx.tenant_id bound as AAD), the small DEK wrapped/unwrapped by a subclass via the KEK. LocalKeyManager wraps with an in-process per-tenant KEK; cloud providers (6.7b) subclass and wrap via the KMS API so the KEK never leaves the customer.
• Guarantees (all tested): confidentiality (ciphertext ≠ plaintext), tamper-evidence (GCM tag), per-tenant isolation (KEK-per-tenant + tenant-bound AAD — even a shared KEK can't read another tenant's blob), and sealing (a tenant with no KEK → KeyUnavailableError, others unaffected).
• cryptography lives in rag-backends; rag-core stays crypto-free (SPI + decorator + noop). ragctl kms demonstrates the round-trip + isolation + sealing.

Scope / deferred: library + local KMS only. 6.7b — cloud KMS providers behind [kms-*] extras + wire EncryptingStorage into the ingest/storage path + per-tenant key config (cfg.kms + TenantConfig key ref). 6.7c — zero-downtime key rotation. Vector encryption is out of scope by design.

Tests & gates

• ~19 new tests: KeyManager contract over noop + local (round-trip, empty, health); envelope (tests/kms/test_envelope.py) — non-determinism, tamper → EncryptionError, malformed, cross-tenant blocked, shared-KEK-still-blocked, sealing on encrypt + decrypt, wrong-size KEK, one-tenant-sealed-doesn't-affect-another, EncryptingStorage (stores ciphertext + delegates key ops); ragctl kms smoke.
• Full tests/ packages/ 2347 passed, 1 skipped. Green: ruff, mypy --strict (305 files), RAG001, policy-coverage, log-schema; no dist drift (no new wire types / config / endpoints in 6.7a).

Documentation

• New ADR-0039 — the BYOK decision (content-not-vectors; DEK+KEK envelope; isolation via KEK + AAD; sealing; slicing).
• New docs/reference/encryption.md + docs/architecture/byok.md.
• docs/README.md — ADR-0039 + reference/architecture rows.

Test plan

• uv run pytest tests/ packages/ -m "not integration" — 2347 passed, 1 skipped
• ruff + mypy --strict (305 files) + RAG001 + policy-coverage + log-schema; no dist drift
• Sealing (kill a tenant's KEK → reads fail, others succeed), cross-tenant isolation, tamper detection all verified

🤖 Generated with Claude Code