Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 11 additions & 3 deletions TRACKER.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,12 @@
| **Last updated** | 2026-06-08 |
| **Current phase** | Phase 6 — Governance & Tenancy (**6 / 10 steps**) |
| **Overall** | **70 / 84 steps** — Phases 0–5 complete |
| **Next action** | **Step 6.7bCloud KMS providers + storage wiring**: AWS / GCP / Azure / Vault `EnvelopeKeyManager` subclasses behind `[kms-*]` extras (lazy SDK, wrap/unwrap via KMS) + wire `EncryptingStorage` into the ingest/storage path + per-tenant key config (`cfg.kms` + `TenantConfig` key ref). Completes Step 6.7 with 6.7c (key rotation). 6.7a — the encryption library + local KMS shipped. |
| **Next action** | **Step 6.7cGCP/Azure/Vault KMS + key rotation**: the remaining cloud providers (GCP KMS / Azure Key Vault / HashiCorp Vault, same `EnvelopeKeyManager` pattern, behind `[kms-*]` extras) + zero-downtime key rotation (old keys retained decrypt-only until expiry). Closes Step 6.7. 6.7a (library + local) + 6.7b (config/factory/wiring + AWS KMS) shipped. |

**Recently shipped**

- **6.7a** 🚧 BYOK envelope encryption library — `KeyManager` SPI + `NoopKeyManager` + `EncryptingStorage` (rag-core); `EnvelopeKeyManager` (AES-256-GCM DEK + `tenant_id` AAD) + `LocalKeyManager` (in-process per-tenant KEK) (rag-backends); per-tenant isolation + **sealing** (`KeyUnavailableError`) + tamper detection; `ragctl kms`; first slice of Step 6.7 (cloud KMS + wiring is 6.7b) — [#155](https://github.com/officialCodeWork/AgentContextOS/pull/155)
- **6.7b** 🚧 Cloud KMS (AWS) + config/factory/wiring — `cfg.kms` (provider `noop`/`local`/`aws`) + `tenants[].kms_key_id`; `build_key_manager_from_config` factory + `app.state.key_manager` gateway seam; **`AwsKmsKeyManager`** (aioboto3, per-tenant CMK, KMS-error → sealing, fake-client unit-tested) — [#156](https://github.com/officialCodeWork/AgentContextOS/pull/156)
- **6.7a** 🚧 BYOK envelope encryption library — `KeyManager` SPI + `NoopKeyManager` + `EncryptingStorage` (rag-core); `EnvelopeKeyManager` (AES-256-GCM DEK + `tenant_id` AAD) + `LocalKeyManager` (in-process per-tenant KEK) (rag-backends); per-tenant isolation + **sealing** (`KeyUnavailableError`) + tamper detection; `ragctl kms` — [#155](https://github.com/officialCodeWork/AgentContextOS/pull/155)
- **6.6** ✅ Immutable audit log — **6.6a** shared `AuditWriter`/store on `app.state` + `GET /v1/audit` (tenant-scoped, `chain_verified`) + `GET /v1/audit/verify` (whole-log) + `cfg.audit.enabled` ([#153](https://github.com/officialCodeWork/AgentContextOS/pull/153)); **6.6b** `AuditExporter` self-verifying WORM bundle (SHA-256 `content_hash` + HMAC, offline `verify()`), `POST /v1/audit/export` (tenant-scoped) + `ragctl audit`, `cfg.audit.export_secret` ([#154](https://github.com/officialCodeWork/AgentContextOS/pull/154))
- **6.5** ✅ PII egress policies — `PiiPolicyEngine` (`rag-pii`) answers `egress_text` over the context (`list[Chunk]`) + agent answer (`str`) the gateway already passes, applying the per-tenant `pii_policy` (allow / redact / mask / block); reuses the Step 1.7 detector + rewriters; opt-in `cfg.pii.enabled`; emits `pii.egress_blocked` — [#152](https://github.com/officialCodeWork/AgentContextOS/pull/152)
- **6.4** ✅ ACL egress verifier — `AclEgressVerifier` re-checks returned `ChunkRef`s against the principal's labels at the gateway router boundary (defense-in-depth, same overlap semantics, independent of the PDP); `acl.egress_violation` event; red-team zero-violation-rate gate — [#151](https://github.com/officialCodeWork/AgentContextOS/pull/151)
Expand Down Expand Up @@ -656,7 +657,7 @@
| 6.4 | ACL egress verifier | ✅ | [#151](https://github.com/officialCodeWork/AgentContextOS/pull/151) — `AclEgressVerifier` re-checks returned chunks at the gateway router boundary (defense-in-depth above the 6.3 push-down); `acl.egress_violation`; zero-violation-rate red-team gate |
| 6.5 | PII policies | ✅ | [#152](https://github.com/officialCodeWork/AgentContextOS/pull/152) — `PiiPolicyEngine` egress_text decorator (allow / redact / mask / block per tenant) over answer + context; reuses Step 1.7 detector; `pii.egress_blocked` |
| 6.6 | Immutable audit log | ✅ | **6.6a** [#153](https://github.com/officialCodeWork/AgentContextOS/pull/153) — read API `GET /v1/audit` + `GET /v1/audit/verify` + shared store + `cfg.audit`. **6.6b** [#154](https://github.com/officialCodeWork/AgentContextOS/pull/154) — `AuditExporter` signed WORM bundle + `POST /v1/audit/export` + `ragctl audit` |
| 6.7 | BYOK (Bring Your Own Key) | 🚧 | **6.7a** ✅ [#155](https://github.com/officialCodeWork/AgentContextOS/pull/155) — `KeyManager` SPI + envelope encryption (`LocalKeyManager`) + `EncryptingStorage` + sealing/isolation/tamper + `ragctl kms`. **6.7b** ⏳ — cloud KMS providers + storage wiring + config; **6.7c** ⏳ — key rotation |
| 6.7 | BYOK (Bring Your Own Key) | 🚧 | **6.7a** ✅ [#155](https://github.com/officialCodeWork/AgentContextOS/pull/155) — `KeyManager` SPI + envelope encryption (`LocalKeyManager`) + `EncryptingStorage` + sealing/isolation/tamper. **6.7b** ✅ [#156](https://github.com/officialCodeWork/AgentContextOS/pull/156) — `cfg.kms` + per-tenant key + `build_key_manager_from_config` factory + `AwsKmsKeyManager`. **6.7c** ⏳ — GCP/Azure/Vault + key rotation |
| 6.8 | SSO / SCIM | ⏳ | OIDC + SAML IdP federation; SCIM 2.0 user provisioning; per-tenant IdP config |
| 6.9 | Air-gapped install bundle | ⏳ | Signed tarball with all images + Helm chart; offline bootstrap; cosign verification |
| 6.10 | Compliance posture | ⏳ | SOC 2 Type II control mapping; GDPR data-residency config; data-retention policies |
Expand Down Expand Up @@ -740,6 +741,12 @@ New ground — the only prior crypto was HMAC signing. The V1 plan calls for en
- **Per-tenant isolation** via KEK-per-tenant **and** tenant-bound AAD (even a shared KEK can't read another tenant's blob); **sealing** via `KeyUnavailableError` (a tenant with no KEK is denied, others unaffected); **tamper-evidence** via the GCM auth tag. `cryptography` in `rag-backends`; `rag-core` stays crypto-free (SPI + decorator + noop)
- **Scope:** library + local KMS only. **Deferred:** cloud KMS providers behind `[kms-*]` extras + wiring `EncryptingStorage` into the ingest/storage path + per-tenant key config (6.7b); zero-downtime key rotation (6.7c); vector encryption is out of scope by design. ~19 new tests (KeyManager contract over noop + local; envelope round-trip / non-determinism / tamper / malformed / cross-tenant / shared-KEK-still-blocked / sealing on encrypt+decrypt / wrong-size KEK / per-tenant non-impact / EncryptingStorage stores ciphertext + delegates key ops) + `ragctl kms` smoke; all gates green (ruff, mypy --strict 305 files, RAG001, policy-coverage, log-schema; no dist drift). [ADR-0039](docs/adr/ADR-0039-byok-envelope-encryption.md), [architecture/byok.md](docs/architecture/byok.md), [reference/encryption.md](docs/reference/encryption.md)

#### 6.7b — Cloud KMS (AWS) + config / factory / wiring ✅ [#156](https://github.com/officialCodeWork/AgentContextOS/pull/156)

- Makes BYOK **configurable + provider-selectable**. New `cfg.kms` (`enabled` / `provider` ∈ `noop`/`local`/`aws` / `default_key_id` / `region` / `local_key`) + **`tenants[].kms_key_id`** (per-tenant key reference — a KMS ARN for `aws`, a hex 32-byte KEK for `local`). **`build_key_manager_from_config(cfg)`** factory (in the gateway, lazy `rag-backends` imports so `rag-backends` stays config-free) selects the provider; `build_app(key_manager=…)` exposes **`app.state.key_manager`** + `app.state.kms_enabled` (passthrough `NoopKeyManager` when disabled)
- **`AwsKmsKeyManager`** (`rag-backends/kms/aws.py`) subclasses `EnvelopeKeyManager` and wraps/unwraps the DEK via **AWS KMS** Encrypt/Decrypt for the tenant's customer-managed key (over the already-present `aioboto3` — **no new dep**). Per-tenant CMK from the key-id map (+ `default_key_id`); any KMS failure (revoked / denied / unreachable) or a missing key id → **`KeyUnavailableError`** (sealing). An **injectable `client` seam** makes it fully unit-testable with a fake KMS — no AWS creds / network
- **Scope:** AWS provider + config + factory + the `app.state.key_manager` seam. **Deferred:** GCP KMS / Azure Key Vault / HashiCorp Vault providers + key rotation (6.7c); wiring `EncryptingStorage` into the ingest path (blocked on tiered-storage plumbing — chunk content is inline today). ~14 new tests (AWS provider over a fake KMS: round-trip / KMS-key-bound cross-tenant block / sealing / default-key / KMS-error mapping; config defaults; factory noop/local/aws; gateway `app.state.key_manager` + `kms_enabled` + a local round-trip). `KmsConfig` → `rag.schema` regenerated; all gates green (ruff, mypy --strict 306 files, RAG001, policy-coverage, log-schema). [ADR-0039](docs/adr/ADR-0039-byok-envelope-encryption.md), [reference/encryption.md](docs/reference/encryption.md), [architecture/byok.md](docs/architecture/byok.md)

---

## Phase 7 — Pilot, Harden, GA (Weeks 34–40) ⏳
Expand Down Expand Up @@ -899,6 +906,7 @@ Complete log of every PR. Routine Dependabot bumps are grouped; everything else
| [#153](https://github.com/officialCodeWork/AgentContextOS/pull/153) | 2026-06-08 | feat(gateway): audit read API + chain verification (Step 6.6a) |
| [#154](https://github.com/officialCodeWork/AgentContextOS/pull/154) | 2026-06-08 | feat(audit): WORM signed export — AuditExporter + POST /v1/audit/export (Step 6.6b) |
| [#155](https://github.com/officialCodeWork/AgentContextOS/pull/155) | 2026-06-08 | feat(crypto): BYOK envelope encryption library + local KMS (Step 6.7a) |
| [#156](https://github.com/officialCodeWork/AgentContextOS/pull/156) | 2026-06-08 | feat(crypto): cfg.kms + key-manager factory + AWS KMS provider (Step 6.7b) |
| #78–#80, #116–#118 | Open | Dependabot bumps — awaiting merge |
| #81 | Closed | Dependabot bump — superseded |

Expand Down
11 changes: 11 additions & 0 deletions apps/gateway/src/rag_gateway/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -376,6 +376,7 @@ def build_app(
pii_detector: Any | None = None,
audit_enabled: bool = True,
audit_export_secret: str = "",
key_manager: Any | None = None,
enable_cors: bool = True,
default_tenant_id: TenantId | None = None,
) -> FastAPI:
Expand Down Expand Up @@ -482,6 +483,16 @@ def build_app(
# (empty → unsigned but still content-hashed). The artifact an operator
# archives to immutable storage (S3 Object Lock).
app.state.audit_exporter = AuditExporter(audit_export_secret)
# BYOK key manager (Step 6.7) — exposed for envelope-encrypting blobs at rest
# (wrap a Storage backend with EncryptingStorage). Defaults to the
# passthrough NoopKeyManager; build_app_from_config selects a real provider
# (LocalKeyManager / AwsKmsKeyManager) from cfg.kms.
from rag_core.spi.noop import NoopKeyManager

if key_manager is None:
key_manager = NoopKeyManager()
app.state.key_manager = key_manager
app.state.kms_enabled = not isinstance(key_manager, NoopKeyManager)
# Corpus router (Step 3.5) sits above the retrieval router; with the
# default empty corpus store it degrades to a single unconstrained
# retrieval, preserving the pre-3.5 demo behaviour. It shares the audit
Expand Down
40 changes: 40 additions & 0 deletions apps/gateway/src/rag_gateway/wiring.py
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,41 @@ def build_corpus_store_from_config(cfg: RagConfig) -> CorpusStore:
return NoopCorpusStore(seed=seed)


def build_key_manager_from_config(cfg: RagConfig) -> Any:
"""Build a BYOK ``KeyManager`` from ``cfg.kms`` (Step 6.7).

Returns a passthrough ``NoopKeyManager`` when disabled / ``provider=noop``; a
``LocalKeyManager`` (in-process per-tenant KEKs decoded from
``tenants[].kms_key_id`` hex) for ``local``; an ``AwsKmsKeyManager`` (per-tenant
KMS key ARNs) for ``aws``. GCP / Azure / Vault land in Step 6.7c. Each
tenant's key is ``tenants[].kms_key_id`` with ``kms.default_key_id`` as the
fallback; a tenant with neither is *sealed* at encrypt/decrypt time.
"""
from rag_config.schema import KmsProvider
from rag_core.spi.noop import NoopKeyManager

kms = cfg.kms
if not kms.enabled or kms.provider is KmsProvider.NOOP:
return NoopKeyManager()

if kms.provider is KmsProvider.LOCAL:
from rag_backends import LocalKeyManager

keks = {t.id: bytes.fromhex(t.kms_key_id) for t in cfg.tenants if t.kms_key_id}
default = bytes.fromhex(kms.local_key) if kms.local_key else None
return LocalKeyManager(keks=keks, default_kek=default)

if kms.provider is KmsProvider.AWS:
from rag_backends import AwsKmsKeyManager

key_ids = {t.id: t.kms_key_id for t in cfg.tenants if t.kms_key_id}
return AwsKmsKeyManager(
key_ids=key_ids, default_key_id=kms.default_key_id or None, region=kms.region
)

return NoopKeyManager() # pragma: no cover - providers above are exhaustive


async def seed_corpus_store(store: CorpusStore, cfg: RagConfig) -> int:
"""Initialise + seed a corpus store from ``cfg.corpora``.

Expand Down Expand Up @@ -716,6 +751,10 @@ def build_app_from_config(cfg: RagConfig, **overrides: Any) -> FastAPI:
audit_enabled = overrides.pop("audit_enabled", cfg.audit.enabled)
audit_export_secret = overrides.pop("audit_export_secret", cfg.audit.export_secret)

# BYOK key manager (Step 6.7) — select the provider from cfg.kms (Noop when
# disabled). Operators inject their own via the ``key_manager`` override.
key_manager = overrides.pop("key_manager", None) or build_key_manager_from_config(cfg)

return build_app(
corpus_store=corpus_store,
retrieval_router=retrieval_router,
Expand All @@ -730,6 +769,7 @@ def build_app_from_config(cfg: RagConfig, **overrides: Any) -> FastAPI:
pii_enabled=pii_enabled,
audit_enabled=audit_enabled,
audit_export_secret=audit_export_secret,
key_manager=key_manager,
**overrides,
)

Expand Down
88 changes: 88 additions & 0 deletions apps/gateway/tests/test_kms.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
"""BYOK config + key-manager factory + gateway wiring (Step 6.7b)."""

from __future__ import annotations

from rag_backends import AwsKmsKeyManager, LocalKeyManager
from rag_config import RagConfig
from rag_config.schema import KmsConfig, KmsProvider, TenantConfig
from rag_core.spi.noop import NoopKeyManager
from rag_gateway import build_app
from rag_gateway.wiring import build_app_from_config, build_key_manager_from_config

_HEX_KEK = "ab" * 32 # 32-byte KEK as hex


def _local_cfg() -> RagConfig:
return RagConfig(
kms=KmsConfig(enabled=True, provider=KmsProvider.LOCAL),
tenants=[TenantConfig(id="acme", name="Acme", kms_key_id=_HEX_KEK)],
)


# ---------------------------------------------------------------------------
# config + factory
# ---------------------------------------------------------------------------
def test_kms_config_defaults_off() -> None:
cfg = RagConfig()
assert cfg.kms.enabled is False
assert cfg.kms.provider is KmsProvider.LOCAL


def test_factory_noop_when_disabled() -> None:
assert isinstance(build_key_manager_from_config(RagConfig()), NoopKeyManager)


def test_factory_local() -> None:
km = build_key_manager_from_config(_local_cfg())
assert isinstance(km, LocalKeyManager)


def test_factory_aws() -> None:
cfg = RagConfig(
kms=KmsConfig(enabled=True, provider=KmsProvider.AWS, default_key_id="arn:x"),
tenants=[TenantConfig(id="acme", name="Acme", kms_key_id="arn:aws:kms:::key/acme")],
)
assert isinstance(build_key_manager_from_config(cfg), AwsKmsKeyManager)


# ---------------------------------------------------------------------------
# gateway wiring
# ---------------------------------------------------------------------------
def test_inert_by_default() -> None:
app = build_app()
assert app.state.kms_enabled is False
assert isinstance(app.state.key_manager, NoopKeyManager)


def test_build_from_config_wires_local_key_manager() -> None:
app = build_app_from_config(_local_cfg())
assert app.state.kms_enabled is True
assert isinstance(app.state.key_manager, LocalKeyManager)


def test_build_from_config_disabled_is_noop() -> None:
app = build_app_from_config(RagConfig())
assert app.state.kms_enabled is False
assert isinstance(app.state.key_manager, NoopKeyManager)


async def test_wired_local_key_manager_round_trips() -> None:
from rag_core.types import (
Principal,
PrincipalId,
PrincipalKind,
RequestContext,
TenantId,
)

km = build_app_from_config(_local_cfg()).state.key_manager
tid = TenantId("acme")
ctx = RequestContext(
tenant_id=tid,
principal=Principal(
id=PrincipalId("p"), kind=PrincipalKind.user, display_name="p", tenant_id=tid
),
)
ct = await km.encrypt(ctx, b"secret")
assert ct != b"secret"
assert await km.decrypt(ctx, ct) == b"secret"
57 changes: 57 additions & 0 deletions dist/rag.schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -890,6 +890,48 @@
"title": "KeywordStoreProvider",
"type": "string"
},
"KmsConfig": {
"additionalProperties": false,
"description": "BYOK envelope encryption (Step 6.7).\n\nWhen ``enabled`` the gateway builds a ``KeyManager`` (selected by ``provider``)\nand exposes it on ``app.state.key_manager``; wrap a ``Storage`` backend with\n``EncryptingStorage`` to encrypt chunk content / blobs **at rest** with\nper-tenant, customer-controlled keys (embedding vectors stay plaintext for\nsearch). Each tenant's key reference is ``tenants[].kms_key_id`` (a KMS key\nARN for ``aws``, a hex 32-byte KEK for ``local``); a tenant with no key \u2014\nand no ``default_key_id`` \u2014 is **sealed** (``KeyUnavailableError``).\n\n**Disabled by default.** ``provider``: ``local`` (in-process KEK, dev /\nair-gapped), ``aws`` (AWS KMS), or ``noop`` (passthrough). GCP / Azure /\nVault providers + key rotation land in Step 6.7c.\n\n* ``default_key_id`` \u2014 fallback key reference for tenants without their own.\n* ``region`` \u2014 AWS region for the ``aws`` provider.\n* ``local_key`` \u2014 hex 32-byte default KEK for the ``local`` provider\n (``${ENV_VAR}``-interpolated; keep it out of the file).",
"properties": {
"enabled": {
"default": false,
"title": "Enabled",
"type": "boolean"
},
"provider": {
"$ref": "#/$defs/KmsProvider",
"default": "local"
},
"default_key_id": {
"default": "",
"title": "Default Key Id",
"type": "string"
},
"region": {
"default": "us-east-1",
"title": "Region",
"type": "string"
},
"local_key": {
"default": "",
"title": "Local Key",
"type": "string"
}
},
"title": "KmsConfig",
"type": "object"
},
"KmsProvider": {
"description": "BYOK key-manager provider (Step 6.7).",
"enum": [
"noop",
"local",
"aws"
],
"title": "KmsProvider",
"type": "string"
},
"LLMConfig": {
"additionalProperties": false,
"properties": {
Expand Down Expand Up @@ -1392,6 +1434,18 @@
],
"default": null,
"title": "Dedicated Index Name"
},
"kms_key_id": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"title": "Kms Key Id"
}
},
"required": [
Expand Down Expand Up @@ -1658,6 +1712,9 @@
"audit": {
"$ref": "#/$defs/AuditConfig"
},
"kms": {
"$ref": "#/$defs/KmsConfig"
},
"webhooks": {
"$ref": "#/$defs/WebhooksConfig"
},
Expand Down
Loading
Loading