Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 18 additions & 7 deletions TRACKER.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,12 +14,13 @@
| | |
|---|---|
| **Last updated** | 2026-06-08 |
| **Current phase** | Phase 6Governance & Tenancy (**9 / 10 steps**) |
| **Overall** | **73 / 84 steps** — Phases 0–5 complete |
| **Next action** | **Step 6.10Compliance posture**: SOC 2 Type II control mapping; GDPR data-residency config; data-retention policies. |
| **Current phase** | Phase 7Pilot, Harden, GA (**0 / 10 steps**) — Phase 6 ✅ complete |
| **Overall** | **74 / 84 steps** — Phases 0–6 complete |
| **Next action** | **Step 7.1Load testing**: Locust suite; 1000 QPS sustained; p99 < 500 ms; chaos under load. |

**Recently shipped**

- **6.10** ✅ Compliance posture (Phase 6 capstone) — new `rag-compliance` package: `RetentionEnforcer` drives tenant-scoped `purge_*` (data retention + GDPR right-to-erasure) over the Feedback/Provenance stores (non-abstract `purge_before`/`purge_tenant` SPI capability with `dry_run` so a preview counts-without-deleting; audit chain never purged in place — its retention is the 6.6b WORM export); `POST /v1/compliance/erase` (tenant-scoped, dry-run default + two-flag delete); per-tenant `data_region` enforced at ingest (`ResidencyViolationError` → 403); `GET /v1/status/compliance` posture + `compliance_posture`/`residency_ok`; PII-free `compliance.*` events; `cfg.compliance` + `tenants[].data_region`/`retention_days`; `ragctl compliance report`/`demo`; SOC 2 + GDPR control-mapping docs backed by the live posture — [#162](https://github.com/officialCodeWork/AgentContextOS/pull/162)
- **6.9** ✅ Air-gapped install bundle — `ragctl airgap build`/`inspect`/`verify`/`install`: one signed `.tar.gz` of all runtime images (`docker save`) + the packaged Helm chart + `rag.yaml` + a standalone installer, for networks with no registry / internet. Integrity = a standard `SHA256SUMS` (verifiable with `sha256sum -c`, no cosign/network) pinned as `manifest.content_hash` + an optional cosign signature over it (the 6.6b content-hash + optional-signature pattern); the same `SHA256SUMS` drives the Python verifier and the standalone `install.{sh,ps1}` (need only docker+helm). Typed/tested logic in `ragctl.airgap` (pure core + stubbable docker/helm/cosign seam; `--dry-run` = verifiable bundle minus blobs); digest-pinned manifest-driven image set (`infra/airgap/images.txt`); `task airgap:*`; `release-airgap.yml` on tags — [#161](https://github.com/officialCodeWork/AgentContextOS/pull/161)
- **6.8** ✅ SSO / SCIM — `rag-sso`: `FederatedAuth` (an `Auth` SPI backend over per-tenant `OidcProvider` / `SamlProvider`) federates a bearer token / SAML assertion → `Principal` at the existing `authenticate` seam (group claims → `acl_labels`); dependency-free defaults (stdlib HS256 JWT + `defusedxml` SAML), asymmetric OIDC / XML-DSig behind `[oidc]` / `[saml]` extras; algorithm-allowlist downgrade defense; SCIM 2.0 `ScimService` over the new tenant-scoped `ScimStore` SPI driving `/scim/v2/{Users,Groups}` (per-tenant bearer token); per-tenant IdP on `tenants[].sso`; `cfg.sso` / `cfg.scim`; `GET /v1/status/sso`; PII-free `sso.*` / `scim.*` events; `ragctl sso` / `scim` — [#159](https://github.com/officialCodeWork/AgentContextOS/pull/159)
- **6.7** ✅ BYOK envelope encryption — `KeyManager` SPI + `EncryptingStorage` + `EnvelopeKeyManager` (AES-256-GCM DEK + tenant AAD); `LocalKeyManager` + four cloud KMS providers (`Aws`/`Gcp`/`AzureKeyVault`/`Vault`, behind `[kms-*]` extras); `cfg.kms` + per-tenant key + factory; per-tenant isolation + sealing + tamper-evidence; zero-downtime rotation (`RotatingKeyManager` + `rewrap`); `ragctl kms` — [#155](https://github.com/officialCodeWork/AgentContextOS/pull/155)–[#158](https://github.com/officialCodeWork/AgentContextOS/pull/158)
Expand Down Expand Up @@ -63,9 +64,9 @@
| 3 | Gateway & Agent Runtime | 11 | **11** | 0 |
| 4 | Reliability | 6 | **6** | 0 |
| 5 | Eval & Observability | 7 | **7** | 0 |
| 6 | Governance & Tenancy | 10 | **9** | 1 |
| 6 | Governance & Tenancy | 10 | **10** | 0 |
| 7 | Pilot, Harden, GA | 10 | 0 | 10 |
| **Total** | | **84** | **73** | **11** |
| **Total** | | **84** | **74** | **10** |

---

Expand Down Expand Up @@ -648,7 +649,7 @@
- **Phase-5 close-out:** Step 5.7 ✅ → **Phase 5 complete (7 / 7)**; deferred items remain documented (per-tenant drift / per-dimension embedding PSI; feedback/breaker/quota Grafana export + Loki-events dashboard; gRPC proto mirror of `corpus_decision` + `experiment`; sequential / multi-metric experiments)
- [reference/experiments.md](docs/reference/experiments.md), [reference/admin-ui.md](docs/reference/admin-ui.md)

## Phase 6 — Governance & Tenancy (Weeks 28–34) 🚧 (1 / 10)
## Phase 6 — Governance & Tenancy (Weeks 28–34)

| Step | Title | Status | Planned deliverables |
|------|-------|:------:|----------------------|
Expand All @@ -661,7 +662,7 @@
| 6.7 | BYOK (Bring Your Own Key) | ✅ | **6.7a** ✅ [#155](https://github.com/officialCodeWork/AgentContextOS/pull/155) — `KeyManager` SPI + envelope encryption (`LocalKeyManager`) + `EncryptingStorage` + sealing/isolation/tamper. **6.7b** ✅ [#156](https://github.com/officialCodeWork/AgentContextOS/pull/156) — `cfg.kms` + per-tenant key + `build_key_manager_from_config` factory + `AwsKmsKeyManager`. **6.7c** ✅ [#157](https://github.com/officialCodeWork/AgentContextOS/pull/157) — `GcpKmsKeyManager` / `AzureKeyVaultKeyManager` / `VaultKeyManager` behind `[kms-*]` extras. **6.7d** ✅ [#158](https://github.com/officialCodeWork/AgentContextOS/pull/158) — `RotatingKeyManager` zero-downtime rotation + `rewrap` |
| 6.8 | SSO / SCIM | ✅ | [#159](https://github.com/officialCodeWork/AgentContextOS/pull/159) — `rag-sso`: `FederatedAuth` (`Auth` SPI backend over per-tenant `OidcProvider` / `SamlProvider`) federates a token / assertion → `Principal` at the existing `authenticate` seam (groups → `acl_labels`); stdlib HS256 + `defusedxml` defaults, asymmetric OIDC / XML-DSig behind `[oidc]` / `[saml]` extras; SCIM 2.0 `ScimService` + `ScimStore` SPI → `/scim/v2/*` (per-tenant bearer); `tenants[].sso` + `cfg.sso` / `cfg.scim`; `GET /v1/status/sso`; `ragctl sso` / `scim` |
| 6.9 | Air-gapped install bundle | ✅ | [#161](https://github.com/officialCodeWork/AgentContextOS/pull/161) — `ragctl airgap build`/`verify`/`install`: signed `.tar.gz` of all images (`docker save`) + Helm chart + config + standalone installer; `SHA256SUMS` content-hash (offline-verifiable) + optional cosign; `infra/airgap/images.txt`; `install.{sh,ps1}`; `task airgap:*`; `release-airgap.yml` |
| 6.10 | Compliance posture | | SOC 2 Type II control mapping; GDPR data-residency config; data-retention policies |
| 6.10 | Compliance posture | | [#162](https://github.com/officialCodeWork/AgentContextOS/pull/162) — `rag-compliance`: `RetentionEnforcer` (data retention + GDPR erasure via `purge_*` SPI capability, `dry_run`) + `POST /v1/compliance/erase`; per-tenant `data_region` enforced at ingest (403); `GET /v1/status/compliance` posture; `cfg.compliance`; PII-free `compliance.*`; `ragctl compliance`; SOC 2 + GDPR mapping docs |

### 6.1 — Logical multi-tenancy ✅ [#148](https://github.com/officialCodeWork/AgentContextOS/pull/148)

Expand Down Expand Up @@ -780,6 +781,16 @@ New ground — the only prior crypto was HMAC signing. The V1 plan calls for en
- **Digest-pinned, manifest-driven image set:** `infra/airgap/images.txt` lists the third-party images (tags); the build resolves each to its digest into `manifest.json`. The gateway image is added from the chart's `image.repository` + `--version`, so the app image is always release-versioned. **Key-based cosign** is the air-gap recommendation (keyless needs Rekor + a deployment-specific identity policy); the release path (`release-airgap.yml` on `v*` tags) signs keyless and attaches the tarball to the GitHub Release
- **Scope:** docker-based image load + the gateway chart. **Deferred:** `ctr`/podman load, registry re-tag/push, multi-arch selection, bundling the *backend* charts (chart deploys the gateway; backends stay external), TUF-rooted offline keyless verify. `task airgap:build` / `build-dry` / `verify`; ~17 tests (manifest / hashing / tamper detection / tarball round-trip / signature paths via stubbed cosign / CLI on a dry-run bundle), shell installer syntax-checked + exercised via `--verify-only`. All gates green (ruff, mypy --strict, RAG001, schema/openapi/config-drift, policy-coverage, log-schema). [ADR-0041](docs/adr/ADR-0041-airgap-bundle.md), [reference/airgap.md](docs/reference/airgap.md), [architecture/airgap-bundle.md](docs/architecture/airgap-bundle.md), [guides/airgap-install.md](docs/guides/airgap-install.md)

### 6.10 — Compliance posture ✅ [#162](https://github.com/officialCodeWork/AgentContextOS/pull/162)

- **Phase 6 capstone.** The platform already shipped the *controls* an auditor maps to (audit / ACL / PII / BYOK / SSO / quotas / breakers / guard); 6.10 adds the three missing pieces — **data retention**, **GDPR erasure / residency**, and a **machine-readable posture** that backs the SOC 2 / GDPR mapping
- **New `rag-compliance` package** (deps rag-core + rag-observability only, config-free — like rag-feedback / rag-drift): **`RetentionEnforcer`** drives tenant-scoped `purge_*` (`purge` = per-class windows, `erase_tenant` = GDPR right-to-erasure); **`compliance_posture`** / **`residency_ok`** are pure
- **Retention is a *capability on the existing stores*, not a new SPI**: non-abstract **`purge_before(ctx, cutoff, *, dry_run)`** + **`purge_tenant(ctx, *, dry_run)`** on `FeedbackStore` + `ProvenanceStore` (default no-op → no backend breaks). `dry_run` lives in the SPI so a preview **counts without deleting** uniformly (`ProvenanceStore` has no `list`). The **audit chain is never purged in place** (it would break tamper-evidence) — audit retention is the Step 6.6b WORM export; `RetentionPolicy.audit_days` is advisory
- **Right-to-erasure** — `POST /v1/compliance/erase` erases the **calling tenant's** data (scope from the principal, never the body); always available (a tenant can always erase its own data), **dry-run by default**, real delete needs `dry_run=false` **and** `confirm=true`
- **Data residency** — `tenants[].data_region` resolves onto `TenantSettings` (reuses the 6.1 resolver) and is enforced at ingest against `cfg.compliance.region`; a mismatch raises **`ResidencyViolationError`** (→ 403) + `compliance.residency_violation`. Opt-in (`cfg.compliance.enabled`), lenient when no region declared — a single-deployment assertion + guard, not multi-region routing
- **The SOC 2 / GDPR mapping is a doc *backed by a live posture*** — **`GET /v1/status/compliance`** serves a `CompliancePosture` (controls + region + retention) computed from the build_app flags; the mapping docs map each Trust Service Criterion / GDPR article to a control, and the posture reports whether it is **on in this deployment** (so the mapping is checkable, not just prose)
- New core types **`DataClass` / `RetentionPolicy` / `ErasureResult` / `CompliancePosture`** + **`ComplianceError`** (400) / **`ResidencyViolationError`** (403); `ComplianceEraseRequest` wire type; `cfg.compliance` + `tenants[].data_region` / `retention_days` (`dist/rag.schema`); PII-free **`compliance.*`** events; **`ragctl compliance report`** (posture from cfg) + **`compliance demo`**. **Scope:** tenant-level erasure + feedback/provenance retention. **Deferred:** subject-level chunk/document erasure (needs an index delete-by-document path), an admin retention-*sweep* endpoint (today the sweep is `ragctl`/cron; erasure is the API), multi-region routing, automated audit-evidence bundles. ~30 tests (enforcer purge/erase/dry-run + posture + residency unit; `purge_*` contract suite; gateway posture + erase + residency-403; ragctl). All gates green (ruff, mypy --strict, RAG001, schema/openapi/config-drift, policy-coverage, log-schema/event-registry/PII). [ADR-0042](docs/adr/ADR-0042-compliance-posture.md), [reference/compliance.md](docs/reference/compliance.md), [architecture/compliance.md](docs/architecture/compliance.md), [compliance/soc2-control-mapping.md](docs/compliance/soc2-control-mapping.md), [compliance/gdpr-mapping.md](docs/compliance/gdpr-mapping.md)

---

## Phase 7 — Pilot, Harden, GA (Weeks 34–40) ⏳
Expand Down
2 changes: 2 additions & 0 deletions apps/gateway/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ dependencies = [
"rag-agent",
"rag-webhooks",
"rag-sso",
"rag-compliance",
"fastapi>=0.115",
"uvicorn[standard]>=0.30",
"python-multipart>=0.0.9",
Expand Down Expand Up @@ -80,6 +81,7 @@ rag-drift = { workspace = true }
rag-agent = { workspace = true }
rag-webhooks = { workspace = true }
rag-sso = { workspace = true }
rag-compliance = { workspace = true }

[tool.hatch.build.targets.wheel]
packages = ["src/rag_gateway"]
89 changes: 89 additions & 0 deletions apps/gateway/src/rag_gateway/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@

from rag_gateway.agent import build_default_agent_loop, make_agent_router
from rag_gateway.audit import make_audit_router
from rag_gateway.compliance import make_compliance_router
from rag_gateway.corpora import make_corpora_router
from rag_gateway.feedback import make_feedback_router
from rag_gateway.middleware import install_request_context_middleware
Expand All @@ -137,6 +138,43 @@
_log = get_logger(__name__)


def _enforce_residency(state: Any, tenant_id: str) -> None:
"""Refuse ingest when a tenant's required region != the deployment region (Step 6.10).

No-op unless ``cfg.compliance.enabled``. Resolves the tenant's ``data_region``
via the Step 6.1 resolver and compares against ``cfg.compliance.region``;
a mismatch raises :class:`~rag_core.errors.ResidencyViolationError` (HTTP 403)
and emits a PII-free ``compliance.residency_violation`` event.
"""
if not getattr(state, "compliance_enabled", False):
return
resolver = getattr(state, "tenant_resolver", None)
if resolver is None:
return
from rag_compliance import residency_ok
from rag_core.errors import ResidencyViolationError
from rag_observability.events import EVT_COMPLIANCE_RESIDENCY_VIOLATION, ComplianceEvent

region = getattr(state, "compliance_region", "")
tenant_region = resolver.resolve(str(tenant_id)).data_region
if residency_ok(tenant_region, region):
return
_log.warning(
ComplianceEvent(
event_name=EVT_COMPLIANCE_RESIDENCY_VIOLATION,
tenant_id=str(tenant_id),
operation="residency_violation",
region=region,
tenant_region=tenant_region or "",
).model_dump_json()
)
raise ResidencyViolationError(
"tenant data region does not match this deployment's region",
tenant_region=tenant_region,
deployment_region=region,
)


# ---------------------------------------------------------------------------
# Default wiring — dev / smoke; production replaces this from rag.yaml in 3.5
# ---------------------------------------------------------------------------
Expand Down Expand Up @@ -382,6 +420,11 @@ def build_app(
scim_enabled: bool = False,
scim_tokens: dict[str, str] | None = None,
sso_enabled: bool = False,
compliance_enabled: bool = False,
compliance_region: str = "",
retention_policy: Any | None = None,
retention_enforcer: Any | None = None,
compliance_posture: Any | None = None,
enable_cors: bool = True,
default_tenant_id: TenantId | None = None,
) -> FastAPI:
Expand Down Expand Up @@ -441,6 +484,13 @@ def build_app(
"create / update / deactivate (per-tenant bearer token)."
),
},
{
"name": "compliance",
"description": (
"Compliance posture — GDPR right-to-erasure of the calling "
"tenant's data (data retention + residency configure via rag.yaml)."
),
},
],
)

Expand Down Expand Up @@ -635,6 +685,40 @@ def build_app(
app.state.feedback_recorder = feedback_recorder
app.state.feedback_store = feedback_store

# Compliance posture (Step 6.10) — the RetentionEnforcer (GDPR right-to-erasure
# via POST /v1/compliance/erase + the operator-run retention sweep) over the
# wired feedback / provenance stores; the deployment data region for the ingest
# residency guard; and the live posture for GET /v1/status/compliance (the
# SOC 2 / GDPR control mapping). ``compliance_enabled`` gates residency
# enforcement; build_app_from_config builds the posture from cfg.compliance.
from rag_compliance import RetentionEnforcer
from rag_compliance import compliance_posture as _build_posture
from rag_core.types import RetentionPolicy

app.state.compliance_enabled = compliance_enabled
app.state.compliance_region = compliance_region
app.state.retention_enforcer = retention_enforcer or RetentionEnforcer(
feedback_store=app.state.feedback_store,
provenance_store=app.state.provenance_store,
)
app.state.compliance_posture = compliance_posture or _build_posture(
region=compliance_region,
controls={
"audit": audit_enabled,
"acl": acl_enabled,
"pii": pii_enabled,
"byok": app.state.kms_enabled,
"sso": sso_enabled,
"scim": scim_enabled,
"quotas": quota_enforcer is not None,
"guard": guard_enabled,
"breakers": breaker_registry is not None,
"retention": compliance_enabled,
"residency": compliance_enabled and bool(compliance_region),
},
retention=retention_policy or RetentionPolicy(),
)

# Drift monitors (Step 5.5) — when a registry is wired, the query path feeds
# the PSI monitors (query length / retrieval score / embedding norm), the
# feedback path feeds citation-clickthrough, and ``GET /v1/status/drift``
Expand Down Expand Up @@ -796,6 +880,10 @@ async def ingest_document(
),
)

# GDPR data-residency (Step 6.10) — refuse ingest when the tenant's required
# region does not match this deployment's region (no-op unless enabled).
_enforce_residency(request.app.state, tenant_id)

# Storage quota (Step 4.5) — admit-or-reject the upload against the
# tenant's storage-bytes gauge before the pipeline runs. A successful
# admission advances the gauge by the upload size.
Expand Down Expand Up @@ -847,6 +935,7 @@ async def ingest_document(
app.include_router(make_status_router())
app.include_router(make_audit_router())
app.include_router(make_scim_router())
app.include_router(make_compliance_router())

return app

Expand Down
Loading
Loading