diff --git a/docs/designs/DECISIONS.md b/docs/designs/DECISIONS.md index 8e636f35..1c0737c3 100644 --- a/docs/designs/DECISIONS.md +++ b/docs/designs/DECISIONS.md @@ -113,6 +113,7 @@ check enforces the mechanical half. Full rationale: | DL-315 | The cross-tenant background/system loops (delivery-cursor sweep, deliver-ack advance, reattach recovery, lag-resync) run under a narrowly-scoped `BYPASSRLS` system role granted ONLY to those named background workers and NEVER on the request path; every request-path query stays fail-closed under RLS (RIG-2861 OQ-4 = option 1) | Active (Matt, 2026-08-31) | [multi-tenancy & NATS substrate](infra/runtime/compass-managed-multitenancy/design.md#resolved-decisions-freeze--matt-2026-08-31) | | DL-317 | Under multi-tenancy, user/system handle uniqueness is PER-ORGANIZATION not global (RIG-2921 = option A): `account_handles` gains a `tenant_id` and its partial-unique indexes become org-scoped, and the handle resolvers gain a tenant filter — two organizations may each hold `@matt`. The sole globally-unique identifier is the organization NAME, CASE-FOLDED for that uniqueness check (normalized index form, display casing preserved) — `Acme` and `acme` are the same organization (GitHub model). Refines DL-271's "user/system handles globally unique" clause to per-organization under multi-tenancy (single-tenant OSS stays degenerate: one org, so per-org == global); implemented on the RIG-2880 `account_handles` storage contract | Active (Matt, 2026-08-31) | [multi-tenancy & NATS substrate](infra/runtime/compass-managed-multitenancy/design.md#resolved-decisions-freeze--matt-2026-08-31) | | DL-326 | The P2 session-volume clone model is **agent self-clone (Option A)**: the Runner prepares/attaches the volume (empty or snapshot-restored) and the agent clones/fetches in-container with its existing `$HOME` machine-user token — no new host-side forge read credential (distinct from DL-052's Server-only write secret). Snapshot amortization is **clone-only (provenance-(a))**: the snapshot carries a provably-clean post-clone tree (agent clone-complete signal → Runner host-side `git status`-clean verify → snapshot, keyed by an `(account, repo)` index), never cross-session build state (`target/`, caches), so a new session of a seen `(account, repo)` skips the cold clone but builds cold on its own volume — zero cross-session leak. Runner-side box-global read-only clones of subscribed repos are a deferred follow-up optimization, and a host-side read credential earns its existence only when a sessionless prebuild service materializes trees with no session; the `Materialize` signature already accommodates that flip. Resolves this record's OQ-1 (both sub-parts). | Active (Matt, 2026-09-05) | [P2 persistent session volume §OQ-1](infra/runtime/compass-elastic-session-runtime/p2-persistent-session-volume.md#oq-1--clone-model--snapshot-provenance-ruled-2026-09-05-matt) | +| DL-328 | The `gateway_credentials` value payload is encrypted at rest application-side (AES-256-GCM envelope; `value_ciphertext`+`value_nonce`+`key_version` columns, no plaintext column ever; AAD binds the stable row identity so ciphertexts are not row-portable), covering api_key and OAuth rows on ONE seal/open path (D7). A 256-bit master key is auto-provisioned zero-human-step into an operator-chosen WRITABLE SecretSpec provider (self-hosted default `age://` — encrypted-at-rest, headless; a cloud store or Vault/OpenBao where present) under a Postgres advisory lock with every-boot read-back verify (+ a key-fingerprint tripwire). The master key and the existing server secrets (primary + reviewer App PEM, webhook, and the three Linear secrets) live in a NEW physically separate `server_secrets` store (mechanism C1: names-only table, bucket-A infrastructure — no `tenant_id`, RLS off, its own GRANT to `compass_app`/`compass_system`; no delivery/kind columns; a SECOND `SpecResolver` instance over it on the shared profile; an admin-gated `SetServerSecret`/`DeleteServerSecret` RPC) so server secrets are STRUCTURALLY undeliverable to agent containers — the container resolver's manifest never contains their names. C1 splits the DECLARATION registries, not the shared provider keyspace, so F1 is a STRUCTURAL reserved-prefix partition enforced on BOTH the admin RPC AND the `authenticatedOpen` user `SetSecret`/`DeleteSecret` path: every server-secret name carries a reserved prefix (`SERVER_` for the six forge secrets, `GATEWAY_CREDENTIALS_` for the master-key family), the admin door REQUIRES one and the user door REJECTS one, so the two keyspaces are disjoint by name and no name declared through either guarded door is ever live in both tables in either order (against the wiped pre-production baseline — the guards are prospective, and the DB is wiped and re-declared, so no legacy reserved-prefixed row pre-exists) — a pure string check, no cross-table membership SELECT and so no cross-tenant RLS-visibility question. The table split alone is necessary but not sufficient for the structural claim; the prefix partition is what makes it sufficient. No proto ENUM change and no FetchSecrets filter — the admin RPC adds two additive SecretsService methods (supersedes the earlier SERVER_ONLY-delivery-kind fold). The six forge secret NAMES are declared into `server_secrets` at boot from the resolved forge config (on the ordinary compass_app path — `server_secrets` is bucket-A, no RLS, so no reconcile and no BYPASSRLS), gated on the same predicates that enable the forge lanes; their VALUES live in the operator-chosen writable provider (F2), populated by the operator directly (e.g. the `age://` file) or through the `compass server-secret` CLI / admin `SetServerSecret` RPC for rotation. Because the provider is populated BEFORE the server runs, the six values are present at first boot with NO running server required to bootstrap them (Matt-ruled — the server boots with the secrets set through the provider/config, not via a running-server RPC); a configured App whose forge value is absent from the provider is a clean STATIC deploy-time startup error (`validateForgeSecret`), fixed by populating the provider and rebooting. The pre-production deployment is wiped and the six values re-provisioned under prefixed names (name-keyed provider, so a provider write), not migrated in place, closing the pre-existing inject-all exposure. The `age://` default requires a secretspec-go bump to `>= 0.17` with the `age` build feature (repo pins v0.15.0) — a gating prerequisite PR. Managed-plane KMS is a provider-URI deployment config, not a code fork; rotation is deferred (single live key, `key_version` landed at v1, bounded key-loss blast radius — credentials are re-obtainable); per-tenant at-rest isolation + gateway-topology exposure is a tracked follow-up | Active (Matt, 2026-09-02) | [gateway-credentials at-rest encryption](server/compass-gateway-credentials-at-rest-encryption.md#d1--application-layer-aes-256-gcm-envelope-encryption) | > Note (2026-07-31, RIG-1570 R5): DL-065's retired internal `ResumeContext > resume = 12` envelope field is NOT silently reintroduced — the collapsed diff --git a/docs/designs/server/compass-gateway-credentials-at-rest-encryption.md b/docs/designs/server/compass-gateway-credentials-at-rest-encryption.md new file mode 100644 index 00000000..aacd01ce --- /dev/null +++ b/docs/designs/server/compass-gateway-credentials-at-rest-encryption.md @@ -0,0 +1,1202 @@ +# gateway_credentials at-rest encryption + master-key custody + +Status: Active + +Tracking: RIG-2863 (parent RIG-1715) + +Addendum to the frozen record +[`compass-server-llm-gateway`](./compass-server-llm-gateway/design.md) +(§Credential storage and rotation, L312–373). Scope: encryption-at-rest for the +`gateway_credentials` value store ONLY. This record does not re-open the +store's shape, the CAS `version` discipline, the scope column, the +RPC-vs-Postgres-direct wiring, or the stack-token channel — those stay as the +frozen record decided. Ships as its own PR; frozen on merge; the +credential-store build proceeds against it. +One deliberate scope expansion, Matt-ruled (D6/T0): server-only secrets get a +physically SEPARATE `server_secrets` store (mechanism C1) — a new table plus +a second resolver instance — so the master key and the existing +PEM/webhook/Linear server secrets are structurally undeliverable to agent +containers (the table split PLUS the F1 reserved-prefix partition — server-secret +names carry a reserved prefix the user declare/set path rejects, making the two +keyspaces disjoint by construction; the split alone is necessary but not +sufficient; see D6). +C1 needs no public proto ENUM change: it REMOVES the +`SECRET_DELIVERY_SERVER_ONLY` proto/store/secrets enum additions (and the +FetchSecrets filter) the earlier delivery-kind mechanism required, and leaves +the container FetchSecrets path byte-for-byte unchanged. It DOES add two +admin-gated methods to `SecretsService` (`proto/compass/v1/compass.proto`, +the public codegen lane) — an additive, non-breaking service-method addition, +not a change to an existing wire enum. + +## Problem / Intent + +The gateway record adds `gateway_credentials`, the FIRST value-persisting store +in compass — deliberately breaking the names-only half of the secrets +invariant. The invariant it breaks is stated at +`go/internal/secrets/secrets.go:20-22`: + +> "Values live only in the provider and this process's memory during a +> resolve; they are never persisted by Compass and never logged." + +The frozen record specifies the new store's shape ("api_key and OAuth-shaped +payloads (access/refresh/expiry), a monotonic `version` per row supplying the +CAS substrate, and a scope column", design.md:324-326) but says nothing about +encryption at rest. A security red-team flagged this HIGH (CWE-311/312/522/532): +a plaintext value column makes every `pg_dump`, read replica, backup, and +operator `SELECT` a bulk all-tenant credential exfiltration path. The record +already concedes the blast radius is total at the process level +(design.md:333-337): + +> "Blast radius, stated: a compromised gateway (holding one stack token) can +> read every tenant's provider credentials — the isolation boundary is the +> per-tenant pool scoping enforced server-side on the credential-list surface +> (below), not the TS process, which by design holds them all." + +That concession covers a compromised *gateway process*; it does not license a +plaintext *database*. This addendum adds the missing at-rest-encryption and +master-key-custody decision before the store is built. + +## Approach + +Matt has ruled the core approach; this section records it as decisions. + +### D1 — Application-layer AES-256-GCM envelope encryption + +The credential value payload (the full api_key / OAuth +access+refresh+expiry JSON) is encrypted application-side before it reaches +Postgres: AES-256-GCM, a fresh random 96-bit nonce per encryption (per write, +never reused), authenticated. The `gateway_credentials` row stores ONLY +`value_ciphertext` + `value_nonce` +(+ `key_version`, D4); no plaintext value column exists in any migration +version — the store is born encrypted. A DB dump, replica, or SELECT alone +yields nothing usable. + +Why the fresh-nonce discipline is absolute, quantified: reusing a 96-bit GCM +nonce under one key forfeits both confidentiality AND authenticity — a +two-message catastrophe (keystream XOR + authentication-key recovery), not a +gradual weakening. Random 96-bit nonces are collision-safe only to ~2^32 +encryptions per key (NIST SP 800-38D). The write rate is not one-per-credential: +every hourly OAuth refresh write-back is a fresh encryption ("OAuth +access/refresh tokens are refreshed by the gateway on the hour", +design.md:322-323). At plausible scale (10^3–10^4 credentials, hourly) that is +~10^7–10^8 encryptions/year — comfortably under 2^32, so v1's single live key +has years of headroom; key rotation (OQ-1) is the nonce-budget release valve +when scale grows, which is itself part of why OQ-1's deferral is safe. + +This is the first at-rest crypto in the Go tree — verified by grep this +session: no `crypto/aes` / `cipher.NewGCM` usage exists anywhere under `go/` +(existing `crypto/rand` uses are IDs/tokens/certs only, e.g. +`go/internal/store/ids.go:4`). The envelope helper is therefore a new (small, +justified) package: no existing abstraction carries symmetric at-rest crypto, +and the seam must be reusable if a later store ever needs the same discipline. + +### D2 — Master key auto-provisioned into the existing SecretSpec provider + +The master key never touches the DB and never requires a human step +(rule://no-human-clicks). Every running Compass already has exactly one +configured SecretSpec provider — the seam described at +`go/internal/secrets/secrets.go:10-13`: + +> "this package reads that registry, generates the SecretSpec manifest the +> resolver resolves against, calls SecretSpec to resolve the actual values +> from the configured provider (keyring/1Password/Vault/…)" + +On boot, Compass resolves the declared master-key secret +(`GATEWAY_CREDENTIALS_MASTER_KEY`). If absent, it generates a fresh 256-bit +key from `crypto/rand` and — serialized against concurrent booters through a +Postgres advisory lock (T2) — provisions it: + +- writes the value into the provider via `secrets.Resolver.Set` + (`go/internal/secrets/resolver.go:219`, `func (r *SpecResolver) Set(ctx + context.Context, name, value string) error` — "Set writes value into the + provider for name via the pinned CLI, feeding the value on stdin (never + argv, so it is not visible in the host process list)", + resolver.go:206-207); +- registers the name in the SEPARATE `server_secrets` store (D6) via the + server-internal `DeclareServerSecret` (T0) — a mirror of + `store.DeclareSecret` (`go/internal/store/secrets.go:82`, `func (s *Store) + DeclareSecret(ctx context.Context, actor AccountID, name string, delivery + SecretDelivery, kind SecretKind, provider, host string) error` — "It + stores NO value — the value lives in the SecretSpec provider", + secrets.go:74-75) MINUS the delivery/kind routing parameters, which do not + exist for server secrets (they are never container-delivered and never + reach the T5 materializer) — never into the user `secrets` table, whose + every row rides the container-delivery manifest; +- re-resolves and byte-compares before the key is ever used to encrypt (the + read-back verify, T2). + +Thereafter the key is resolved at boot — through the SERVER-SECRET resolver +instance (D6), the second `SpecResolver` reading `server_secrets` — and held +in process memory only, following the existing declared-secret boot-resolve +pattern the Forge App PEM uses (`newDeclaredSecretResolver`, +`go/server/serve.go:1469`: "resolves the declared server_only secret NAME to +its raw value bytes on each call", serve.go:1463-1464 — a "server_only" that +is convention-only prose today; D6 makes it a physically separate store). +Unlike the webhook secret's per-request resolve, the master key is resolved +once at startup and cached for the process lifetime +(a decrypt happens on every credential read; a provider round-trip per decrypt +would be the same amplification `cachedWebhookSecret` exists to prevent, +serve.go:1484-1491). + +### D3 — Managed plane: KMS as just another provider URI, zero core change + +The managed plane points the ONE declared master-key secret at a KMS-backed +SecretSpec provider URI. The core stays cloud-agnostic; there is no KMS SDK, +no cloud-conditional code path, and no self-host/managed fork in the crypto +code. KMS-grade custody is a deployment configuration, not a code change. + +### D4 — Key versioning column from day one + +Each row carries a `key_version` (smallint, starts at 1) identifying which +master-key generation encrypted it. Whether v1 ships active rotation is an +open question (OQ-1), but the column is load-bearing NOW: adding it later +means a schema migration plus a backfill under ambiguity about which key +encrypted which row. Cheap at creation, expensive retrofit. + +### D5 — Redaction discipline: Stringer alone is not enough + +Breaking the never-persisted half of the invariant does not touch the +never-logged half — but the established Stringer pattern +(`ResolvedSecret.String()`/`GoString()`, `go/internal/secrets/secrets.go:150-156`: + +> `func (s ResolvedSecret) String() string { return fmt.Sprintf("ResolvedSecret{name: %q, kind: %d, delivery: %d, value: }", …) }` +> `func (s ResolvedSecret) GoString() string { return s.String() }` + +) covers only fmt-verb formatting — and slog under a TextHandler, which +formats via the fmt path. It does NOT cover a slog `JSONHandler` (reflection +over exported fields, ignores `String()`), `json.Marshal`, or direct field +access — and T4's decrypted-payload type is exactly the OAuth JSON shape that +gets marshaled toward the gateway RPC, so it will have serializable fields. +A one-line handler swap must not silently un-redact the fleet. + +Therefore the value-bearing types this record adds MUST either: + +- keep secret fields UNEXPORTED with accessor methods (the `envelope.Key` + posture: unexported `[32]byte`, reflection-proof), or — where the payload + shape needs exported fields — implement `slog.LogValuer` AND a redacting + `MarshalJSON` alongside the `String()`/`GoString()` pair; and +- never serve as the RPC serialization type: the gateway-bound response is + built by an explicit proto/DTO conversion, never `json.Marshal` of the + payload type itself. + +The store never logs a row's value fields, and error paths wrap without +embedding plaintext or key material. + +### D6 — A physically separate `server_secrets` store (mechanism C1, Matt-ruled) + +The threat D2 must defeat: the declared-secrets registry is inject-all by +design — "NO per-agent grant column (the MVP injects the whole store into +every agent; per-agent scoping is a named FUTURE seam)" +(`go/internal/store/migrations/0001_init.sql:391-393`). FetchSecrets resolves +the WHOLE registry with no filter — `resolved, err := h.resolver.Resolve(ctx, +"runner fetch")` then `for _, s := range resolved { out = append(out, +resolvedSecretToProto(s)) }` (`go/internal/runnerhub/handler.go:303-311`) — +and the Runner materializes every resolved value into the container pre-exec +(`resolved, err := h.link.FetchSecretsByContainer(ctx, name)` → +`h.materializer.Install(ctx, handle.ID(), handle.HomeDir(), +handle.WorkspaceUID(), resolved)`, `go/internal/runner/host.go:378-383`). +Declared into the `secrets` table at all, the master key — which decrypts +every tenant credential — would land on disk inside every untrusted agent +sandbox, defeating the encryption entirely. + +**Matt ruled: a separate store, not a delivery flag.** Server-only secrets +get their OWN physically separate table, `server_secrets`, so they can NEVER +get mixed up with user secrets that are delivered to agent containers. +Container delivery becomes default-CLOSED by construction, not +default-open-minus-a-filter. + +Why the table boundary IS the delivery boundary: the delivery surface is the +resolver's MANIFEST. `SpecResolver` reads its declared set through the +`declarations` interface — `DeclaredSecrets(ctx context.Context) +([]store.SecretDeclaration, error)` (`go/internal/secrets/resolver.go:29-31`; +the `store declarations` struct field, resolver.go:57) — and `buildManifest` +"renders the SecretSpec manifest TOML for a declared set: one `[project]` +block and one `[profiles.]` block with every declared name as a +required key" (resolver.go:99-101; the function, resolver.go:105-128); +`Resolve` can only return names present in that manifest. Today ONE resolver +instance (`resolver := secrets.NewSpecResolver(st, secretsStateDir(cfg))`, +`go/server/serve.go:528`) serves BOTH the container path (FetchSecrets → +`Resolve`, handler.go:303-311) and the boot consumers (PEM/webhook via +`newDeclaredSecretResolver` → `Resolve`, serve.go:1469-1481). C1 splits the +READ: a store view over `server_secrets` (`ServerDeclaredSecrets`, T0) feeds +a SECOND `SpecResolver` instance. The container resolver keeps reading +`secrets`; its manifest never contains a server-secret name, so a server +secret is undeliverable to containers via the container manifest — there is +nothing to filter. That structural property holds only while no name is +ever present in BOTH tables. F1 makes that structural, by NAME construction: +every server-secret name carries a RESERVED PREFIX (the master-key family keeps +`GATEWAY_CREDENTIALS_`; the six configured forge secrets carry `SERVER_`), and +the two store doors enforce the partition with a pure string check — the admin +`SetServerSecret`/`DeclareServerSecret` path REQUIRES a reserved server-secret +prefix, and the user `SetSecret`/`DeclareSecret` path REJECTS any name carrying +one. The two keyspaces are therefore disjoint by name construction: no name +declared THROUGH EITHER GUARDED DOOR can land in both tables in either order, so +the both-tables end state — a user `SetSecret` minting a shadow `secrets` row +under a server-secret name, or an admin `SetServerSecret` declaring under a name +already live in `secrets` — is UNREACHABLE, not merely healed after the fact. +The guards are prospective (they gate declares, not existing rows), so the claim +is stated against the clean pre-production baseline this record's wipe +establishes: the DB is wiped and re-declared (Matt-ruled — single operator, no +live data), so no legacy `secrets` row carrying a reserved prefix pre-exists to +break the partition, and none can be created thereafter. F1 needs no cross-table +membership SELECT (so no cross-tenant RLS-visibility question) and no boot +reconcile: the partition is enforced at declaration time on both doors. The +table split alone is necessary but not sufficient; the reserved-prefix partition +is what makes it sufficient. The prefix check is a name test only — it does not +widen the underlying declare/set/rollback trio's single-writer contract ("is +not atomic and assumes no concurrent same-name writer", secrets_service.go:88-91), +which the existing `SetSecret` path already carries; per-name write atomicity +across paths is deferred to the multi-writer work that comment flags, tracked +with the per-tenant defer (RIG-3237). + +C1 keeps the SAME SecretSpec profile for both instances — the shared project +is `manifestProject = "compass"` (resolver.go:19) and the profile +`defaultProfile = "default"` (resolver.go:23; `WithProfile` exists, +resolver.go:78, but C1 does not use it) — so the provider keyspace is shared. +Because the reserved-prefix partition renames the six forge secrets under +`SERVER_`, and the pre-production deployment carries no live data worth +preserving, the existing six declarations are simply WIPED; their NAMES are +re-declared at boot under their prefixed names from the resolved forge config +(OQ-4), and the operator re-supplies each VALUE in the writable provider under +the prefixed name (name-keyed provider, so the rename requires a provider +write) BEFORE the server boots — not an in-place row move, and not a +running-server RPC call (the RPC/CLI is the rotation surface, not the first-boot +bootstrap path; F2). + +**C1 REMOVES the public proto ENUM change** the previous delivery-kind mechanism +required: no `SECRET_DELIVERY_SERVER_ONLY` proto enum value, no +`SecretDeliveryServerOnly` store enum, no secrets-package +`DeliveryServerOnly`, no widened migration CHECK, and no FetchSecrets +delivery filter. The container FetchSecrets path is byte-for-byte UNCHANGED. +The delivery enum gains no new value in any of the four representations — +two live delivery kinds throughout (plus the proto3 `UNSPECIFIED` zero +sentinel, which is not a delivery kind): +`delivery SMALLINT NOT NULL CHECK (delivery IN (0, 1))` (0001_init.sql:399); +`SecretDeliveryFile SecretDelivery = 0` / `SecretDeliveryEnv SecretDelivery += 1` (`go/internal/store/secrets.go:18-25`); `DeliveryFile` / `DeliveryEnv` +(`go/internal/secrets/secrets.go:39-46`); `SECRET_DELIVERY_UNSPECIFIED = 0; +SECRET_DELIVERY_FILE = 1; SECRET_DELIVERY_ENV = 2;` +(`proto/compass/v1/compass.proto:160-164`). + +Writes into `server_secrets` go through a NEW admin-gated +`SetServerSecret`/`DeleteServerSecret` RPC (T0) — mirroring the user +`SetSecret` declare-then-Set flow (`go/server/secrets_service.go:92`) minus +delivery/kind, classified `adminOnly` in `classifyProcedure` +(`go/internal/auth/admin_gate.go:47`, :27; "An unrecognized path (ok=false) +is treated as adminOnly — fail closed, never admit an unknown method as +open", admin_gate.go:44-46). Today's user-facing `SetSecret` cannot declare a +row *as* a server secret — `secretRoutingFromProto` admits only File/Env +delivery (secrets_service.go:275-284), with no server-only delivery value — so +operators declare server secrets through the new RPC. It CAN, however, mint an +ordinary `secrets` row under a NAME that collides with a server secret +(delivery=File/Env), which is exactly why the F1 prefix guard is mandatory on +this user path and not merely on the admin RPC — the user path rejects any +reserved-prefixed name, so the collision cannot be declared. + +This retroactively fixes a pre-existing exposure: the App PEM, webhook +signing, and Linear OAuth secrets are server_only by convention only +("Declared server_only secret NAME holding the PRIMARY App PEM private key", +`go/cmd/compass-server/main.go:414-417`; webhook :418-421; Linear :435-450; +boot-resolved by name via `newDeclaredSecretResolver`, serve.go:1469-1481) +and today ride the same inject-all path into every agent container. They are +re-declared under `SERVER_`-prefixed names in the separate `server_secrets` +store in this PR chain (OQ-4, RESOLVED) — the pre-production deployment is +wiped, the NAMES re-declared at boot from the resolved forge config, and the +VALUES re-populated in the writable provider (the admin `SetServerSecret` RPC / +`compass server-secret` CLI serve rotation on a running server, not first-boot +bootstrap; F2), not migrated in place. + +### D7 — One seal/open path for `api_key` and OAuth payloads + +Both stored payload shapes — an OAuth token bundle and a bare `api_key` — are +sealed and opened through the SAME envelope path (D1); there is no plaintext +branch and no per-shape code fork. Both are secrets of the same sensitivity: a +stored `api_key` is as long-lived and as disclosure-critical as an OAuth +refresh token, and a split path would leave the most static secret class in +plaintext for zero benefit. T3's single encrypted column and T4's single +wiring path encode this. (Matt-ruled; resolves OQ-3.) + +## Alternatives considered + +The core choice (envelope encryption + auto-provisioned key in the existing +provider) is Matt-ruled and not re-opened; the custody fork and the +server-secret containment-mechanism fork are recorded here because the +rejected branches are the ones a future reader will reach for first. + +- **Mechanism A: a real SERVER_ONLY delivery kind + FetchSecrets filter — + REJECTED (superseded by Matt's C1 ruling).** The previously folded design: + mint `SERVER_ONLY` across the four delivery representations (migration + CHECK widen, store enum, secrets package, a PUBLIC proto enum value) plus + a delivery filter in the FetchSecrets handler skipping SERVER_ONLY rows + before the append (handler.go:307-310). Rejected because it is + default-OPEN minus a subtractive filter: server and user secrets share one + table and one resolver manifest, and one edit — a dropped filter clause, a + mis-mapped enum arm — re-exposes the master key to every container. It + also costs a public proto ENUM widening — a change to an existing wire + contract — where C1's proto delta is two additive `SecretsService` methods. +- **Mechanism C2: separate table + a separate SecretSpec PROFILE — + considered, DEFERRED.** The same `server_secrets` table, but the server + resolver pinned to its own profile (`WithProfile`, resolver.go:78) so even + the provider keyspace is isolated. Fullest isolation — but it is NOT the + cheaper-migration loser it might appear: under the reserved-prefix partition + BOTH mechanisms now require a provider write under new keys (the six values + are re-provisioned regardless), so the honest distinction is WHERE the write + lands, not WHETHER one is needed. C1 writes under new NAMES in the shared + profile via a pure string check at the DECLARATION layer, with no second + profile to configure or operate; C2 would write under the same names in a + new PROFILE, additionally isolating the provider keyspace (closing the + residual named below). The pre-production wipe premise (D6) makes either + write a re-declare, not an in-place migration. C1 was ruled sufficient; the + profile split remains available later if provider-keyspace isolation is ever + needed. Named residual under C1: the shared profile leaves the provider + keyspace a shared mutable surface — no longer reachable by the user + `SetSecret` path (the F1 prefix guard rejects reserved-prefixed names there), + but still reachable by an operator's out-of-band `secretspec set` against the + shared provider — so all SEVEN server-secret names' provider values (the + master key plus the six re-declared names: primary and reviewer App PEM, the + GitHub webhook secret, and the three Linear secrets — client id, client + secret, and webhook) are + guarded by the + F1 reserved-prefix partition (T0, both paths — the user path rejects + reserved-prefixed names, the admin path requires them) at the DECLARATION + layer rather than at the provider-keyspace layer; the boot read-back verify + additionally covers the master key. C2 would make the provider-keyspace + isolation structural for all of them. + Acceptable with the guard in place; the follow-up (per-tenant credential + at-rest isolation + gateway-topology exposure) is where C2 is reconsidered + (RIG-3237). +- **pgcrypto / key-in-DB — REJECTED.** `pgp_sym_encrypt` or a key stored in a + Postgres table/GUC puts the key inside the same blast radius as the + ciphertext: one `pg_dump` carries both, reducing the encryption to obfuscation. + The red-team requirement is specifically that the key lives OUTSIDE the DB. +- **Require an external cloud KMS — REJECTED for the OSS core.** A generic + self-hoster has no AWS/GCP KMS and must not need one; they have already + configured exactly one SecretSpec provider (keyring / 1Password / Vault / + env) to run Compass at all. Mandating KMS either forks the code + (self-host vs managed paths) or gates self-hosting on a cloud account. + The chosen approach *subsumes* this alternative: the managed plane gets + KMS-backed custody by pointing the declared secret at a KMS provider URI + (D3) — KMS becomes deployment config, not a code requirement. +- **Plaintext now + filed follow-up — REJECTED.** Ships a known HIGH hole and + creates a migration burden (encrypt-in-place over live rows) that the + build-it-encrypted-first ordering avoids entirely. Matt ruled the encryption + record lands BEFORE the store is built. + +## Plan + +### Global Constraints + +- Go server under `go/` (gateway itself is TS/Bun, but this record's code is + entirely Go server + store side); existing lint/test discipline applies. +- AES-256-GCM only; nonces from `crypto/rand`, 96-bit, fresh per encryption, + never counter-derived; key is 256-bit from `crypto/rand`. +- The master key NEVER appears in the DB, in logs, in argv (Set feeds stdin, + resolver.go:206-207), or in error strings. +- Auto-provisioning is zero-human-step (rule://no-human-clicks): first boot + generates, stores, and declares the key with no operator action. +- The names-only invariant (`secrets.go:20-22`) is preserved for EVERYTHING + except `gateway_credentials` values; both declared-name registries + (`secrets` AND the new `server_secrets`) stay names-only — the master + key's declaration row is names-only like any other. `server_key_state` is + not a declared-name registry and holds no value: a salted digest of the + master key plus its salt, which discloses nothing about a 256-bit random + key — the names-only invariant is untouched by it. +- Every value-bearing type redacts under `%s`/`%v`/`%#v` (D5). +- The master key lives in the separate `server_secrets` store (D6): its name + never appears in the container resolver's manifest, so it is NEVER + materialized into an agent container — a structural property of the split + store, not a filter to maintain. +- Every Seal binds AAD = the row's stable identity (T4); ciphertexts are not + portable between rows. +- Every `file:line` citation in this record is anchored by its SYMBOL name; the + line numbers are a convenience that drifts — re-grep the symbol before trusting + a line. + +### T0 — The `server_secrets` store: table, resolver split, admin RPC (prerequisite) + +Implements D6 (mechanism C1). PREREQUISITE of T2 — the key cannot be +declared into a store that does not exist. + +- `Interfaces:` + - Migration (a NEW migration file, never an edit to 0001): `CREATE TABLE + server_secrets` — names-only, mirroring `secrets` + (0001_init.sql:393-425) MINUS the container-delivery routing: `name TEXT + PRIMARY KEY` (same env-var-name grammar, validated at the store door); + `declared_by TEXT REFERENCES accounts (id) ON DELETE RESTRICT`, + NULLABLE — `NULL` = server-provisioned (the master key; contrast + `secrets.declared_by`, which is `NOT NULL REFERENCES accounts (id) ON + DELETE RESTRICT`, 0001_init.sql:410 — justification below); + `created_at`/`updated_at`. NO `delivery` column (server secrets are + never container-delivered — that IS the point) and NO `kind` column + (they never reach the T5 materializer). + - Key-state table (same migration): a single-row `server_key_state` table + (`id SMALLINT PRIMARY KEY DEFAULT 1 CHECK (id = 1)`, `key_version SMALLINT + NOT NULL DEFAULT 1`, `key_fingerprint BYTEA NOT NULL`, `fingerprint_salt + BYTEA NOT NULL`, `updated_at`) holds the T2/T5 key-swap tripwire's + non-secret salted digest. It is NOT a declared-name registry, so the + names-only invariant (§Global Constraints) is untouched — a salted digest + of a key is not that key's VALUE. Same-migration `GRANT SELECT, INSERT, + UPDATE ON server_key_state TO compass_app, compass_system` (the grant is + not inherited — see the `server_secrets` grant rationale below). It is + likewise **bucket-A infrastructure** (deployment-global, no `tenant_id`, + RLS NOT enabled) and joins `server_secrets` in the `bucketA` allow-list + (rls_pgtest_test.go:543) so the self-auditing RLS guard test stays a + conscious gate. + - Grants + RLS (mandatory — the new-table integration the shipped + multi-tenancy regime, RIG-3106, requires): the same new migration + issues `GRANT SELECT, INSERT, UPDATE, DELETE ON server_secrets TO + compass_app, compass_system`. This is NOT inherited: 0001's grant is a + one-time `ON ALL TABLES IN SCHEMA` snapshot (0001_init.sql:896) and there + is no `ALTER DEFAULT PRIVILEGES` anywhere in the migrations, so a table + created by a later migration receives no grant and every server-side + resolve would fail `permission denied for table server_secrets` (fails + closed, but breaks the T2 boot path). RLS posture: `server_secrets` is + declared **bucket-A infrastructure** (Matt-ruled) — NO `tenant_id` + column, RLS NOT enabled: the master key and the PEM/webhook/Linear + secrets are deployment-global (the master key decrypts EVERY tenant's + credentials), so there is no tenant to scope by. T0 adds `server_secrets` + to the `bucketA` allow-list in go/internal/store/rls_pgtest_test.go:543 + so the self-auditing RLS guard test stays a CONSCIOUS gate (that test + keys off `tenant_id`, so a bucket-A table is otherwise invisible to it — + the allow-list edit is what records the deliberate exemption). Live + cross-tenant EXPOSURE (one shared gateway process holding all tenants' + creds) is a gateway-topology property of the parent record + (design.md:333-337), out of scope here and tracked as a follow-up + (RIG-3237). + - declared_by nullability, justified: the boot provisioner declares the + master key with no human actor. `NULL` = server-provisioned is honest + provenance; attributing the row to the bootstrap-admin account + (available at that point in boot, serve.go:358-362) would falsify the + audit trail and couple key provisioning to account-bootstrap ordering. + Operator-declared rows (the new RPC) always carry the caller's account + id; `ON DELETE RESTRICT` still protects them. + - Store: `DeclareServerSecret(ctx, actor, name)` / + `DeleteServerSecretDeclaration(ctx, actor, name)` mirroring + `DeclareSecret` / `DeleteSecretDeclaration` (store/secrets.go:82, :160) + minus delivery/kind/provider/host (actor nullable-empty for the + server-provisioned path). `DeclareServerSecret` carries the F1 PREFIX guard + at the store door — it REQUIRES the name to carry a reserved server-secret + prefix (`SERVER_` or `GATEWAY_CREDENTIALS_`), rejecting any unprefixed name — + so no writer (the admin RPC, or any future second writer) can create a + `server_secrets` row under a name the user keyspace owns. Combined with the + user path's rejection of those same prefixes (below), the two doors partition + the keyspace by name, so F1 holds at BOTH doors, not only at the RPC — a pure + string check, no cross-table read, no transaction spanning both tables. It + ALSO adds the read + `ServerDeclaredSecrets(ctx)` — + a thin store view whose `DeclaredSecrets(ctx context.Context) + ([]store.SecretDeclaration, error)` method (the `declarations` + interface shape, resolver.go:29-31) reads `server_secrets`, mapping + rows to `store.SecretDeclaration` with generic kind / zero delivery + (the resolver uses only the name to build its manifest), so + `NewSpecResolver` is reused UNCHANGED. + - Resolver split: a SECOND `SpecResolver` instance (the SERVER resolver) + constructed over that view — SAME profile (`defaultProfile = "default"`, + resolver.go:23) and project (`manifestProject = "compass"`, + resolver.go:19), its own manifest state dir. The container resolver's + CONSTRUCTION at serve.go:528 is unchanged — no filter, nothing to filter + — but its boot CONSUMERS must be re-pointed: because the six server-secret + names are re-declared under `SERVER_`-prefixed names in `server_secrets` + (not `secrets`), every boot consumer that + still resolves those names through the serve.go:528 instance breaks, + because that instance's manifest no longer declares them. THREE failure + modes verified in the tree: (a) HARD BOOT FAILURE — `validateForgeSecret` + (serve.go:1539-1549) returns `forge secret %q not declared` for an absent + name and is called on the primary App PEM (:1013) and the App webhook + secret (:1016) — both inside `buildBoardWebhookWiring` under + `buildForgeReadWiring`, propagated to `failStartup`, so every GitHub-App + deployment fails its first post-upgrade boot; (b) SILENT SECURITY + DEGRADATION — `forgeSecretDeclared` (serve.go:1746-1757) returns + `(false, nil)` for an absent name rather than an error, and its caller + `buildLinearWebhookWiring` maps that to a legitimate off-state — `if + !declared { return nil, nil }` (serve.go:1082-1083) — so the Linear webhook + ingress silently unmounts, and `buildLinearTokenSource` (serve.go:1699-1717) + drops the Linear write + notify lanes the same way (`return nil, nil` at + :1716); + (c) SILENT CAPABILITY LOSS — `wireForgeWriteCaller` (:587) resolves the + forge-write set and gates on `forgeWritesEnabled` (:1583, over + `forgeWriteAppsConfigured` :270-275): with BOTH PEM names absent from the + container resolver's set the predicate reads both-absent, + `warnPartialForgeWriteSecrets` returns SILENTLY (:1826-1827, `havePrimary + == haveReviewer`), `wireForgeWriteCaller` returns nil (:1585), + `hub.SetForgeCaller` is NEVER called, and the reviewer-PEM + `validateForgeSecret` at :1638 is UNREACHABLE — so agent forge WRITES + fail-closed to `CodeUnavailable` fleet-wide with no error at boot. + (Ordering note: (c) is MASKED by (a) on a FULL no-repoint T0 — + `forgeWriteAppsConfigured` gates `havePrimary` on `App.AppID != 0` + (serve.go:272), which is exactly `boardIngestionEnabled()` (serve.go:224-226), + so any writes-enabled deployment hard-fails at the primary-App-PEM + `validateForgeSecret` (serve.go:1013) before reaching `wireForgeWriteCaller` + (:587). (c) is the PARTIAL-repoint mode: it becomes live the moment the read + path is repointed and the write path is not — which is precisely why the + write lane needs its own positive assertion rather than inheriting the read + lane's.) So T0 + has an explicit CONSUMER-REPOINT deliverable with TWO seams. (1) Resolver + instance: thread the SERVER resolver instance (not the serve.go:528 + container instance) into `buildForgeReadWiring` (serve.go:567), + `buildLinearWebhookWiring`, `buildLinearTokenSource`, `wireForgeWriteCaller` + (:587) and `buildForgeWriteService`, and into the `newDeclaredSecretResolver` + / `newCachedWebhookSecret` closures (serve.go:1025/1039/1085/1644) that mint + the App token + verify webhook HMAC. (2) Resolve-side NAME prefix: the + prefix RENAMES the six (`LINEAR_FORGE_CLIENT_SECRET` → + `SERVER_LINEAR_FORGE_CLIENT_SECRET`), and the provider keyspace is keyed by + NAME, so the DECLARE side (`SetServerSecret`, T0) and the RESOLVE side must + agree on the prefixed name or every server resolve misses. `resolved()` + (serve.go:232-246) applies NO prefix — its output feeds operator-facing text + (log/error lines, the `warnPartialForgeWriteSecrets` and + `buildLinearTokenSource` partial-config Warn strings, serve.go:1719-1721 / + 1833-1834), which must stay unprefixed. But that same unprefixed name is ALSO + compared against the SERVER resolver's resolved set at several sites, and a + comparison against a `SERVER_`-prefixed resolved set MUST use the prefixed + name or it never matches. So the seam is an INVARIANT, not a hand-listed set: + every argument matched against a resolved `secrets.ResolvedSecret.Name` is + `serverSecretName()`-wrapped, and every string that reaches a log, an error, + or operator config is not. The wrapped comparison sites are + `validateForgeSecret` (serve.go:1013/1016), `newDeclaredSecretResolver` / + `newCachedWebhookSecret` (:1025/1039/1085/1638/1644 plus + `buildLinearTokenSource`'s id+secret pair), `secretDeclared` inside + `forgeWriteAppsConfigured` (serve.go:272-273 — the write-Apps gate: it scans + the resolved set by `s.Name == name`, serve.go:280-290, so an unwrapped name + would leave `havePrimary`/`haveReviewer` false forever and silently disable + forge writes fleet-wide), and `forgeSecretDeclared` (serve.go:1078 — the + Linear webhook mount gate, another `s.Name == name` scan, serve.go:1746-1757; + unwrapped it unmounts the webhook even after the value is provisioned). The + same helper applies the prefix at the `SetServerSecret` provisioning site, so + the declared and resolved names agree by construction. + `buildLinearWebhookWiring` is called from INSIDE `buildDoors` (serve.go:788), + whose single `resolver` parameter (:677) ALSO feeds `buildNetworkServer` + (:803) → `runnerhub.NewMountedHandler` (network_door.go:313) — the container + FetchSecrets path that MUST keep reading `secrets`; so `buildDoors` takes + BOTH resolver instances (or receives a pre-built `linearWebhookHandler` + from Serve), because swapping its one `resolver` argument would repoint the + container manifest at `server_secrets` and deliver every server secret into + every agent container, inverting D6. + - RPC: `SetServerSecret`/`DeleteServerSecret` on the secrets service — + mirrors the `SetSecret` declare-then-Set flow and its rollback + discipline (secrets_service.go:92-145) minus delivery/kind, targeting + `server_secrets`; admin-gated (`adminOnly` in `classifyProcedure`, + admin_gate.go:47). Carries the F1 PREFIX guard: it REQUIRES the declared + name to carry a reserved server-secret prefix (`SERVER_` for the six forge + secrets, `GATEWAY_CREDENTIALS_` for the master-key family), rejecting any + unprefixed name with an actionable error. This is the admin half of the + structural partition (D6): because the user path (below) rejects those same + prefixes, a reserved-prefix name can only ever live in `server_secrets` and + an unprefixed name can only ever live in `secrets` — the two keyspaces are + disjoint by name, so no name DECLARED THROUGH EITHER GUARDED DOOR is ever + live in both tables in either order (against the wiped pre-production + baseline, D6), with no cross-table read. The reserved master-key name + `GATEWAY_CREDENTIALS_MASTER_KEY` is additionally rejected on + `SetServerSecret`/`DeleteServerSecret` (rotation is OQ-1 machinery, never a + raw overwrite), so an admin cannot clobber the auto-provisioned key. + - User-path prefix guard (F1 — mandatory, NOT admin-RPC-only): C1 + shares the provider keyspace (§D2 read-back verify), so the + `authenticatedOpen` user `secretsService.SetSecret`/`DeleteSecret` path + (any authenticated account, admin_gate.go:122-125) can overwrite a + server secret's provider value AND — absent the guard — mint a shadow + `secrets` row under a server-secret name, which the inject-all path then + delivers into every container. So the user path is the other half of the + partition: `SetSecret`/`DeleteSecret` (and `DeclareSecret` at the store + door, so the shadow row can never be created at all) MUST REJECT any name + carrying a reserved server-secret prefix (`SERVER_` or + `GATEWAY_CREDENTIALS_`) — checked BEFORE `resolver.Set`/`Delete` + (secrets_service.go:124/208). A pure string check, not a membership SELECT: + it needs no read of `server_secrets` and no cross-tenant visibility. A T0 + deliverable on the existing user path, not only the new + admin RPC. + - Value custody + operator provisioning (Matt-ruled, F2): the VALUES of all + server secrets — the six forge secrets and the `GATEWAY_CREDENTIALS_` + master key — live in an operator-chosen WRITABLE SecretSpec provider, not + in the DB. The self-hosted default is `age://` (an age-encrypted file with + a local age identity key: writable, so the master key can mint into it; + encrypted at rest, which this whole record is about; headless- and + container-safe with no D-Bus dependency, unlike `keyring`). A deployment + with a cloud secret store points the SERVER resolver at it instead + (`awssm`/`awsps` on AWS, `aac`/`akv` on Azure), and one already running + Vault/OpenBao uses that — the provider is a per-resolver-INSTANCE config + choice (`WithProvider`, resolver.go:73/75), not a hardcode, so this is a + recommended default rather than a fixed backend. Two things make this an + EXECUTABLE T0 deliverable rather than prose: (1) DEPENDENCY PREREQUISITE — + `age` is a secretspec 0.17+ provider behind an `age` build feature, but + the repo pins `github.com/cachix/secretspec/secretspec-go` at v0.15.0 + (go/go.mod:22), where `age://` does not resolve and T2's master-key + write-back has no writable target; T0 therefore REQUIRES a bump to `>= + 0.17` with the `age` build feature enabled in the resolved secretspec + build (a separate prerequisite PR, Matt-ruled). On the current pin the + writable providers per the secretspec matrix are `dotenv`/`keyring`, so + absent the bump the target-state `age://` default is unavailable; `age://` + is the target-state default, gated behind the bump. (2) WIRING SEAM — + today serve.go:528 constructs the single resolver with NO provider option + (the SDK default chain); T0 constructs the SERVER resolver with + `secrets.WithProvider()` (resolver.go:75) fed by + a NEW server flag/env carrying that URI (defaulting to the `age://` path), + while serve.go:528's container resolver keeps its existing construction. + The master key is server-minted on first boot and written back to that + provider (T2, unchanged). The six forge secrets are operator-SUPPLIED + values: the operator populates them in the provider — for the `age://` + default, deploy tooling seeds the age file; `compass server-secret set` + (the T0 CLI below) writes a value through `resolver.Set` for rotation on a + running server. The prefix RENAMES them (`LINEAR_FORGE_CLIENT_SECRET` → + `SERVER_LINEAR_FORGE_CLIENT_SECRET`) and the provider keyspace is keyed by + NAME (`setArgs`, resolver.go:258-267), so a value is populated under the + prefixed name via the `serverSecretName()` seam (CONSUMER-REPOINT above). + The NAMES are declared into `server_secrets` at boot from the RESOLVED + config `cfg.Forge.resolved()` (serve.go:232-246) — the same accessor every + live forge consumer uses — gated on the SAME enablement predicates that + gate the forge lanes (`boardIngestionEnabled`, serve.go:224-226, plus the + Linear-configured gate), so a deployment running neither App nor Linear + declares and requires nothing. This boot declare runs on EVERY boot, so it + is idempotent by construction: it tolerates `ErrConflict` from the + non-idempotent `DeclareServerSecret` (secrets.go:100-102) as the + already-declared no-op — the same tolerated-conflict arm `SetSecret` uses + (secrets_service.go:116-118) — so every boot after the first re-declares + the six names cleanly rather than surfacing a duplicate-name error as a + startup failure. Because the operator populates the provider DIRECTLY (an + age file or a cloud secret store — the provider MUST be writable, since + the master key is server-minted and written back through `resolver.Set`; + `env` is read-only in secretspec and so is NOT a valid SERVER-resolver + provider even though it would suffice for the six operator-supplied forge + values) BEFORE the server runs, the six values are present at first boot + with NO running server required to bootstrap them — the R16 + chicken-and-egg (a running server needed to reach the provisioning RPC) is + dissolved. `validateForgeSecret` (serve.go:1013/1016) keeps its hard-fail, + but it is now a clean STATIC deploy-time error: a configured App whose + forge value is absent from the provider fails startup with an actionable + "set `SERVER_` in the provider" message, fixed by populating the + provider and rebooting — never by reaching a not-yet-running RPC. The + complete set is SIX names: the PRIMARY App PEM (`appKeySecret`, + main.go:414-417), the webhook secret (`appWebhook`, main.go:418-421), the + REVIEWER App PEM (`reviewerAppKeySecret`, main.go:430; consumed + serve.go:1644), and the three Linear secrets (client id, client secret, + webhook — per `cfg.Forge.resolved()`, provenance main.go:435-450). + Provisioning is idempotent at the RPC layer: `SetServerSecret` tolerates a + re-provision of an already-declared name — it rewrites the provider value, + mirroring `SetSecret`'s ErrConflict-tolerant arm + (secrets_service.go:117-118), rather than erroring — while store + `DeclareServerSecret` itself mirrors the non-idempotent `DeclareSecret` + (ErrConflict on a duplicate name, secrets.go:100-102). Closes the + pre-existing inject-all exposure (D6) for every configured name once + provisioned. Same PR chain: OQ-4, RESOLVED. +- Consumes: nothing from T1-T5 (pure prerequisite). +- Tests: a secret declared in `server_secrets` is ABSENT from a FetchSecrets + response (because it lives in the other table — no filter involved) while + `secrets` rows in the same boot ARE present; the server resolver instance + RESOLVES it; the container resolver's generated manifest never contains its + name; the new RPC rejects a non-admin caller; the reserved master-key name + is rejected on SetServerSecret/DeleteServerSecret AND on the user-path + SetSecret/DeleteSecret (a non-admin authenticated user calling SetSecret + with a `GATEWAY_CREDENTIALS_`-prefixed name is rejected and the provider + value is unchanged — F1); the server resolver can READ `server_secrets` + through the normal compass_app store path (the GRANT is present — F3); all + six configured names (primary + reviewer App PEM, webhook, and the three + Linear secrets) resolve through the server resolver under their + `SERVER_`-prefixed names and are gone from FetchSecrets; a ROTATION of any + of the six through the `SetServerSecret` RPC rewrites its provider value and + is picked up on the next resolve (the RPC is the rotation path, not the + first-boot bootstrap path); and provisioning is IDEMPOTENT at the RPC layer + — a re-provision of the same name through `SetServerSecret` succeeds and + rewrites the provider value (mirroring `SetSecret`'s ErrConflict-tolerant + arm, secrets_service.go:117-118), with no duplicate-row error surfaced to + the caller; store `DeclareServerSecret` itself returns ErrConflict on a + duplicate name exactly as `DeclareSecret` does (secrets.go:100-102). A + server whose App is UNCONFIGURED (no `--forge-app-id`) and whose Linear is + unset declares and requires none of the six and boots clean. A SECOND boot + of a configured deployment (the six names already present in + `server_secrets`) declares idempotently and boots clean — red if the boot + declare surfaces `ErrConflict` from `DeclareServerSecret` as a startup error + rather than tolerating it as the already-declared no-op. A server with the + App CONFIGURED and the six present in the provider boots with the forge + lanes live; with the App configured but a forge value ABSENT from the + provider, boot fails with an actionable static "set `SERVER_` in the + provider" error (`validateForgeSecret`, serve.go:1013/1016) — fixed by + populating the provider and rebooting, not by reaching a running-server RPC. + A deployment with the Linear pair configured under the DEFAULT names + (`defaultForgeLinearClientIDSecretName` / + `defaultForgeLinearClientSecretName`) and NO flag/env set is provisioned and + resolved under their `SERVER_`-prefixed names via the `serverSecretName()` + seam and is absent from FetchSecrets (red if the resolve side reads the raw + flag/env layer instead of the prefixed `resolved()` name). CONSUMER-REPOINT + positive assertions (the (b)/(c) silent modes need them, since an absent + name is indistinguishable from a legitimate off-state): boot a server with + all six names provisioned and assert the GitHub App token source mints, the + GitHub and Linear webhook handlers are MOUNTED (non-nil), the Linear token + source is non-nil, AND the forge WRITE caller is MOUNTED + (`hub.SetForgeCaller` called / `RelayForgeCall` does not fail-close to + `CodeUnavailable`) — i.e. the forge read AND write lanes still wire off the + server resolver, not just that the names are in `server_secrets`. F1 PREFIX + PARTITION assertions: (user path) a non-admin authenticated caller invoking + `SetSecret`/`DeclareSecret` with a `SERVER_`- or + `GATEWAY_CREDENTIALS_`-prefixed name is REJECTED before `resolver.Set`, no + `secrets` row is created, and the provider value is unchanged; (admin path) + an admin invoking `SetServerSecret` with an UNPREFIXED name is REJECTED + before the `server_secrets` insert and before `resolver.Set`, no + `server_secrets` row is created. Together these assert the two keyspaces are + disjoint by name, so no name can ever be live in both tables — the + structural F1 property, checked with pure string tests, no cross-table or + cross-tenant read. + +### T1 — Envelope-crypto helper package + +New package `go/internal/secrets/envelope` (child of the secrets seam it +serves; no import cycle — it depends on nothing in `secrets`). + +- `Interfaces:` + - `type Key struct{ /* unexported [32]byte */ }` — redacts under + `%s`/`%v`/`%#v` (`String()`/`GoString()` → `"envelope.Key{}"`). + - `func NewKey() (Key, error)` — 32 bytes from `crypto/rand`. + - `func KeyFromBytes(b []byte) (Key, error)` — errors unless len == 32. + - `func (k Key) Seal(plaintext, aad []byte) (ciphertext, nonce []byte, err error)` + — AES-256-GCM, fresh 96-bit random nonce per call; `aad` is + authenticated, not encrypted (binds the ciphertext to its row identity — + T4). + - `func (k Key) Open(ciphertext, nonce, aad []byte) ([]byte, error)` — GCM + auth failure (tamper OR aad mismatch) returns an error naming no + plaintext/key material. + - Key encoding for provider storage: base64(std) of the 32 raw bytes + (SecretSpec values are strings; `Set` rejects empty, resolver.go:225-227). +- Consumes: `crypto/aes`, `crypto/cipher`, `crypto/rand` only. +- Tests: round-trip; tamper (flip a ciphertext/nonce byte → error); `Open` + under a different `aad` → error; nonce uniqueness across calls; redaction + of `Key` under all three verbs; `KeyFromBytes` length validation. + +### T2 — Master-key boot-provision seam + +Boot-time resolve-or-provision, in the server wiring next to the existing +declared-secret consumers (`go/server/serve.go`). DEPENDS ON T0 (the +`server_secrets` store and its resolver instance must exist before the key +is declared into it) and T1. + +- `Interfaces:` + - `func provisionGatewayMasterKey(ctx context.Context, resolver secrets.Resolver, st *store.Store) (envelope.Key, error)` + — `resolver` is the SERVER-SECRET resolver instance (T0). Resolve + `GATEWAY_CREDENTIALS_MASTER_KEY` through it; on absence: + `envelope.NewKey()` → `resolver.Set(ctx, name, encodedKey)` + (resolver.go:219; the value rides stdin, never argv, + resolver.go:206-207) → `st.DeclareServerSecret(ctx, "", name)` with + `declared_by = NULL` (server-provisioned; T0's nullable FK). No + delivery, no kind — those columns do not exist on `server_secrets`. + - **Concurrency — advisory-lock serialized (mandatory):** the whole + resolve→generate→Set→Declare sequence runs under a Postgres advisory + lock (`pg_advisory_xact_lock` on a constant key) — Postgres is the one + store all instances share. This replaces the draft's Set-then-Declare + with tolerated ErrConflict, which was a check-then-set race: two + concurrently booting instances both resolve-absent and both Set (last + writer wins in the provider); the loser's Declare hits ErrConflict, is + tolerated, and that instance proceeds to encrypt with a key the + provider no longer holds — silently undecryptable rows, discovered at + read time. The write path this builds on explicitly disclaims + concurrent safety: "The declare/set/rollback trio is not atomic and + assumes no concurrent same-name writer (the single-Runner MVP: + SetSecret is user-driven CLI)" (go/server/secrets_service.go:88-91). + Ordering inside the lock stays Set-before-Declare: the inverse leaves a + crash-window orphan declaration, and "an orphaned declaration is + required=true in the resolve manifest and would poison EVERY live + session's FetchSecrets" (secrets_service.go:86-88) — under C1 the + blast radius shifts but stays severe: an orphaned `server_secrets` + declaration is required=true in the SERVER resolver's manifest and + would fail every server-side resolve (master key, PEM, webhook, + Linear). A crash between Set + and Declare converges on the next boot (the undeclared name does not + resolve, so the provisioner re-generates, re-Sets, and Declares — no + row was ever encrypted under the orphaned value). + - **Bounded critical section (mandatory — a stuck provider must not wedge + the fleet):** the provider round-trips inside the lock inherit only the + caller's ctx, which at boot is long-lived — but the two halves bound + DIFFERENTLY. `SpecResolver.Set` IS ctx-bounded: it shells out via + `exec.CommandContext(ctx, r.cli, …)` (resolver.go:233), so a ctx deadline + genuinely kills it. `SpecResolver.Resolve` is NOT: it threads ctx only into + `DeclaredSecrets` (resolver.go:136); the actual provider round-trip is + `b.Load()` (resolver.go:165), whose SDK signature carries NO ctx + (`func (b *Builder) Load() (*Resolved, error)`, verified against + secretspec-go v0.15.0 secretspec.go:245, the current pin — re-verify this + signature after the `>= 0.17` bump T0 requires (F2/H-1 prerequisite); if + `Load` gains a ctx-carrying form, the goroutine-offload design below + simplifies to a plain ctx-bounded call) and which blocks in an uncancellable + FFI call + (`nativeResolve` → `C.secretspec_resolve`, binding_cgo.go:30 / + binding_purego.go:118). A hung provider (1Password awaiting biometric + approval, an unreachable Vault, a half-open TCP) would otherwise hold the + transaction-scoped lock indefinitely, and because the key is a shared + constant EVERY other booting instance blocks on it — one stuck provider + becomes a fleet-wide boot wedge. So the provisioner (1) derives a ctx with + an explicit timeout (mirror the 30s `&http.Client{Timeout: 30 * + time.Second}` precedent that already bounds the Linear boot mint at + serve.go:1731) and, because the Resolve-side call cannot be cancelled, RUNS + THE ctx-LESS `Resolve` ON ITS OWN GOROUTINE AND SELECTS ON THAT CTX (`Set` is + ctx-bounded and stays on the parent, inside the lock) — so the + provisioner returns a diagnosable bounded startup error (naming the hung + provider) and its transaction is rolled back, RELEASING the xact-scoped + advisory lock, while the orphaned FFI goroutine is knowingly leaked for the + process's remaining boot-failing lifetime (acceptable: the boot is aborting + anyway). The PARENT goroutine owns the transaction (`pgx.Tx` is not + concurrency-safe): it takes the advisory lock, offloads ONLY the ctx-less + provider READ (`Resolve`) and NEVER a provider write, and on the timeout + branch performs the `Rollback` itself and discards the buffered result + without acting on it. A `Set` reached after the parent's Rollback would land + OUTSIDE the released advisory lock and could overwrite a key another booter + has already provisioned and begun sealing rows under — reintroducing the + silently-undecryptable-rows failure the lock exists to prevent — so the + offloaded goroutine performs no `Set`; the parent runs the (ctx-bounded) + `Set` itself, in-lock, only on the success branch. The offloaded call + reports through a BUFFERED (cap-1) channel so the orphaned FFI goroutine can + complete its send and exit rather than blocking forever on an abandoned + receiver (bounding the leak on a crash-looping boot). And (2) + acquires the lock with `pg_try_advisory_xact_lock` in a + bounded retry loop (or sets a session `lock_timeout`) so a booter that + cannot get the lock fails closed with a diagnosable startup error naming + the contended provisioning lock rather than parking forever. Test (T2): a + provider that never returns yields a bounded, diagnosable boot failure + whose transaction (and advisory lock) is released even though the provider + call itself cannot be cancelled, and a second instance blocked on the lock + also fails bounded rather than hanging. + - **Read-back verify, every boot:** after provisioning AND on every + subsequent boot, re-resolve the name and byte-compare against the key + the process is about to encrypt with; on mismatch, refuse to serve + gateway-credential writes (fail closed). This re-resolve is the SAME + uncancellable `Load` (resolver.go:165) and uses the SAME bounded-offload + path as the provisioning resolve (timeout ctx + own goroutine + buffered + cap-1 channel), so on a steady-state boot — key already provisioned, + nothing to serialize — a hung provider still yields a bounded, diagnosable + startup error rather than a parked process. This is necessary because the + provider keyspace is a shared mutable surface: C1 pins BOTH resolver + instances to the same SecretSpec project + profile (`manifestProject = + "compass"`, `defaultProfile = "default"`, resolver.go:19/23), so the + master key's provider VALUE is reachable by any writer of that + keyspace — an operator's out-of-band `secretspec set`, AND (absent the + guard below) the user-driven `SetSecret` RPC. The verify is a tripwire, + not a boundary: on a plain boot the resolved value IS "the key the + process is about to encrypt with", so a byte-compare cannot by itself + distinguish "my key" from "a swapped key". T5 strengthens it by binding + a non-secret key fingerprint (a salted digest, persisted in the T0 + `server_key_state` row — `key_fingerprint`/`fingerprint_salt`, not a + declared-name registry so the names-only invariant is untouched) so a + swapped provider value is DETECTED at boot rather than silently adopted. + - **Master-key overwrite guard (F1 prefix partition — on EVERY + provider-writing path):** C1 splits the DECLARATION registries (two tables) + but NOT the provider keyspace (one shared profile, above). So the isolation + C1 buys is that server secrets are structurally undeliverable to CONTAINERS + (the manifest separation, §Mechanism C1) — it does NOT by itself make the + master key's provider value unreachable from the user path. The user + `SetSecret`/`DeleteSecret` RPC (`authenticatedOpen`, any authenticated + account — admin_gate.go:122-125) declares into `secrets` but then calls + `resolver.Set`, which shells `secretspec set --profile default` + (resolver.go:258-267) against the SHARED keyspace — so absent a guard a user + calling `SetSecret` with name `GATEWAY_CREDENTIALS_MASTER_KEY` would + OVERWRITE the master key's provider value (the running process keeps its + cached key, but the next boot adopts the attacker-chosen key → every existing + row fails GCM auth, every new row is sealed under a known key: the exact + bulk-disclosure D1 prevents). The F1 prefix guard (D6) closes this: the + master-key name carries the reserved `GATEWAY_CREDENTIALS_` prefix, which the + user `secretsService.SetSecret`/`DeleteSecret` path REJECTS BEFORE + `resolver.Set`/`Delete` — a T0/T2 deliverable, the SAME string check that + enforces the keyspace partition, so no separate membership read is needed. + Tested red-green (a non-admin user calling `SetSecret` with a + `GATEWAY_CREDENTIALS_`-prefixed name is rejected and the provider value is + unchanged). Rotation is OQ-1's machinery, never a raw overwrite through + either surface. The name-keyed global user delete ("a row is keyed by name + alone, not (actor, name)", go/internal/store/secrets.go:150-153) is why the + guard covers the delete path too, once a real provider hard-delete lands. + - **Nil-resolver deployment:** a server built with no secrets surface is + legitimate today ("resolver may be nil on a server built with no + secrets surface (FetchSecrets then fails CodeFailedPrecondition rather + than panicking)", go/internal/runnerhub/handler.go:52-54; agents still + start, go/internal/runner/host.go:367-370). On such a server the + gateway credential store is NOT constructed; enabling the gateway is a + configuration error naming the missing secrets surface; a no-gateway + boot proceeds unchanged. + - Resolve path mirrors `newDeclaredSecretResolver` (serve.go:1469) but is + pointed at the SERVER-SECRET resolver instance and invoked once at + boot; the decoded `envelope.Key` is held in memory for the process + lifetime. + - Boot fails closed: a resolve/provision fault is a startup error, never a + fall-back-to-plaintext. +- Consumes: T0's `server_secrets` store + server resolver instance, T1 + `envelope`, `secrets.Resolver`, `store.DeclareServerSecret`. +- Produces: the process-lifetime `envelope.Key` handed to T4. +- Tests: fresh-boot provisions (Set + Declare called, key usable); + second-boot resolves without Set; two-writer interleaving (concurrent + provisioners converge on ONE key both read back identically); + Set-succeeded/Declare-crashed reboot converges; read-back mismatch → + fail closed; reserved-name SetServerSecret/DeleteServerSecret → + actionable reject; + nil-resolver + gateway enabled → configuration error naming the missing + surface; nil-resolver without gateway → boot proceeds unchanged; + provider fault → boot error. + +### T3 — Schema: ciphertext columns on gateway_credentials + +The `gateway_credentials` migration (owned by the store build the frozen +record plans) carries, for the value payload: + +- `Interfaces:` columns `value_ciphertext BYTEA NOT NULL`, + `value_nonce BYTEA NOT NULL`, `key_version SMALLINT NOT NULL DEFAULT 1`. + NO plaintext value column exists in any migration version — the store is + born encrypted (per the rejected plaintext-first alternative). +- The row's non-secret metadata (provider, scope, owner_user_id, `version` + CAS counter, expiry timestamp if queried-on) stays plaintext-queryable; + ONLY the credential value payload is inside the envelope. Expiry inside + vs. beside the ciphertext is settled by the store build; default: beside + (the gateway lists refreshable rows by expiry without decrypting). +- **Dependency edge, explicit:** this migration task DEPENDS ON T1+T2 + merged (same PR chain or a stated blocking dependency in the store + build's tracker) — the encrypt-before-any-row ordering must be + structural, not a prose sentence. `gateway_credentials` exists nowhere + under `go/` today (grep this session, zero matches), so the ordering is + currently satisfiable with no existing-rows hazard. T5's + ciphertext-at-rest assertion is the named CI tripwire that reds if a + plaintext value column ever appears. +- Tests: migration applies; NOT NULL enforced. + +### T4 — Wire crypto into the CredentialStore read/write path + +Every write path (initial credential save, gateway OAuth-refresh write-back +per design.md:367-371) seals before INSERT/UPDATE; every read path opens +after SELECT. This sits server-side under the RPC surface the frozen record +recommends (design.md:348-358), so the TS gateway never sees the key or the +crypto — it receives plaintext credentials over the stack-token-authenticated +RPC exactly as the frozen record already specifies. + +- `Interfaces:` the store's credential accessors take/return the decrypted + payload type (per D5: secret fields unexported with accessors, or + `slog.LogValuer` + redacting `MarshalJSON` alongside + `String`/`GoString`); the gateway-bound RPC response is built by an + explicit proto/DTO conversion, never by marshaling the payload type. The + `envelope.Key` is a construction-time dependency of the store/service + wrapper, not a per-call parameter. **AAD binds row identity:** every + Seal/Open passes `aad = row primary key + key_version` — the STABLE row + identity, never the CAS `version` counter (which increments per write). + Without AAD an attacker with DB write access can swap two rows' + ciphertext+nonce pairs and both still authenticate — a cross-tenant + credential substitution crossing exactly the boundary the frozen record + names ("the isolation boundary is the per-tenant pool scoping enforced + server-side", design.md:333-337). Writes stamp the current `key_version`; + reads select the key by the row's `key_version` (v1: single live key — a + mismatched version is a diagnosable error naming the expected/found + version numbers, never key material; see OQ-1). +- Consumes: T1, T2's key, T3's columns. +- Tests: see T5. + +### T5 — Tests: ciphertext-at-rest + redaction assertions + +- `Interfaces:` (test-only) + - **Ciphertext-at-rest assertion (the named CI tripwire, T3):** write a + credential through the store, then read the raw row via SQL — assert + the known plaintext substring (e.g. the api key literal) appears + NOWHERE in any column; assert `value_ciphertext != plaintext` and + decrypt-with-key round-trips. This is the guard that reds if a + plaintext value column ever appears. + - **Row-swap assertion (AAD):** swap two rows' ciphertext+nonce pairs at + the SQL level — `Open` MUST fail for both rows (the AAD binds row + identity). + - **Redaction assertion (beyond fmt verbs):** format the + decrypted-payload type and `envelope.Key` under `%s`, `%v`, `%+v`, + `%#v`; `json.Marshal` the payload type; log it through a + `slog.JSONHandler` capture — assert no secret bytes appear in ANY of + these. The fmt verbs alone mirror the existing pattern's intent + (secrets.go:155-156) but do not cover the reflection paths (D5). + - **Key-swap tripwire (F1):** persist a non-secret key fingerprint (a + salted digest of the master key, written to the T0 `server_key_state` row — + `key_fingerprint` + `fingerprint_salt`, alongside `key_version`) at provision + time; the boot read-back verify recomputes it and refuses to serve on + mismatch. Test: swap the + provider value out-of-band, reboot → boot fails closed (the fingerprint + distinguishes "a key" from "my key", which a bare byte-compare cannot). + - Refresh write-back path: refreshed OAuth tokens land re-sealed with a + fresh nonce (nonce differs from the previous row state). + - Wrong-key / tampered-row read → error, no partial plaintext; a + key_version-mismatch error is distinguishable from a GCM auth failure + (names the versions, never key material). + +## Tasks + +- [ ] T0 — `server_secrets` store (mechanism C1): new migration (table + + single-row `server_key_state` table for the key-swap tripwire digest + + GRANT SELECT/INSERT/UPDATE/DELETE on `server_secrets` and + SELECT/INSERT/UPDATE on `server_key_state` to compass_app/compass_system + + bucketA allow-list edit in rls_pgtest_test.go), + `ServerDeclaredSecrets` store view + second SpecResolver instance (same + profile), admin-gated SetServerSecret/DeleteServerSecret RPC carrying + the F1 PREFIX guard (REQUIRE a reserved server-secret prefix — `SERVER_` + or `GATEWAY_CREDENTIALS_` — checked before the `server_secrets` insert + and before `resolver.Set`, AND at the `store.DeclareServerSecret` store + door, so no writer can create a `server_secrets` row under an unprefixed + name) AND the matching user-path half on + `secretsService.SetSecret`/`DeleteSecret` and `store.DeclareSecret` + (REJECT any reserved-prefixed name before `resolver.Set`/`Delete`) — + together the STRUCTURAL F1 PREFIX PARTITION, the two doors making the + keyspaces disjoint by name so no name is ever live in both tables in + either order, a pure string check with no cross-table read; PLUS the + reserved `GATEWAY_CREDENTIALS_MASTER_KEY` name rejected on the admin + path too (rotation is OQ-1 machinery, not a raw overwrite), and operator + provisioning of the SIX server-secret values (primary + reviewer App + PEM, webhook, three Linear): their VALUES live in an operator-chosen + WRITABLE SecretSpec provider (self-hosted default `age://` — writable, + encrypted-at-rest, headless; a cloud store or Vault/OpenBao where + present — F2) resolved by a SERVER resolver constructed + `secrets.WithProvider()` (resolver.go:75) off a + NEW server flag/env for that URI (serve.go:528's container resolver + unchanged), populated by the operator directly (deploy tooling seeds the + age file) or, for rotation on a running server, through the NEW `compass + server-secret set/list/delete` CLI (go/cmd/compass/, mirroring + `newSecretCmd` secret.go:42-49 — value read from stdin never argv per + secret.go:39-41 — dialing the admin + `SetServerSecret`/`DeleteServerSecret` RPCs over `dialSecretsClient`), + which is idempotent at the RPC layer (a re-provision rewrites the + provider value, tolerating ErrConflict per secrets_service.go:117-118; + store `DeclareServerSecret` itself is non-idempotent, mirroring + `DeclareSecret`); the NAMES are declared into `server_secrets` at boot + from the resolved forge config on the ordinary compass_app store path + (`server_secrets` is bucket-A, RLS NOT enabled, so NO BYPASSRLS and NO + cross-tenant read is needed) — a configured App whose forge value is + absent from the provider fails startup with an actionable static "set + `SERVER_` in the provider" error (`validateForgeSecret`, + serve.go:1013/1016), fixed by populating the provider and rebooting, + never by reaching a running-server RPC; an unconfigured App/Linear + declares and requires nothing and boots clean — plus the + `serverSecretName()` prefix seam applied as an INVARIANT (every argument + compared against a resolved `s.Name` is prefix-wrapped — including + `secretDeclared` in `forgeWriteAppsConfigured` serve.go:272-273 (BOTH + the primary and reviewer arms) and `forgeSecretDeclared` serve.go:1078 — + while every log/error/operator-config string is not) so declared and + resolved names agree — PLUS the doc-comment repairs T0's own changes + require: the five "three/3 procedures authenticatedOpen" SecretsService + comments (admin_gate.go:116-121, network_door.go:288-290, + secrets_service.go:6, serve.go:555, serve.go:749-750) are restated as + three authenticatedOpen + two adminOnly once the two admin methods land; + the `adminOnly` type doc (admin_gate.go:22-26) widens its "privileged + CompassService agent-session RPCs (and token issuance)" enumeration to + admit the two `SecretsService` server-secret writes; and the + `SecretsService` service comment + (proto/compass/v1/compass.proto:175-179) is restated to cover the two + admin-gated server-secret methods and to note their authz is door-side + (`adminOnly` in `classifyProcedure`), not handler-side — this one + propagates through `moon run compass-proto:gen` into two checked-in TS + gen trees — `packages/compass-client/src/gen` (the PUBLIC lane, + buf.gen.yaml) and `packages/compass-agent/src/gen` (the internal-only + agent lane, buf.gen.agent-ts.yaml, proto/moon.yml:35-41) — which the + drift gate does NOT catch as staleness on its own (it regenerates and + diffs, so stale comment text regenerates identically and passes). + (Opportunistic in the same edit, flagged pre-existing not T0-caused: + admin_gate.go:40-41's "every generated CompassService and CommsService + procedure" phrasing is already stale — the switch covers SecretsService + too — as is the identical phrasing at classify_exhaustive_test.go:45-46; + correct both in the same pass, but neither gates T0.) Red-green tests: + absent-from-FetchSecrets + server-resolver-resolves + + user-path-prefix-guard (a `SetSecret` on a reserved-prefixed name is + rejected, no `secrets` row created) + admin-path-prefix-guard (a + `SetServerSecret` on an UNPREFIXED name is rejected, no `server_secrets` + row created) + provision-idempotent (a re-provision of the same name via + `SetServerSecret` succeeds and rewrites the provider value, no + duplicate-row error to the caller) + boots-from-provider (an + unconfigured App/Linear boots clean requiring none of the six; a + configured App with the six present in the provider boots with the lanes + live; a configured App with a forge value absent from the provider fails + startup with the actionable static error). DEPENDENCY PREREQUISITE + (gating, Matt-ruled separate PR): bump + `github.com/cachix/secretspec/secretspec-go` from the pinned v0.15.0 + (go/go.mod:22) to `>= 0.17` with the `age` build feature enabled in the + resolved secretspec build — `age` is a 0.17+ build-feature provider, so + on the current pin the `age://` default does not resolve and T2's + master-key write-back has no writable target. PREREQUISITE of T2. Proto + delta: two additive `SecretsService` methods, no enum change — the + checked-in public gen trees are drift-gated, so this needs the + `compass.proto` edit + `moon run compass-proto:gen`, and both new + procedure paths MUST be added to `classifyProcedure` as `adminOnly` or + `classify_exhaustive_test` reds CI + (`go/internal/auth/classify_exhaustive_test.go`). +- [ ] T1 — `go/internal/secrets/envelope`: Key/NewKey/KeyFromBytes/Seal/Open + (AAD-carrying) + redaction + unit tests +- [ ] T2 — boot resolve-or-provision seam via the server-secret resolver + (advisory-lock serialized, read-back verify + key-fingerprint tripwire, + nil-resolver gating, fail-closed boot; the F1 prefix guards are T0's, on BOTH + the admin SetServerSecret/DeleteServerSecret RPC AND the authenticatedOpen + user SetSecret/DeleteSecret path). + DEPENDS ON T0 + T1. +- [ ] T3 — `gateway_credentials` migration columns: `value_ciphertext`, + `value_nonce`, `key_version`; no plaintext column ever. DEPENDS ON + T1+T2 merged (the store build's migration blocks on them). +- [ ] T4 — seal/open wiring in the CredentialStore read/write + refresh + write-back paths; AAD = row PK + key_version; key as + construction-time dependency +- [ ] T5 — ciphertext-at-rest assertion (CI tripwire), row-swap AAD test, + redaction incl. json.Marshal + slog JSONHandler, tamper/wrong-key + tests, and the key-swap tripwire (persist a salted key fingerprint at + provision; boot recompute + fail-closed on mismatch). + +## Open Questions + +- **OQ-1 (deferrable, recommendation: defer): master-key rotation + + row re-encryption.** How does the master key rotate and how are existing + rows re-encrypted? Recommendation: the `key_version` column lands at v1 + (D4 — load-bearing now because retrofitting it is a migration under + ambiguity), but active rotation machinery — lazy re-encrypt on next write + plus a one-shot re-encrypt sweep — is DEFERRABLE to a follow-up record; v1 + runs a single live key version and treats a version mismatch as an error. + Why deferral is SAFE, not merely convenient: (1) provider key loss makes + every row unrecoverable, but the blast radius is BOUNDED — gateway + credentials are re-obtainable from their owners (re-enter the API key, + re-run the OAuth grant; the store holds api_key and OAuth-shaped payloads + a user configures, parent design.md:324-326), so the worst case is a + fleet re-authentication event, never permanent data loss; (2) rotation is + also the nonce-budget release valve, and D1's write-rate arithmetic shows + years of headroom under the 2^32 bound. Carried into T4/T5 regardless of + deferral: a wrong-key / key_version-mismatch decrypt error MUST be + operator-distinguishable from row corruption — the error names the + expected/found key_version numbers, never key material. +- **OQ-2 — RESOLVED (Matt; mechanism updated by the C1 ruling): master-key + containment.** The master key is declared into the physically separate + `server_secrets` store that D6/T0 build, boot-resolved through the + server-secret resolver instance (mirroring the Forge App PEM's + serve.go:1469 pattern), and never materialized into an agent container — + its name is not in the container resolver's manifest at all. An earlier + fold resolved this with a minted SERVER_ONLY delivery kind + FetchSecrets + filter; Matt's C1 ruling replaced that with the separate store (see + Alternatives, mechanism A) and REMOVED the public proto ENUM change it + required (the C1 admin RPC is an additive service-method delta, not a wire + enum change). +- **OQ-3 — RESOLVED (Matt: yes, D7): api_key rows are encrypted identically + to OAuth rows.** One seal/open code path for both payload shapes; both are + secrets of the same sensitivity (a stored api_key is as long-lived and as + disclosure-critical as an OAuth refresh token), and a split path would + leave the most static secret class in plaintext for zero benefit. T3's + single-column schema and T4's single wiring path encode this — there is no + plaintext branch. +- **OQ-4 — RESOLVED (Matt: same PR chain, mechanism C1): the + PEM/webhook/Linear server-secret rows move to `server_secrets` in THIS + record's PR chain**, closing the live pre-existing exposure (they ride + the inject-all path into every agent container today, D6). Because the + reserved-prefix partition (D6) renames the six under `SERVER_`, and the + deployment is pre-production, the ALREADY-declared rows are WIPED and + re-provisioned under their prefixed names rather than migrated in place: the + shared profile is name-keyed, so the rename requires the operator to populate + each value in the provider under the new prefixed name (a provider write, not + a free row move). The NAMES are declared into `server_secrets` at boot from + the resolved forge config, gated on the same predicates that enable the forge + lanes; the VALUES live in an operator-chosen writable provider (self-hosted + default `age://`; a cloud store or Vault/OpenBao where present — F2), populated + by the operator directly (deploy tooling seeds the age file) or through the + `compass server-secret` CLI / admin `SetServerSecret` RPC (T0) for rotation on + a running server. Because the provider is populated BEFORE the server runs, the + six values are present at first boot with no running server required to + bootstrap them (Matt-ruled — the server must be able to boot with the secrets + set through the provider/config, not depend on a running server to reach a + provisioning RPC). +- **OQ-5 — WITHDRAWN (Matt: prefix partition, no reconcile): the DL-315 + BYPASSRLS-allow-list widening is no longer needed.** An earlier draft of this + record moved the six server-secret rows with a boot reconcile that had to run + under `store.WithSystemRole` (BYPASSRLS) to see cross-tenant `secrets` rows, + which would have added a BYPASSRLS entrypoint beyond the set DL-315 governs. + Matt's reserved-prefix partition (D6) removed the reconcile entirely — server + secrets are re-declared under `SERVER_`-prefixed names on the ordinary + compass_app store path (`server_secrets` is bucket-A, no RLS), so this record + adds NO cross-tenant BYPASSRLS site and needs no DL-315 amendment. The encryption row landed as + DL-328 (concurrent merges took the intervening numbers: RIG-3070's + runner-trust-model split took DL-325, RIG-3170's runner-records scrub chain + took DL-326 for the P2 session-volume clone model, and the sibling + SubjectService PR takes DL-327). This record adds no BYPASSRLS site, + so there is no separate DL-315-amendment row. The separate, PRE-EXISTING observation that DL-315's own + entrypoint enumeration is stale (the shipped `hub.go:814` forge-notification-ack + arm is a BYPASSRLS site DL-315's four names omit — `WithSystemRole`'s own doc + comment already lists it, tenant_tx.go:41-47) is not this record's to fix, since + this record no longer touches the BYPASSRLS surface; it is left as a pre-existing + DL-315 ledger-prose observation for a separate follow-up, not tracked by this + record.