feat(kms): authenticate RA-TLS clients by attestation instead of by issuer

CHANGELOG.md

@@ -29,6 +29,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - sdk: the Go and Python compose-hash helpers silently dropped every app-compose field they did not declare, so `getComposeHash` returned a digest for an app-compose that was not the one being deployed — and that digest is what gets whitelisted on chain. The missing fields are named above; both now keep unrecognised keys as well, so a guest that gains a field before the SDK does still hashes correctly
 
 ### Changed
+- kms: client certificates are authenticated by the attestation they carry rather than by their issuer. Rocket configures mutual TLS through rustls' `WebPkiClientVerifier`, which pins a CA — but an RA-TLS certificate is self-issued and carries its identity in a TEE quote, so there is nothing to chain to. `GetTempCaCert` bridged the gap by handing every caller a shared CA private key purely so the minted certificate would chain somewhere; the CA established nothing (its key is public by design, and the endpoint is unauthenticated) and the check that has always carried the meaning is the quote verification that runs afterwards. The KMS now hands rustls a verifier that requires an attestation and ignores the issuer. Nothing changes for callers: guests and KMS-to-KMS onboarding still mint their client certificates from the temp CA, and those are now accepted for the attestation they carry. What changes is that the TLS layer went from admitting any certificate signed by a public key to requiring an attested one, and that a self-issued certificate is now accepted — which is what lets callers be migrated off `GetTempCaCert` in a follow-up. `[rpc.tls.mutual]` is no longer the trust anchor and is dropped from `kms.toml` and the KMS config templates; leaving it in an existing deployment's config is inert. The gateway's `[tls.mutual]` is unaffected — it pins the KMS root CA, which is a real trust anchor
 - guest-agent: the `/metrics` exposition gains a conventions-compliant `dstack_guest_*` series set (application prefix; `_bytes`/`_seconds` unit suffixes; no `_total` suffix on gauges, which reads as a counter to every tool; the four OS/kernel/CPU gauges folded into one `dstack_guest_info`; `disk_used_ratio` 0–1 instead of a percentage). The old `system_*`/`disk_*` names are still emitted verbatim, marked deprecated, and will be removed in a future release — this endpoint is tenant-facing, so existing dashboards get a migration window
 - dstack-util: a CVM re-registers with the gateway node that last accepted it, before falling back to the configured order. The list used to be walked from the top every time, so every CVM piled onto the first URL and the whole fleet snapped back to it the moment it recovered from an outage — and each move rewrites the instance record from a different node's memory
 - vmm: optionally randomize the KMS and gateway URL orders written to each CVM's system configuration so new CVMs distribute their initial requests across service nodes; both are enabled by default in `vmm.toml`

docs/security/public-security-reports.md

@@ -37,7 +37,7 @@ These issues were filed as concrete vulnerability reports, security audit findin
 | [#558](https://github.com/Dstack-TEE/dstack/issues/558) Path traversal in KMS `remove_cache` | Closed | Valid report, fixed | [#601](https://github.com/Dstack-TEE/dstack/pull/601) validates cache paths before deletion |
 | [#559](https://github.com/Dstack-TEE/dstack/issues/559) Zero `mr_config_id` bypasses verification and weakens `mr_aggregated` identity | Closed | Not a production vulnerability | Zero `mr_config_id` remains an unset-value compatibility case, and configuration changes are still reflected through RTMR-based measurements. No code fix was applied |
 | [#560](https://github.com/Dstack-TEE/dstack/issues/560) Admin token comparison not constant-time | Closed | Not a production vulnerability | The comparison is over a SHA-256 digest of a high-entropy token, not the raw token. No code fix was applied |
-| [#561](https://github.com/Dstack-TEE/dstack/issues/561) KMS TLS client certificates are non-mandatory in Rocket config | Closed | Valid report, documented | The TLS listener allows unauthenticated bootstrap, temp-CA bootstrap, and public endpoints. `GetTempCaCert` returns temp CA private material for bootstrap. App/KMS key release requires verified caller attestation, and certificate signing verifies the CSR signature and embedded attestation. No code fix was applied |
+| [#561](https://github.com/Dstack-TEE/dstack/issues/561) KMS TLS client certificates are non-mandatory in Rocket config | Closed | Valid report, fixed | [#1106](https://github.com/Dstack-TEE/dstack/pull/1106) makes the KMS verify client certificates by the attestation they carry instead of by an issuer CA, so a presented certificate must now be attested at the handshake. The listener still allows unauthenticated bootstrap and public endpoints. App/KMS key release requires verified caller attestation, and certificate signing verifies the CSR signature and embedded attestation |
 | [#562](https://github.com/Dstack-TEE/dstack/issues/562) Configfs path overridable through an environment variable | Closed | Not a production vulnerability | A process that can choose its own quote path is already inside the measured CVM behavior. No code fix has been applied. A production guard for `DCAP_TDX_QUOTE_CONFIGFS_PATH` remains possible hardening |
 | [#563](https://github.com/Dstack-TEE/dstack/issues/563) `simulate_quote` runtime path in production guest agent | Closed | Valid report, fixed | [#582](https://github.com/Dstack-TEE/dstack/pull/582) isolates the simulator into a dedicated binary |
 | [#564](https://github.com/Dstack-TEE/dstack/issues/564) `GetAppEnvEncryptPubKey` unauthenticated app ID enumeration | Closed | Not a production vulnerability | The RPC returns a public encryption key before an app has an attested identity, and `app_id` is not treated as secret. No code fix was applied |
@@ -59,7 +59,7 @@ These issues were filed as concrete vulnerability reports, security audit findin
 | [#616](https://github.com/Dstack-TEE/dstack/issues/616) Host-controlled Docker registry mirror enables image substitution attacks | Closed | Not a production vulnerability | Registry mirrors are untrusted transport. Digest-pinned image references and measured compose configuration protect against substitution. No code fix was applied |
 | [#617](https://github.com/Dstack-TEE/dstack/issues/617) Guest agent exposes raw private keys to all local processes | Closed | Not a production vulnerability | dstack treats a CVM as one application trust domain. It does not provide per-container key isolation inside the same measured application. No code fix was applied |
 | [#618](https://github.com/Dstack-TEE/dstack/issues/618) Disk encryption disableable via kernel cmdline, not measured in RTMR | Closed | Not a production vulnerability | The kernel command line is measured into RTMR2, so changing `dstack.storage_encrypted=false` changes attestation evidence. No code fix was applied |
-| [#619](https://github.com/Dstack-TEE/dstack/issues/619) KMS `get_temp_ca_cert` returns temp CA private key without authentication | Closed | Duplicate | The report duplicates the private advisory response for the temp CA bootstrap flow |
+| [#619](https://github.com/Dstack-TEE/dstack/issues/619) KMS `get_temp_ca_cert` returns temp CA private key without authentication | Closed | Duplicate | The report duplicates the private advisory response for the temp CA bootstrap flow. [#1106](https://github.com/Dstack-TEE/dstack/pull/1106) removes the reason the temp CA existed - the KMS no longer pins it - so callers can be migrated off it; the RPC itself is retained until guests and KMS-to-KMS onboarding are |
 
 ## Related security roadmap and hardening
 

docs/security/security-best-practices.md

@@ -111,7 +111,7 @@ Development settings are intentionally easy to audit, but they are not productio
 - The KMS contract pins a concrete gateway app id. Do not use `gateway_app_id = "any"` for production traffic.
 - TEE quotes are evaluated by deployment policy, including TCB status and expected OS/application measurements.
 
-The KMS TLS listener may keep `rpc.tls.mutual.mandatory = false` because bootstrap, temp-CA bootstrap, and public metadata endpoints need to be reachable before a client has an RA-TLS certificate. `GetTempCaCert` returns temp CA private material for the bootstrap flow; treat it as bootstrap-sensitive.
+The KMS TLS listener verifies client certificates by the attestation they carry rather than by an issuer CA, so it needs no `rpc.tls.mutual` section. It still accepts connections without a client certificate, because bootstrap and public metadata endpoints must be reachable before a client has an RA-TLS certificate. `GetTempCaCert` remains in use by guests and by KMS-to-KMS onboarding, which still mint their client certificates from that CA; it returns temp CA private material, so treat it as bootstrap-sensitive.
 
 App key release and KMS key handover still require verified caller attestation from the RA-TLS client certificate. Certificate signing verifies the CSR signature and embedded attestation before signing.
 

docs/security/security-model.md

@@ -337,11 +337,11 @@ Production verifiers should reject deployments that use these development settin
 
 ### KMS mTLS is route-enforced for sensitive operations
 
-The KMS Rocket TLS listener permits connections without a client certificate because some bootstrap and public metadata endpoints must be reachable before a client has an RA-TLS certificate. That listener setting is not the authorization boundary for key material.
+The KMS Rocket TLS listener permits connections without a client certificate because some bootstrap and public metadata endpoints must be reachable before a client has an RA-TLS certificate. A certificate that is presented must carry an attestation, but the issuer that signed it is not checked and is not the authorization boundary for key material.
 
 App key release and KMS key handover require verified caller attestation from the RA-TLS client certificate. Certificate signing verifies the CSR signature and the attestation embedded in the CSR before signing.
 
-The unauthenticated or non-client-certificate surface includes bootstrap and temp-CA bootstrap material retrieval, env-encryption public-key retrieval, metadata, health, and metrics behavior documented for operators. `GetTempCaCert` returns temp CA private material for the bootstrap flow, so operators must treat it as bootstrap-sensitive rather than harmless public metadata.
+The unauthenticated or non-client-certificate surface includes bootstrap and temp-CA bootstrap material retrieval, env-encryption public-key retrieval, metadata, health, and metrics behavior documented for operators. `GetTempCaCert` returns temp CA private material and remains in use by guests and by KMS-to-KMS onboarding, which mint their client certificates from that CA; operators must treat it as bootstrap-sensitive rather than harmless public metadata.
 
 ## Limitations
 

docs/tutorials/kms-build-configuration.md

@@ -183,14 +183,11 @@ port = 9100
 key = "/etc/kms/certs/rpc.key"
 certs = "/etc/kms/certs/rpc.crt"
 
-# Mutual TLS (mTLS) Configuration
-[rpc.tls.mutual]
-ca_certs = "/etc/kms/certs/tmp-ca.crt"
-# Keep the TLS listener optional because bootstrap/public endpoints must be
-# reachable before a client has an RA-TLS certificate. Temp-CA bootstrap material
-# is bootstrap-sensitive. Key-release RPCs still require verified caller
+# No mutual-TLS section: client certificates are verified by the attestation they
+# carry, not by an issuer CA, so there is nothing to pin. Connections without a
+# certificate are still accepted, because bootstrap and public endpoints must be
+# reachable before a client has one. Key-release RPCs still require verified caller
 # attestation; certificate signing verifies CSR signature and attestation.
-mandatory = false
 
 # Core KMS Configuration
 [core]
@@ -457,14 +454,11 @@ port = 9100
 key = "/etc/kms/certs/rpc.key"
 certs = "/etc/kms/certs/rpc.crt"
 
-# Mutual TLS (mTLS) Configuration
-[rpc.tls.mutual]
-ca_certs = "/etc/kms/certs/tmp-ca.crt"
-# Keep the TLS listener optional because bootstrap/public endpoints must be
-# reachable before a client has an RA-TLS certificate. Temp-CA bootstrap material
-# is bootstrap-sensitive. Key-release RPCs still require verified caller
+# No mutual-TLS section: client certificates are verified by the attestation they
+# carry, not by an issuer CA, so there is nothing to pin. Connections without a
+# certificate are still accepted, because bootstrap and public endpoints must be
+# reachable before a client has one. Key-release RPCs still require verified caller
 # attestation; certificate signing verifies CSR signature and attestation.
-mandatory = false
 
 # Core KMS Configuration
 [core]

docs/tutorials/kms-cvm-deployment.md

@@ -73,8 +73,8 @@ Inside the CVM at `/etc/kms/certs/`:
 | `root-ca.key` | Root CA signing key (P256 ECDSA) |
 | `rpc.crt` | TLS certificate for RPC server |
 | `rpc.key` | RPC server private key |
-| `tmp-ca.crt` | Temporary CA for mutual TLS |
-| `tmp-ca.key` | Temporary CA private key |
+| `tmp-ca.crt` | Temp CA, served by the legacy `GetTempCaCert` bootstrap RPC |
+| `tmp-ca.key` | Temp CA private key, served alongside it |
 | `root-k256.key` | Ethereum signing key (secp256k1) |
 | `bootstrap-info.json` | Public keys and TDX attestation quote |
 
@@ -165,13 +165,11 @@ configs:
       key = "/etc/kms/certs/rpc.key"
       certs = "/etc/kms/certs/rpc.crt"
 
-      [rpc.tls.mutual]
-      ca_certs = "/etc/kms/certs/tmp-ca.crt"
-      # Keep the TLS listener optional because bootstrap/public endpoints must be
-      # reachable before a client has an RA-TLS certificate. Temp-CA bootstrap
-      # material is bootstrap-sensitive. Key-release RPCs still require verified
-      # caller attestation; certificate signing verifies CSR signature and attestation.
-      mandatory = false
+      # No mutual-TLS section: client certificates are verified by the attestation
+      # they carry, not by an issuer CA. Connections without a certificate are still
+      # accepted, because bootstrap and public endpoints must be reachable before a
+      # client has one. Key-release RPCs still require verified caller attestation;
+      # certificate signing verifies CSR signature and attestation.
 
       [core]
       cert_dir = "/etc/kms/certs"

dstack/crates/dstack-cli-core/src/config.rs

@@ -130,10 +130,6 @@ port = 8000
 key = "/kms/certs/rpc.key"
 certs = "/kms/certs/rpc.crt"
 
-[rpc.tls.mutual]
-ca_certs = "/kms/certs/tmp-ca.crt"
-mandatory = false
-
 [core]
 cert_dir = "/kms/certs"
 # single-node: the KMS does not self-attest to its own auth API before

dstack/kms/dstack-app/compose-dev.yaml

@@ -57,10 +57,6 @@ configs:
       key = "/kms/certs/rpc.key"
       certs = "/kms/certs/rpc.crt"
 
-      [rpc.tls.mutual]
-      ca_certs = "/kms/certs/tmp-ca.crt"
-      mandatory = false
-
       [core]
       cert_dir = "/kms/certs"
 

dstack/kms/dstack-app/compose-simple.yaml

@@ -33,10 +33,6 @@ configs:
       key = "/kms/certs/rpc.key"
       certs = "/kms/certs/rpc.crt"
 
-      [rpc.tls.mutual]
-      ca_certs = "/kms/certs/tmp-ca.crt"
-      mandatory = false
-
       [core]
       cert_dir = "/kms/certs"
 

dstack/kms/kms.toml

@@ -18,9 +18,9 @@ port = 8000
 key = "/etc/kms/certs/rpc.key"
 certs = "/etc/kms/certs/rpc.crt"
 
-[rpc.tls.mutual]
-ca_certs = "/etc/kms/certs/tmp-ca.crt"
-mandatory = false
+# No `[rpc.tls.mutual]`: client certificates are verified by the attestation they
+# carry, not by an issuer CA (see ra-rpc's ratls_client_verifier). Leaving a
+# `mutual` section here is harmless — it is no longer the trust anchor.
 
 [core]
 cert_dir = "/etc/kms/certs"

dstack/kms/rpc/proto/kms_rpc.proto

@@ -74,6 +74,11 @@ message KmsKeyResponse {
   repeated KmsKeys keys = 2;
 }
 
+// See the note on `GetTempCaCert`.
+//
+// `temp_ca_key` is a CA private key handed to any caller, so it authenticates
+// nobody. It exists because the KMS pinned this CA for mutual TLS, which left a
+// self-issued RA-TLS certificate with nothing to chain to.
 message GetTempCaCertResponse {
   string temp_ca_cert = 1;
   string temp_ca_key = 2;
@@ -101,7 +106,19 @@ service KMS {
   rpc GetAppEnvEncryptPubKey(AppId) returns (PublicKeyResponse);
   // Request the KMS instance metadata for use as a probe and health check.
   rpc GetMeta(google.protobuf.Empty) returns (GetMetaResponse);
-  // Request the temporary CA certificate and key
+  // Request the temporary CA certificate and key.
+  //
+  // Both current callers - guests at boot, and KMS-to-KMS onboarding - fetch this CA
+  // and mint their client certificate from it, because the KMS used to pin the CA for
+  // mutual TLS and a self-issued certificate had nothing to chain to.
+  //
+  // That pin is gone: the KMS now verifies client certificates by the attestation
+  // they carry and ignores the issuer, so a self-issued certificate would be accepted
+  // on the same terms. Neither caller has been migrated yet, so this RPC still has to
+  // work; it is scheduled for removal once both are.
+  //
+  // Do not build new callers on it. A caller that only needs the KMS root CA can read
+  // `ca_cert` from `GetMeta` instead.
   rpc GetTempCaCert(google.protobuf.Empty) returns (GetTempCaCertResponse);
   // Sign a certificate
   rpc SignCert(SignCertRequest) returns (SignCertResponse);

dstack/kms/src/main.rs

@@ -6,12 +6,14 @@ use anyhow::{anyhow, bail, Context, Result};
 use clap::Parser;
 use config::KmsConfig;
 use main_service::{KmsState, RpcHandler};
+use ra_rpc::ratls_client_verifier::RaTlsClientAuth;
 use ra_rpc::rocket_helper::QuoteVerifier;
 use ra_tls::attestation::AttestationVerifier;
 use rocket::{
     fairing::AdHoc,
     figment::{providers::Serialized, Figment},
     response::content::{RawHtml, RawText},
+    tls::Resolver as _,
     Shutdown, State,
 };
 use tracing::{info, warn};
@@ -180,6 +182,11 @@ async fn main() -> Result<()> {
         .clone()
         .merge(Serialized::defaults(figment.find_value("rpc")?));
     let mut rocket = rocket::custom(figment)
+        // Verify client certificates by their attestation rather than by issuer. The
+        // certificates guests and onboarding mint from the temp CA today keep working
+        // unchanged - they are now accepted for the attestation they carry rather than
+        // for who signed them - and a self-issued certificate would be accepted too.
+        .attach(RaTlsClientAuth::fairing())
         .attach(AdHoc::on_response("Add app version header", |_req, res| {
             Box::pin(async move {
                 res.set_raw_header("X-App-Version", app_version());

dstack/kms/src/main_service.rs

@@ -494,6 +494,20 @@ impl KmsRpc for RpcHandler {
         })
     }
 
+    /// Serve the temp CA certificate and key.
+    ///
+    /// Both current callers - guests at boot, and KMS-to-KMS onboarding
+    /// ([`crate::onboard_service`]) - fetch this CA and mint their client certificate
+    /// from it, because the KMS used to pin the CA for mutual TLS and a self-issued
+    /// certificate had nothing to chain to.
+    ///
+    /// That pin is gone: client certificates are now verified by the attestation they
+    /// carry (`ra_rpc::ratls_client_verifier`), so a self-issued certificate would be
+    /// accepted on the same terms. Neither caller has been migrated yet, so this RPC
+    /// still has to work.
+    ///
+    /// The key it returns authenticates nobody: it is handed to any caller. Removing
+    /// this RPC needs both callers migrated first.
     async fn get_temp_ca_cert(self) -> Result<GetTempCaCertResponse> {
         let self_boot_info = self
             .ensure_self_allowed()