Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- sdk: the Go and Python compose-hash helpers silently dropped every app-compose field they did not declare, so `getComposeHash` returned a digest for an app-compose that was not the one being deployed — and that digest is what gets whitelisted on chain. The missing fields are named above; both now keep unrecognised keys as well, so a guest that gains a field before the SDK does still hashes correctly

### Changed
- dstack-util, cert-client, kms: guests and KMS-to-KMS onboarding mint self-issued RA-TLS client certificates instead of fetching the temp CA and minting from it. The quote inside the certificate is the identity the KMS authenticates, so no CA material is needed; the KMS root CA now comes from `GetMeta`, over the same unauthenticated connection as before and pinned afterwards by `verify_key_provider_id` against the measured `app_compose.key_provider_id`. `.appkeys.json` no longer carries `tmp_ca_key`/`tmp_ca_cert`; readers ignore them in files written by older images. Onboarding does the same and drops the `GetTempCaCert` round trip entirely: with no CA to fetch first, it now runs over a single client that carries both the client certificate and the `cert_validator`, so the source KMS's attestation is verified on the handshake that actually carries the root key rather than on an earlier, separate connection - `RaClient::new_mtls`, its only user, set no validator at all and is removed. The source is re-checked against local policy (`ensure_kms_allowed`) after the fetch instead of before it. `GetTempCaCert` now has no caller in this tree and is retained only so guest images built before the switch keep booting. **This narrows which peers work**: a guest or a joining KMS built from this change is refused at the TLS handshake by any KMS that still pins the temp CA, which is every release up to and including 0.5.11. So a guest needs a KMS at 0.6.0 or later, and onboarding needs a *source* at 0.6.0 or later - upgrading a 0.5.x cluster past 0.6.0 goes through 0.6.0 as a bridge hop, the same two-hop shape 0.5.4 already needs. Guests and KMS nodes built before this change are unaffected and keep working against both
- kms: client certificates are authenticated by the attestation they carry rather than by their issuer. Rocket configures mutual TLS through rustls' `WebPkiClientVerifier`, which pins a CA — but an RA-TLS certificate is self-issued and carries its identity in a TEE quote, so there is nothing to chain to. `GetTempCaCert` bridged the gap by handing every caller a shared CA private key purely so the minted certificate would chain somewhere; the CA established nothing (its key is public by design, and the endpoint is unauthenticated) and the check that has always carried the meaning is the quote verification that runs afterwards. The KMS now hands rustls a verifier that requires an attestation and ignores the issuer. Nothing changes for callers: guests and KMS-to-KMS onboarding still mint their client certificates from the temp CA, and those are now accepted for the attestation they carry. What changes is that the TLS layer went from admitting any certificate signed by a public key to requiring an attested one, and that a self-issued certificate is now accepted — which is what lets callers be migrated off `GetTempCaCert` in a follow-up. `[rpc.tls.mutual]` is no longer the trust anchor and is dropped from `kms.toml` and the KMS config templates; leaving it in an existing deployment's config is inert. The gateway's `[tls.mutual]` is unaffected — it pins the KMS root CA, which is a real trust anchor
- guest-agent: the `/metrics` exposition gains a conventions-compliant `dstack_guest_*` series set (application prefix; `_bytes`/`_seconds` unit suffixes; no `_total` suffix on gauges, which reads as a counter to every tool; the four OS/kernel/CPU gauges folded into one `dstack_guest_info`; `disk_used_ratio` 0–1 instead of a percentage). The old `system_*`/`disk_*` names are still emitted verbatim, marked deprecated, and will be removed in a future release — this endpoint is tenant-facing, so existing dashboards get a migration window
- dstack-util: a CVM re-registers with the gateway node that last accepted it, before falling back to the configured order. The list used to be walked from the top every time, so every CVM piled onto the first URL and the whole fleet snapped back to it the moment it recovered from an outage — and each move rewrites the instance record from a different node's memory
Expand Down
14 changes: 9 additions & 5 deletions docs/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -430,13 +430,17 @@ Additional KMS instances can onboard from an existing KMS to share the same root
**How it works:**

1. New KMS starts in onboard mode (empty `auto_bootstrap_domain`)
2. New KMS calls `GetTempCaCert` on source KMS
3. New KMS generates RA-TLS certificate with TDX quote
4. New KMS calls `GetKmsKey` with mTLS authentication
5. Source KMS verifies attestation via `bootAuth/kms` webhook
6. If approved, source KMS returns root keys
2. New KMS generates a self-issued RA-TLS certificate with a TDX quote
3. New KMS calls `GetKmsKey` with mTLS authentication
4. Source KMS verifies attestation via `bootAuth/kms` webhook
5. If approved, source KMS returns root keys
6. New KMS checks the source against its own policy before adopting the keys
7. Both KMS instances now derive identical keys

The source KMS must be running 0.6.0 or later, because it has to accept a self-issued
client certificate. Onboarding from an older source is not supported; upgrade that source
to 0.6.0 first.

**Configure new KMS for onboarding:**

```toml
Expand Down
10 changes: 6 additions & 4 deletions docs/encrypted-env-spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,9 +170,7 @@ Path inside TEE: `/dstack/.host-shared/.appkeys.json`
"key_provider": {
"Kms": {
"url": "https://kms.example.com/prpc",
"pubkey": "...",
"tmp_ca_key": "-----BEGIN PRIVATE KEY-----\n...",
"tmp_ca_cert": "-----BEGIN CERTIFICATE-----\n..."
"pubkey": "..."
}
}
}
Expand Down Expand Up @@ -200,11 +198,15 @@ Rust externally tagged enum — an object with exactly one key:
{"None": {"key": "<PEM>"}}
{"Local": {"key": "<PEM>", "mr": "<hex>"}}
{"Tpm": {"key": "<PEM>", "pubkey": "<hex>"}}
{"Kms": {"url": "...", "pubkey": "<hex>", "tmp_ca_key": "<PEM>", "tmp_ca_cert": "<PEM>"}}
{"Kms": {"url": "...", "pubkey": "<hex>"}}
```

The tag is one of `"None"` / `"Local"` / `"Tpm"` / `"Kms"`.

`Kms` carried `tmp_ca_key` / `tmp_ca_cert` before guests switched to self-issued
RA-TLS client certificates. Readers ignore the extra fields, so files written by
older guest images still parse.

## Runtime File/Path Contract (dstack)

For dstack runtime integration, treat these names/locations as protocol-level
Expand Down
2 changes: 1 addition & 1 deletion docs/security/public-security-reports.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ These issues were filed as concrete vulnerability reports, security audit findin
| [#616](https://github.com/Dstack-TEE/dstack/issues/616) Host-controlled Docker registry mirror enables image substitution attacks | Closed | Not a production vulnerability | Registry mirrors are untrusted transport. Digest-pinned image references and measured compose configuration protect against substitution. No code fix was applied |
| [#617](https://github.com/Dstack-TEE/dstack/issues/617) Guest agent exposes raw private keys to all local processes | Closed | Not a production vulnerability | dstack treats a CVM as one application trust domain. It does not provide per-container key isolation inside the same measured application. No code fix was applied |
| [#618](https://github.com/Dstack-TEE/dstack/issues/618) Disk encryption disableable via kernel cmdline, not measured in RTMR | Closed | Not a production vulnerability | The kernel command line is measured into RTMR2, so changing `dstack.storage_encrypted=false` changes attestation evidence. No code fix was applied |
| [#619](https://github.com/Dstack-TEE/dstack/issues/619) KMS `get_temp_ca_cert` returns temp CA private key without authentication | Closed | Duplicate | The report duplicates the private advisory response for the temp CA bootstrap flow. [#1106](https://github.com/Dstack-TEE/dstack/pull/1106) removes the reason the temp CA existed - the KMS no longer pins it - so callers can be migrated off it; the RPC itself is retained until guests and KMS-to-KMS onboarding are |
| [#619](https://github.com/Dstack-TEE/dstack/issues/619) KMS `get_temp_ca_cert` returns temp CA private key without authentication | Closed | Duplicate | The report duplicates the private advisory response for the temp CA bootstrap flow. [#1106](https://github.com/Dstack-TEE/dstack/pull/1106) removes the reason the temp CA existed - the KMS no longer pins it - and guests and KMS-to-KMS onboarding now mint self-issued certificates, so the RPC has no caller left in-tree; it is retained only so guest images built before that switch keep booting |

## Related security roadmap and hardening

Expand Down
2 changes: 1 addition & 1 deletion docs/security/security-best-practices.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ Development settings are intentionally easy to audit, but they are not productio
- The KMS contract pins a concrete gateway app id. Do not use `gateway_app_id = "any"` for production traffic.
- TEE quotes are evaluated by deployment policy, including TCB status and expected OS/application measurements.

The KMS TLS listener verifies client certificates by the attestation they carry rather than by an issuer CA, so it needs no `rpc.tls.mutual` section. It still accepts connections without a client certificate, because bootstrap and public metadata endpoints must be reachable before a client has an RA-TLS certificate. `GetTempCaCert` remains in use by guests and by KMS-to-KMS onboarding, which still mint their client certificates from that CA; it returns temp CA private material, so treat it as bootstrap-sensitive.
The KMS TLS listener verifies client certificates by the attestation they carry rather than by an issuer CA, so it needs no `rpc.tls.mutual` section. It still accepts connections without a client certificate, because bootstrap and public metadata endpoints must be reachable before a client has an RA-TLS certificate. `GetTempCaCert` has no caller left in-tree - guests and KMS-to-KMS onboarding both mint self-issued certificates - but is retained so guest images built before that switch keep booting; it returns temp CA private material, so treat it as bootstrap-sensitive.

App key release and KMS key handover still require verified caller attestation from the RA-TLS client certificate. Certificate signing verifies the CSR signature and embedded attestation before signing.

Expand Down
2 changes: 1 addition & 1 deletion docs/security/security-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -341,7 +341,7 @@ The KMS Rocket TLS listener permits connections without a client certificate bec

App key release and KMS key handover require verified caller attestation from the RA-TLS client certificate. Certificate signing verifies the CSR signature and the attestation embedded in the CSR before signing.

The unauthenticated or non-client-certificate surface includes bootstrap and temp-CA bootstrap material retrieval, env-encryption public-key retrieval, metadata, health, and metrics behavior documented for operators. `GetTempCaCert` returns temp CA private material and remains in use by guests and by KMS-to-KMS onboarding, which mint their client certificates from that CA; operators must treat it as bootstrap-sensitive rather than harmless public metadata.
The unauthenticated or non-client-certificate surface includes bootstrap and temp-CA bootstrap material retrieval, env-encryption public-key retrieval, metadata, health, and metrics behavior documented for operators. `GetTempCaCert` returns temp CA private material. It has no caller left in-tree - guests and KMS-to-KMS onboarding both mint self-issued certificates - but is retained so guest images built before that switch keep booting; operators must treat it as bootstrap-sensitive rather than harmless public metadata.

## Limitations

Expand Down
15 changes: 6 additions & 9 deletions dstack/cert-client/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ use dstack_types::{AppKeys, KeyProvider};
use ra_rpc::client::{RaClient, RaClientConfig};
use ra_tls::{
attestation::AttestationVerifier,
cert::{generate_ra_cert, CaCert, CertSigningRequestV2},
cert::{generate_self_signed_ra_cert, CaCert, CertSigningRequestV2},
};

pub enum CertRequestClient {
Expand Down Expand Up @@ -70,14 +70,11 @@ impl CertRequestClient {
.context("Failed to create CA")?;
Ok(CertRequestClient::Local { ca: Box::new(ca) })
}
KeyProvider::Kms {
url,
tmp_ca_key,
tmp_ca_cert,
..
} => {
let client_cert = generate_ra_cert(tmp_ca_cert.clone(), tmp_ca_key.clone())
.context("Failed to generate RA cert")?;
KeyProvider::Kms { url, .. } => {
// Self-issued: the KMS authenticates the quote inside this certificate,
// not whoever signed it.
let client_cert =
generate_self_signed_ra_cert(None).context("Failed to generate RA cert")?;
let ra_client = RaClientConfig::builder()
.remote_uri(url.clone())
.tls_client_cert(client_cert.cert_pem)
Expand Down
4 changes: 0 additions & 4 deletions dstack/dstack-types/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2263,8 +2263,6 @@ pub enum KeyProvider {
url: String,
#[serde(with = "hex_bytes")]
pubkey: Vec<u8>,
tmp_ca_key: String,
tmp_ca_cert: String,
},
}

Expand Down Expand Up @@ -2326,8 +2324,6 @@ mod key_provider_tests {
let kms = KeyProvider::Kms {
url: "https://kms.example".into(),
pubkey: vec![0xab; 32],
tmp_ca_key: String::new(),
tmp_ca_cert: String::new(),
};
assert_eq!(kms.id(), &[0xab; 32]);
}
Expand Down
34 changes: 14 additions & 20 deletions dstack/dstack-util/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ use k256::schnorr::SigningKey;
use ra_rpc::Attestation;
use ra_tls::{
attestation::{AttestationQuote, QuoteContentType, VersionedAttestation},
cert::{generate_ra_cert, generate_ra_cert_with_app_id},
cert::{generate_ra_cert, generate_self_signed_ra_cert},
kdf::{derive_key, derive_p256_key_pair_from_bytes},
rcgen::KeyPair,
};
Expand Down Expand Up @@ -633,13 +633,13 @@ async fn cmd_get_keys(args: GetKeysArgs) -> Result<()> {
None
};

// Step 1: Get temporary CA certificate
// Step 1: Fetch the KMS root CA
eprintln!("Connecting to KMS: {kms_url}");
let tls_no_check = root_ca_pem.is_none();
if tls_no_check {
eprintln!("Warning: no --root-ca provided, TLS certificate verification is disabled for initial connection");
}
let tmp_ca = {
let root_ca = {
let client = RaClientConfig::builder()
.remote_uri(kms_url.clone())
.tls_no_check(tls_no_check)
Expand All @@ -648,21 +648,17 @@ async fn cmd_get_keys(args: GetKeysArgs) -> Result<()> {
.build()
.into_client()
.context("failed to create client")?;
let kms_client = KmsClient::new(client);
kms_client
.get_temp_ca_cert()
KmsClient::new(client)
.get_meta()
.await
.context("Failed to get temp CA cert")?
.context("Failed to get KMS meta")?
.ca_cert
};

// Step 2: Generate RA-TLS client certificate
// Step 2: Generate a self-issued RA-TLS client certificate. The quote inside it
// is the identity the KMS authenticates, so no CA material is needed.
let app_id = decode_app_id(args.app_id.as_deref())?;
let cert_pair = generate_ra_cert_with_app_id(
tmp_ca.temp_ca_cert.clone(),
tmp_ca.temp_ca_key.clone(),
app_id,
)
.context("Failed to generate RA cert")?;
let cert_pair = generate_self_signed_ra_cert(app_id).context("Failed to generate RA cert")?;

// Step 3: Create authenticated client and request app keys
let ra_client = RaClientConfig::builder()
Expand All @@ -671,7 +667,7 @@ async fn cmd_get_keys(args: GetKeysArgs) -> Result<()> {
.remote_uri(kms_url.clone())
.tls_client_cert(cert_pair.cert_pem)
.tls_client_key(cert_pair.key_pem)
.tls_ca_cert(tmp_ca.ca_cert.clone())
.tls_ca_cert(root_ca.clone())
.build()
.into_client()
.context("Failed to create RA client")?;
Expand All @@ -686,13 +682,13 @@ async fn cmd_get_keys(args: GetKeysArgs) -> Result<()> {
.context("Failed to get app key")?;

// Step 4: Build AppKeys structure
let (_, ca_pem) = x509_parser::pem::parse_x509_pem(tmp_ca.ca_cert.as_bytes())
.context("Failed to parse CA cert")?;
let (_, ca_pem) =
x509_parser::pem::parse_x509_pem(root_ca.as_bytes()).context("Failed to parse CA cert")?;
let x509 = ca_pem.parse_x509().context("Failed to parse CA cert")?;
let root_pubkey = x509.public_key().raw.to_vec();

let keys = utils::AppKeys {
ca_cert: tmp_ca.ca_cert,
ca_cert: root_ca,
disk_crypt_key: response.disk_crypt_key,
env_crypt_key: response.env_crypt_key,
k256_key: response.k256_key,
Expand All @@ -701,8 +697,6 @@ async fn cmd_get_keys(args: GetKeysArgs) -> Result<()> {
key_provider: KeyProvider::Kms {
url: kms_url,
pubkey: root_pubkey,
tmp_ca_key: tmp_ca.temp_ca_key,
tmp_ca_cert: tmp_ca.temp_ca_cert,
},
};

Expand Down
Loading
Loading