feat(guest-agent): let Attest return the boot-time GPU attestation evidence

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 - sdk: `verify_signature` and `verify_signature_chain` in all four SDKs, replacing the removed guest-agent `Verify` RPC. `verify_signature_chain` is new capability rather than a port: it walks all three links of a `Sign` signature chain -- payload signature, the app root key attesting `"{purpose}:{hex(pubkey)}"`, and the KMS root attesting that app root for this `app_id` -- and requires the chain to anchor at a KMS root public key **the caller supplies**. That anchor has to come from somewhere independently trusted (the `DstackKms` contract's `kmsInfo().k256Pubkey`, or a pinned value); read it from the KMS being checked and an attacker who can answer that query can also mint a self-consistent chain. The four ports are pinned against one committed set of test vectors, `sdk/tests/vectors/signature_chain.json`, generated from the real KMS and guest-agent primitives -- this repo has shipped cross-language crypto drift twice already
+- guest-agent: `Attest` accepts `include_boottime_gpu_evidence` and returns the boot-time GPU attestation evidence in `AttestResponse.boottime_gpu_evidence`, so a verifier can fetch the quote and the GPU evidence in one round trip instead of also calling `GpuInfo`. Exposed in the Rust, Python, Go and JS SDKs
 - sdk: `AppCompose` in the Go SDK gained `init_script`, `storage_fs`, `swap_size`, `event_log_version`, `port_policy` and `verity_volumes`, and `Requirements` gained `gpu_policy` in the Go and Python SDKs
 - shared API authentication (`dstack-api-auth`) protecting the full VMM HTTP/pRPC/UI surface and unifying Gateway/KMS admin auth: bearer/`X-Admin-Token`/HTTP Basic/bcrypt htpasswd, constant-time verification (#796)
 - gateway: `Admin.Status` reports `health_gating`, so an operator can see whether this node's health polling is switched on. With it off, instances that opted in sit at `unknown` forever and are all in rotation, which is otherwise indistinguishable on the dashboard from being held out pending a first answer

docs/attestation-tdx.md

@@ -45,7 +45,7 @@ evaluation, `gpu-attestation`. The `gpu-policy-hash` payload is
 The `gpu-attestation` payload is JSON containing the verified device count,
 CC/DevTools state, and `evidence_sha256`.
 
-The guest-agent `GpuInfo` API returns the complete `nvattest` JSON captured during boot. It is not trustworthy by itself. After verifying the TDX quote and replaying the event log to RTMR3, hash the exact UTF-8 bytes of `GpuInfo.attestation` and require the result to equal the `gpu-attestation` event's `evidence_sha256`. See [GPU Security for AI Workloads](./security/security-model.md#gpu-security-for-ai-workloads) for the event schema, ordering, Rego example, and platform differences.
+The guest-agent `GpuInfo` API returns the complete `nvattest` JSON captured during boot; `Attest` returns the same bytes in `boottime_gpu_evidence` when called with `include_boottime_gpu_evidence`, so a verifier can fetch the quote and the GPU evidence in one round trip. It is not trustworthy by itself. After verifying the TDX quote and replaying the event log to RTMR3, hash the exact UTF-8 bytes of `GpuInfo.attestation` (or `Attest.boottime_gpu_evidence`) and require the result to equal the `gpu-attestation` event's `evidence_sha256`. See [GPU Security for AI Workloads](./security/security-model.md#gpu-security-for-ai-workloads) for the event schema, ordering, Rego example, and platform differences.
 
 ### 2.2. Determining expected MRs
 MRTD, RTMR0, RTMR1, and RTMR2 correspond to the image. dstack OS builds all related software from source.

docs/security/security-model.md

@@ -144,7 +144,7 @@ boot-mr-done
 }
 ```
 
-`GpuInfo` returns that complete boot-time `nvattest` JSON in its `attestation` string; it does not run a new attestation. To bind the API result to TDX evidence: verify the quote, replay the event log to the quote's RTMR3, require exactly one pre-`system-ready` `gpu-attestation` event, decode its JSON payload, and compare `evidence_sha256` with `SHA-256(UTF-8(GpuInfo.attestation))`. Only after this comparison should the verifier inspect the returned claims. This exact-byte comparison includes any whitespace or trailing newline in the returned string.
+`GpuInfo` returns that complete boot-time `nvattest` JSON in its `attestation` string, and `Attest` returns the same bytes in `boottime_gpu_evidence` when called with `include_boottime_gpu_evidence`; neither runs a new attestation. To bind the API result to TDX evidence: verify the quote, replay the event log to the quote's RTMR3, require exactly one pre-`system-ready` `gpu-attestation` event, decode its JSON payload, and compare `evidence_sha256` with `SHA-256(UTF-8(GpuInfo.attestation))` (equivalently `SHA-256(UTF-8(Attest.boottime_gpu_evidence))`). Only after this comparison should the verifier inspect the returned claims. This exact-byte comparison includes any whitespace or trailing newline in the returned string.
 
 A verifier must replay the measured event log, require exactly one `gpu-policy-hash` event immediately after `compose-hash`, and compare its 32-byte payload with the expected policy digest (`SHA-256(JCS({}))` for the omitted/default policy). When MrConfigV3 includes `gpu_policy_hash`, it must match the same digest. When GPU protection is required, the verifier must also require exactly one pre-`system-ready` `gpu-attestation` event with `devices > 0` and, when applicable, the expected deployment count. The raw `attestation.out` file is not trusted by itself; if it is supplied for inspection, its digest must match the `gpu-attestation` event.
 

dstack/cert-client/src/bin/dstack-kms-sign-cert-fixture.rs

@@ -3,7 +3,7 @@
 //! Generate a v2 KMS CSR whose key is bound to fresh guest attestation.
 
 use anyhow::{Context, Result};
-use dstack_guest_agent_rpc::{dstack_guest_client::DstackGuestClient, RawQuoteArgs};
+use dstack_guest_agent_rpc::{dstack_guest_client::DstackGuestClient, AttestArgs};
 use http_client::prpc::PrpcClient;
 use ra_tls::{
     attestation::{PlatformEvidence, QuoteContentType, VersionedAttestation},
@@ -31,8 +31,9 @@ async fn main() -> Result<()> {
     let address = dstack_types::dstack_agent_address();
     let client = DstackGuestClient::new(PrpcClient::new(address));
     let response = client
-        .attest(RawQuoteArgs {
+        .attest(AttestArgs {
             report_data: report_data.to_vec(),
+            include_boottime_gpu_evidence: false,
         })
         .await
         .context("failed to obtain key-bound guest attestation")?;

dstack/gateway/src/distributed_certbot.rs

@@ -12,7 +12,7 @@ use std::time::Duration;
 
 use anyhow::{bail, Context, Result};
 use certbot::{AcmeClient, Dns01Client};
-use dstack_guest_agent_rpc::RawQuoteArgs;
+use dstack_guest_agent_rpc::{AttestArgs, RawQuoteArgs};
 use ra_tls::attestation::QuoteContentType;
 use ra_tls::rcgen::KeyPair;
 use tokio::sync::Mutex;
@@ -658,7 +658,13 @@ impl DistributedCertBot {
         };
 
         // Get attestation
-        let attestation_str = match agent.attest(RawQuoteArgs { report_data }).await {
+        let attestation_str = match agent
+            .attest(AttestArgs {
+                report_data,
+                include_boottime_gpu_evidence: false,
+            })
+            .await
+        {
             Ok(resp) => serde_json::to_string(&resp).unwrap_or_default(),
             Err(err) => {
                 warn!("failed to get attestation for ACME account: {err:?}");
@@ -734,7 +740,13 @@ impl DistributedCertBot {
         };
 
         // Get attestation
-        let attestation = match agent.attest(RawQuoteArgs { report_data }).await {
+        let attestation = match agent
+            .attest(AttestArgs {
+                report_data,
+                include_boottime_gpu_evidence: false,
+            })
+            .await
+        {
             Ok(resp) => serde_json::to_string(&resp).unwrap_or_default(),
             Err(err) => {
                 warn!(domain, "failed to get attestation: {err:?}");

dstack/guest-agent-simulator/src/main.rs

@@ -13,7 +13,7 @@ use dstack_guest_agent::{
     config::{self, Config},
     run_server, AppState,
 };
-use dstack_guest_agent_rpc::{AttestResponse, GetQuoteResponse};
+use dstack_guest_agent_rpc::GetQuoteResponse;
 use mock_attestation::tdx::TdxGenerator;
 use ra_tls::attestation::VersionedAttestation;
 use serde::Deserialize;
@@ -102,7 +102,7 @@ impl PlatformBackend for SimulatorPlatform {
         )
     }
 
-    fn attest_response(&self, report_data: [u8; 64]) -> Result<AttestResponse> {
+    fn attest_cvm(&self, report_data: [u8; 64]) -> Result<VersionedAttestation> {
         simulator::simulated_attest_response(
             &self.attestation,
             report_data,
@@ -183,9 +183,13 @@ mod tests {
     fn simulator_attest_response_preserves_legacy_wire_format() {
         let platform = load_fixture_platform();
         let report_data = [0x5a; 64];
-        let response = platform.attest_response(report_data).unwrap();
-        assert_eq!(response.attestation.first(), Some(&0x00));
-        let patched = VersionedAttestation::from_bytes(&response.attestation)
+        let encoded = platform
+            .attest_cvm(report_data)
+            .unwrap()
+            .to_bytes()
+            .unwrap();
+        assert_eq!(encoded.first(), Some(&0x00));
+        let patched = VersionedAttestation::from_bytes(&encoded)
             .unwrap()
             .into_v1();
         assert_eq!(patched.report_data().unwrap(), report_data);
@@ -296,7 +300,7 @@ mod tests {
         // relying parties on GCP at Attest.
         let attested = simulator::simulated_attest_response(&gcp_tdx, report_data, true, None)
             .expect("Attest must work on GCP TDX too");
-        let round_tripped = VersionedAttestation::from_bytes(&attested.attestation)
+        let round_tripped = VersionedAttestation::from_bytes(&attested.to_bytes().unwrap())
             .unwrap()
             .into_v1();
         assert!(round_tripped.platform.tpm_quote().is_some());
@@ -312,8 +316,12 @@ mod tests {
         let original = fixture.clone().into_v1().report_data().unwrap();
         let platform = SimulatorPlatform::new(fixture, false, None).unwrap();
         let report_data = [0x5a; 64];
-        let response = platform.attest_response(report_data).unwrap();
-        let patched = VersionedAttestation::from_bytes(&response.attestation)
+        let encoded = platform
+            .attest_cvm(report_data)
+            .unwrap()
+            .to_bytes()
+            .unwrap();
+        let patched = VersionedAttestation::from_bytes(&encoded)
             .unwrap()
             .into_v1();
         assert_eq!(patched.report_data().unwrap(), original);

dstack/guest-agent-simulator/src/simulator.rs

@@ -6,7 +6,7 @@ use std::path::Path;
 
 use anyhow::{anyhow, Context, Result};
 use dcap_qvl::quote::Quote;
-use dstack_guest_agent_rpc::{AttestResponse, GetQuoteResponse};
+use dstack_guest_agent_rpc::GetQuoteResponse;
 use mock_attestation::tdx::TdxGenerator;
 use ra_tls::attestation::{
     AttestationV1, PlatformEvidence, QuoteContentType, TdxAttestationExt, VersionedAttestation,
@@ -59,20 +59,17 @@ pub fn simulated_attest_response(
     report_data: [u8; 64],
     patch_report_data: bool,
     generator: Option<&TdxGenerator>,
-) -> Result<AttestResponse> {
+) -> Result<VersionedAttestation> {
     let preserve_legacy = matches!(source, VersionedAttestation::V0 { .. });
     let mut attestation =
         prepare_attestation(source, report_data, patch_report_data, generator, "attest")?;
     if let Some(event_log) = attestation.platform.tdx_event_log_mut() {
         cc_eventlog::tdx::fill_v2_preimages(event_log);
     }
-    let attestation = if preserve_legacy {
+    Ok(if preserve_legacy {
         attestation.try_into_legacy()?.into_versioned()
     } else {
         VersionedAttestation::V1 { attestation }
-    };
-    Ok(AttestResponse {
-        attestation: attestation.to_bytes()?,
     })
 }
 

dstack/guest-agent/rpc/proto/agent_rpc.proto

@@ -53,7 +53,7 @@ service DstackGuest {
 
   // Generates a versioned attestation with the given report data.
   // Returns a dstack-defined attestation format that supports different attestation modes across platforms.
-  rpc Attest(RawQuoteArgs) returns (AttestResponse) {}
+  rpc Attest(AttestArgs) returns (AttestResponse) {}
 
   // Get app info
   rpc Info(google.protobuf.Empty) returns (AppInfo) {}
@@ -180,6 +180,22 @@ message RawQuoteArgs {
   bytes report_data = 1;
 }
 
+// The request to get a versioned attestation
+message AttestArgs {
+  // 64 bytes of report data
+  bytes report_data = 1;
+  // Field 2 and 3 carried `include_ccel` and `include_preimages` while this RPC
+  // took a `RawQuoteArgs`. Both were bools, so reusing either number here would
+  // make a pre-0.6.0 client's `include_ccel = true` arrive as a request for
+  // something else entirely.
+  reserved 2, 3;
+  reserved "include_ccel", "include_preimages";
+  // Also return the boot-time GPU attestation evidence in
+  // `boottime_gpu_evidence`. This does not sample the GPU now and does not
+  // answer `report_data`; see that field.
+  bool include_boottime_gpu_evidence = 4;
+}
+
 message TdxQuoteResponse {
   // TDX quote
   bytes quote = 1;
@@ -196,6 +212,21 @@ message TdxQuoteResponse {
 message AttestResponse {
   // The attestation
   bytes attestation = 1;
+  // Complete JSON output produced by nvattest at boot, the same bytes `GpuInfo`
+  // serves. Only `DstackGuest.Attest` populates it, and only when the request set
+  // `include_boottime_gpu_evidence` and boot-time GPU attestation output exists.
+  //
+  // Not bound to `report_data`: nvattest ran at boot against its own nonce, so a
+  // fresh `report_data` says nothing about it. Bind it by replaying the runtime
+  // event log and comparing sha256 of these exact UTF-8 bytes against the
+  // `evidence_sha256` field of the measured `gpu-attestation` event.
+  //
+  // This is a historical statement about the boot, not a live one: it does not
+  // prove the GPU is still attached. Sampling the GPU at attestation time would
+  // not fix that -- an NVIDIA report binds the device and a nonce but not the TD
+  // the device is attached to, so a fresh report can be relayed from a genuine
+  // remote GPU. Only TDISP/TEE-IO device binding closes that.
+  string boottime_gpu_evidence = 2;
 }
 
 message GpuInfoResponse {

dstack/guest-agent/src/backend.rs

@@ -3,15 +3,18 @@
 // SPDX-License-Identifier: Apache-2.0
 
 use anyhow::{Context, Result};
-use dstack_guest_agent_rpc::{AttestResponse, GetQuoteResponse};
+use dstack_guest_agent_rpc::GetQuoteResponse;
 use ra_tls::attestation::Attestation;
 use ra_tls::attestation::{QuoteContentType, VersionedAttestation};
 
 pub trait PlatformBackend: Send + Sync {
     fn attestation_for_info(&self) -> Result<VersionedAttestation>;
     fn certificate_attestation(&self, pubkey: &[u8]) -> Result<VersionedAttestation>;
     fn quote_response(&self, report_data: [u8; 64], vm_config: &str) -> Result<GetQuoteResponse>;
-    fn attest_response(&self, report_data: [u8; 64]) -> Result<AttestResponse>;
+    /// Attest the CVM itself: the attestation `Attest` and `AttestAppKey`
+    /// return, with digest preimages filled in. Encoding it is the RPC
+    /// layer's job.
+    fn attest_cvm(&self, report_data: [u8; 64]) -> Result<VersionedAttestation>;
 }
 
 #[derive(Debug, Default)]
@@ -44,12 +47,10 @@ impl PlatformBackend for RealPlatform {
         })
     }
 
-    fn attest_response(&self, report_data: [u8; 64]) -> Result<AttestResponse> {
+    fn attest_cvm(&self, report_data: [u8; 64]) -> Result<VersionedAttestation> {
         let mut attestation =
             Attestation::quote(&report_data).context("Failed to get attestation")?;
         attestation.fill_event_preimages();
-        Ok(AttestResponse {
-            attestation: attestation.into_versioned().to_bytes()?,
-        })
+        Ok(attestation.into_versioned())
     }
 }

dstack/guest-agent/src/rpc_service.rs

@@ -15,10 +15,10 @@ use dstack_guest_agent_rpc::{
     dstack_guest_server::{DstackGuestRpc, DstackGuestServer},
     tappd_server::{TappdRpc, TappdServer},
     worker_server::{WorkerRpc, WorkerServer},
-    AppInfo, AttestAppKeyRequest, AttestResponse, DeriveK256KeyResponse, DeriveKeyArgs, GetKeyArgs,
-    GetKeyResponse, GetQuoteResponse, GetTlsKeyArgs, GetTlsKeyResponse, GpuInfoResponse,
-    HealthResponse, RawQuoteArgs, SignRequest, SignResponse, TdxQuoteArgs, TdxQuoteResponse,
-    WorkerVersion,
+    AppInfo, AttestAppKeyRequest, AttestArgs, AttestResponse, DeriveK256KeyResponse, DeriveKeyArgs,
+    GetKeyArgs, GetKeyResponse, GetQuoteResponse, GetTlsKeyArgs, GetTlsKeyResponse,
+    GpuInfoResponse, HealthResponse, RawQuoteArgs, SignRequest, SignResponse, TdxQuoteArgs,
+    TdxQuoteResponse, WorkerVersion,
 };
 use dstack_types::{AppKeys, SysConfig, GPU_ATTESTATION_OUTPUT};
 use ed25519_dalek::ed25519::signature::hazmat::PrehashSigner;
@@ -65,6 +65,16 @@ fn read_gpu_attestation(path: &Path) -> String {
     }
 }
 
+/// GPU evidence to return alongside an attestation. Opt-in, so a caller that
+/// does not care about GPUs neither pays the disk read nor carries the payload.
+fn boottime_gpu_evidence(include: bool, path: &Path) -> String {
+    if include {
+        read_gpu_attestation(path)
+    } else {
+        String::new()
+    }
+}
+
 #[derive(Clone)]
 pub struct AppState {
     inner: Arc<AppStateInner>,
@@ -216,8 +226,8 @@ impl AppState {
             .quote_response(report_data, &self.inner.vm_config)
     }
 
-    fn attest_response(&self, report_data: [u8; 64]) -> Result<AttestResponse> {
-        self.inner.platform.attest_response(report_data)
+    fn attest_cvm(&self, report_data: [u8; 64]) -> Result<Vec<u8>> {
+        self.inner.platform.attest_cvm(report_data)?.to_bytes()
     }
 }
 
@@ -443,9 +453,15 @@ impl DstackGuestRpc for InternalRpcHandler {
         })
     }
 
-    async fn attest(self, request: RawQuoteArgs) -> Result<AttestResponse> {
+    async fn attest(self, request: AttestArgs) -> Result<AttestResponse> {
         let report_data = pad64(&request.report_data).context("Report data is too long")?;
-        self.state.attest_response(report_data)
+        Ok(AttestResponse {
+            attestation: self.state.attest_cvm(report_data)?,
+            boottime_gpu_evidence: boottime_gpu_evidence(
+                request.include_boottime_gpu_evidence,
+                Path::new(GPU_ATTESTATION_OUTPUT),
+            ),
+        })
     }
 
     async fn version(self) -> Result<WorkerVersion> {
@@ -641,7 +657,12 @@ impl WorkerRpc for ExternalRpcHandler {
 
     async fn attest_app_key(self, request: AttestAppKeyRequest) -> Result<AttestResponse> {
         let report_data = self.app_key_report_data(&request.algorithm).await?;
-        self.state.attest_response(report_data)
+        Ok(AttestResponse {
+            attestation: self.state.attest_cvm(report_data)?,
+            // This method attests a key, not the machine. A caller that wants
+            // the boot-time GPU evidence asks `Attest` or `GpuInfo` for it.
+            boottime_gpu_evidence: String::new(),
+        })
     }
 }
 
@@ -748,6 +769,17 @@ mod tests {
         assert_eq!(read_gpu_attestation(output.path()), attestation);
     }
 
+    #[test]
+    fn attest_returns_boottime_gpu_evidence_only_when_requested() {
+        let mut output = tempfile::NamedTempFile::new().unwrap();
+        let evidence = r#"{"result_code":0,"claims":[]}"#;
+        output.write_all(evidence.as_bytes()).unwrap();
+        output.flush().unwrap();
+
+        assert_eq!(boottime_gpu_evidence(true, output.path()), evidence);
+        assert_eq!(boottime_gpu_evidence(false, output.path()), "");
+    }
+
     #[test]
     fn missing_gpu_attestation_output_reads_as_empty() {
         let dir = tempfile::tempdir().unwrap();
@@ -951,11 +983,9 @@ pNs85uhOZE8z2jr8Pg==
                 })
             }
 
-            fn attest_response(&self, report_data: [u8; 64]) -> Result<AttestResponse> {
+            fn attest_cvm(&self, report_data: [u8; 64]) -> Result<VersionedAttestation> {
                 let attestation = patch_report_data(&self.attestation, report_data);
-                Ok(AttestResponse {
-                    attestation: VersionedAttestation::V1 { attestation }.to_bytes()?,
-                })
+                Ok(VersionedAttestation::V1 { attestation })
             }
         }