feat(guest-agent): collect on-demand GPU attestation evidence

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 - sdk: `verify_signature` and `verify_signature_chain` in all four SDKs, replacing the removed guest-agent `Verify` RPC. `verify_signature_chain` is new capability rather than a port: it walks all three links of a `Sign` signature chain -- payload signature, the app root key attesting `"{purpose}:{hex(pubkey)}"`, and the KMS root attesting that app root for this `app_id` -- and requires the chain to anchor at a KMS root public key **the caller supplies**. That anchor has to come from somewhere independently trusted (the `DstackKms` contract's `kmsInfo().k256Pubkey`, or a pinned value); read it from the KMS being checked and an attacker who can answer that query can also mint a self-consistent chain. The four ports are pinned against one committed set of test vectors, `sdk/tests/vectors/signature_chain.json`, generated from the real KMS and guest-agent primitives -- this repo has shipped cross-language crypto drift twice already
+- guest-agent: `AttestGpu` collects vendor-native GPU evidence on demand against a caller-supplied 32-byte nonce. It returns opaque, versioned evidence bundles identified by vendor and format for independent appraisal. The response format is extensible to additional GPU vendors. Exposed in the Rust, Python, Go, and JS SDKs
 - guest-agent: `Attest` accepts `include_boottime_gpu_evidence` and returns the boot-time GPU attestation evidence in `AttestResponse.boottime_gpu_evidence`, so a verifier can fetch the quote and the GPU evidence in one round trip instead of also calling `GpuInfo`. Exposed in the Rust, Python, Go and JS SDKs
 - sdk: `AppCompose` in the Go SDK gained `init_script`, `storage_fs`, `swap_size`, `event_log_version`, `port_policy` and `verity_volumes`, and `Requirements` gained `gpu_policy` in the Go and Python SDKs
 - shared API authentication (`dstack-api-auth`) protecting the full VMM HTTP/pRPC/UI surface and unifying Gateway/KMS admin auth: bearer/`X-Admin-Token`/HTTP Basic/bcrypt htpasswd, constant-time verification (#796)

docs/attestation-tdx.md

@@ -45,7 +45,7 @@ evaluation, `gpu-attestation`. The `gpu-policy-hash` payload is
 The `gpu-attestation` payload is JSON containing the verified device count,
 CC/DevTools state, and `evidence_sha256`.
 
-The guest-agent `GpuInfo` API returns the complete `nvattest` JSON captured during boot; `Attest` returns the same bytes in `boottime_gpu_evidence` when called with `include_boottime_gpu_evidence`, so a verifier can fetch the quote and the GPU evidence in one round trip. It is not trustworthy by itself. After verifying the TDX quote and replaying the event log to RTMR3, hash the exact UTF-8 bytes of `GpuInfo.attestation` (or `Attest.boottime_gpu_evidence`) and require the result to equal the `gpu-attestation` event's `evidence_sha256`. See [GPU Security for AI Workloads](./security/security-model.md#gpu-security-for-ai-workloads) for the event schema, ordering, Rego example, and platform differences.
+The guest-agent `GpuInfo` API returns the complete `nvattest` JSON captured during boot; `Attest` returns the same bytes in `boottime_gpu_evidence` when called with `include_boottime_gpu_evidence`, so a verifier can fetch the quote and the GPU evidence in one round trip. It is not trustworthy by itself. (`AttestGpu` runs a *fresh* attestation against a caller nonce, but its result is not bound to the TD and must not be used as remote evidence; only the boot-time record below is.) After verifying the TDX quote and replaying the event log to RTMR3, hash the exact UTF-8 bytes of `GpuInfo.attestation` (or `Attest.boottime_gpu_evidence`) and require the result to equal the `gpu-attestation` event's `evidence_sha256`. See [GPU Security for AI Workloads](./security/security-model.md#gpu-security-for-ai-workloads) for the event schema, ordering, Rego example, and platform differences.
 
 ### 2.2. Determining expected MRs
 MRTD, RTMR0, RTMR1, and RTMR2 correspond to the image. dstack OS builds all related software from source.

docs/security/security-model.md

@@ -158,7 +158,7 @@ The events make the following **boot-time** statement: immediately before key pr
 - A mixed launch cannot attest only its TEE-capable subset. Non-NVIDIA display GPUs are rejected, and the sysfs, NVML, and nvattest claim counts must all agree. A non-CC NVIDIA GPU either prevents evidence collection/appraisal or causes the default appraisal, application policy, or CC-state check to fail.
 - Copying another CVM's result into a file or `report_data` does not work. Only measured pre-application code can place the event before `system-ready`, and event-log replay binds it to the quoted RTMR/PCR value.
 
-This is **not a lifetime or physical co-location guarantee**. After `system-ready`, an application with sufficient guest privileges can unload the NVIDIA driver, and a malicious host may attempt PCI hot-remove/replacement or proxy GPU traffic. The boot event remains a true historical statement but does not prove that the same device is still attached. dstack also cannot rule out a live relay/cuckoo attack to a genuine remote GPU: current Hopper/Blackwell deployments do not provide a CPU-TEE-verifiable TEE-I/O/TDISP device binding. Applications that mutate the driver or PCI topology are outside this guarantee; higher-assurance deployments must prevent that behavior and re-attest before using a newly initialized GPU.
+This is **not a lifetime or physical co-location guarantee**. After `system-ready`, an application with sufficient guest privileges can unload the NVIDIA driver, and a malicious host may attempt PCI hot-remove/replacement or proxy GPU traffic. The boot event remains a true historical statement but does not prove that the same device is still attached. dstack also cannot rule out a live relay/cuckoo attack to a genuine remote GPU: current Hopper/Blackwell deployments do not provide a CPU-TEE-verifiable TEE-I/O/TDISP device binding. Applications that mutate the driver or PCI topology are outside this guarantee; higher-assurance deployments must prevent that behavior and re-attest before using a newly initialized GPU; the guest-agent `AttestGpu` API does that re-check against a caller-chosen nonce. Being an NVIDIA report, its result is subject to the same relay caveat as any attestation-time GPU sample: it establishes that a genuine CC-enabled GPU is reachable and responsive now, not that the device is bound to this TD, so it must not be forwarded to a remote relying party as proof of GPU possession.
 
 AMD SEV-SNP has no runtime measurement register in the current dstack stack. The local boot gate can still fail closed, but a `gpu-attestation` event carried beside an SNP report is not remotely bound to that report and must not be accepted as dual-attestation evidence. SNP needs a measured vTPM/PCR channel before it can provide the same remote binding.
 

dstack/Cargo.toml

@@ -59,6 +59,7 @@ members = [
     "serde-duration",
     "dstack-mr",
     "dstack-mr/cli",
+    "nvattest",
     "nvidia-attest-proxy",
     "verifier",
     "size-parser",
@@ -121,6 +122,7 @@ serde-duration = { path = "serde-duration" }
 dstack-mr = { path = "dstack-mr" }
 dstack-verifier = { path = "verifier", default-features = false }
 size-parser = { path = "size-parser" }
+nvattest = { path = "nvattest" }
 wavekv = "2.1"
 
 # Core dependencies

dstack/dstack-util/Cargo.toml

@@ -10,6 +10,7 @@ edition.workspace = true
 license.workspace = true
 
 [dependencies]
+nvattest.workspace = true
 aes-gcm.workspace = true
 anyhow.workspace = true
 clap.workspace = true
@@ -68,4 +69,5 @@ safe-write.workspace = true
 errify.workspace = true
 
 [dev-dependencies]
+base64.workspace = true
 rand.workspace = true

dstack/dstack-util/src/system_setup.rs

@@ -1434,11 +1434,8 @@ async fn do_sys_setup(stage0: Stage0<'_>) -> Result<()> {
 mod gpu {
     use super::*;
 
-    const NVATTEST: &str = "/usr/bin/nvattest";
-    const ATTESTATION_TIMEOUT: Duration = Duration::from_secs(300);
     const EVENT_VERSION: u32 = 2;
     const POLICY_ENTRYPOINT: &str = "data.policy.nv_match";
-    const TRUST_OUTPOST_POLICY: &str = "/usr/share/nvattest/policies/allow_trust_outpost_ocsp.rego";
     /// Bound Rego evaluation so a runaway application policy cannot hang boot.
     const POLICY_TIMEOUT: Duration = Duration::from_secs(10);
 
@@ -1578,23 +1575,6 @@ mod gpu {
         Ok(inventory.nvidia)
     }
 
-    /// Run a GPU tool with a bounded timeout so a wedged driver/GPU cannot
-    /// hang the boot indefinitely (dstack-prepare is a oneshot unit with no
-    /// start timeout of its own).
-    async fn run_command(
-        program: &str,
-        args: &[&str],
-        timeout: Duration,
-    ) -> Result<std::process::Output> {
-        tokio::time::timeout(
-            timeout,
-            tokio::process::Command::new(program).args(args).output(),
-        )
-        .await
-        .with_context(|| format!("{program} timed out"))?
-        .with_context(|| format!("failed to run {program}"))
-    }
-
     fn init_nvml(expected_devices: u32) -> Result<nvml_wrapper::Nvml> {
         let nvml = nvml_wrapper::Nvml::init().context("failed to initialize NVML")?;
         let devices = nvml
@@ -1726,52 +1706,6 @@ mod gpu {
         serde_json::to_vec(&event).context("failed to serialize GPU attestation event")
     }
 
-    fn normalize_proxy_url(proxy_url: Option<&str>) -> Result<Option<String>> {
-        let Some(proxy_url) = proxy_url.map(str::trim).filter(|url| !url.is_empty()) else {
-            return Ok(None);
-        };
-        let parsed = url::Url::parse(proxy_url).context("invalid NVIDIA attestation proxy URL")?;
-        if !matches!(parsed.scheme(), "http" | "https") || parsed.host_str().is_none() {
-            bail!("NVIDIA attestation proxy must be an absolute HTTP(S) URL");
-        }
-        if parsed.query().is_some()
-            || parsed.fragment().is_some()
-            || !parsed.username().is_empty()
-            || parsed.password().is_some()
-            || parsed.path() != "/"
-        {
-            bail!(
-                "NVIDIA attestation proxy URL must not contain credentials, path, query, or fragment"
-            );
-        }
-        Ok(Some(parsed.as_str().trim_end_matches('/').to_string()))
-    }
-
-    fn nvattest_args(nonce: &str, proxy_url: Option<&str>) -> Result<Vec<String>> {
-        let mut args = vec![
-            "attest".to_string(),
-            "--device".to_string(),
-            "gpu".to_string(),
-            "--verifier".to_string(),
-            "local".to_string(),
-            "--nonce".to_string(),
-            nonce.to_string(),
-            "--format".to_string(),
-            "json".to_string(),
-        ];
-        if let Some(proxy_url) = normalize_proxy_url(proxy_url)? {
-            args.extend([
-                "--ocsp-url".to_string(),
-                format!("{proxy_url}/ocsp"),
-                "--rim-url".to_string(),
-                proxy_url,
-                "--relying-party-policy".to_string(),
-                TRUST_OUTPOST_POLICY.to_string(),
-            ]);
-        }
-        Ok(args)
-    }
-
     /// Run local GPU attestation via nvattest with a fresh evidence nonce. If
     /// sys-config selects a collateral proxy, both RIM and OCSP traffic is
     /// routed through it and NVIDIA's Trust Outpost policy accepts cached OCSP
@@ -1781,34 +1715,20 @@ mod gpu {
         expected_devices: u32,
         proxy_url: Option<&str>,
     ) -> Result<GpuAttestationResult> {
-        if !Path::new(NVATTEST).exists() {
+        if !nvattest::available() {
             bail!("nvattest is not available in this image");
         }
         // Certificate/OCSP validation needs a sane clock even when
         // secure_time is off; best-effort step chrony before attesting.
         if let Err(err) = cmd!(chronyc makestep) {
             warn!("failed to step system clock: {err:?}");
         }
-        let nonce = hex::encode(rand::thread_rng().gen::<[u8; 32]>());
-        let args = nvattest_args(&nonce, proxy_url)?;
-        if args.iter().any(|arg| arg == "--relying-party-policy")
-            && !Path::new(TRUST_OUTPOST_POLICY).is_file()
-        {
-            bail!("NVIDIA attestation proxy is configured but {TRUST_OUTPOST_POLICY} is missing");
-        }
-        let args = args.iter().map(String::as_str).collect::<Vec<_>>();
-        let output = run_command(NVATTEST, &args, ATTESTATION_TIMEOUT).await?;
-        if !output.stderr.is_empty() {
-            info!("nvattest: {}", truncated_lossy(&output.stderr, 2048));
-        }
+        let nonce: [u8; nvattest::NONCE_LEN] = rand::thread_rng().gen();
+        let (nonce, output) = nvattest::run(&nonce, proxy_url, nvattest::DEFAULT_TIMEOUT).await?;
+        // Persist before judging the exit status: a failed appraisal is exactly
+        // when the evidence is worth having on disk.
         save_attestation_output(&output.stdout).context("failed to save GPU attestation output")?;
-        if !output.status.success() {
-            bail!(
-                "nvattest exited with {}: {}",
-                output.status,
-                truncated_lossy(&output.stderr, 512),
-            );
-        }
+        nvattest::check_status(&output)?;
         let claims = validate_attestation_output(&output.stdout, &nonce, expected_devices)?;
         Ok(GpuAttestationResult {
             claims: claims.raw,
@@ -1880,15 +1800,6 @@ mod gpu {
         Ok(())
     }
 
-    fn truncated_lossy(bytes: &[u8], limit: usize) -> String {
-        let text = String::from_utf8_lossy(bytes);
-        let text = text.trim();
-        match text.char_indices().nth(limit) {
-            Some((idx, _)) => format!("{}...", &text[..idx]),
-            None => text.to_string(),
-        }
-    }
-
     #[cfg(test)]
     mod tests {
         use super::*;
@@ -1926,6 +1837,48 @@ mod gpu {
         const H100_ATTESTATION_OUTPUT: &[u8] =
             include_bytes!("../tests/fixtures/gpu_attestation_h100.json");
 
+        /// The `AttestGpu` API documents that its output cannot be verified by a
+        /// third party, because the local verifier reports a conclusion rather
+        /// than the GPU's signed report. That is a claim about NVIDIA's output
+        /// format, so pin it: if a future SDK starts signing the detached EAT,
+        /// this fails and the API docs need revisiting rather than quietly
+        /// becoming wrong.
+        #[test]
+        fn local_verifier_output_is_unsigned_self_report() {
+            let output: Value = serde_json::from_slice(H100_ATTESTATION_OUTPUT).unwrap();
+            let eat = &output["detached_eat"];
+            let jwt = eat[0][1].as_str().expect("detached EAT carries a JWT");
+            let (header_b64, rest) = jwt.split_once('.').unwrap();
+            let (_, signature) = rest.split_once('.').unwrap();
+            assert!(
+                signature.is_empty(),
+                "detached EAT is signed; AttestGpu docs claim it is not"
+            );
+
+            use base64::Engine as _;
+            let header = base64::engine::general_purpose::URL_SAFE_NO_PAD
+                .decode(header_b64)
+                .unwrap();
+            let header: Value = serde_json::from_slice(&header).unwrap();
+            assert_eq!(header["alg"], "none");
+
+            // And the signed artifacts really are absent: the claims carry
+            // verdicts about the certificate chain, not the chain itself.
+            let claim = &output["claims"][0];
+            assert_eq!(
+                claim["x-nvidia-gpu-attestation-report-signature-verified"],
+                true
+            );
+            assert!(
+                claim["x-nvidia-gpu-attestation-report-cert-chain"]
+                    .as_object()
+                    .expect("cert-chain claim is a verdict object")
+                    .keys()
+                    .all(|key| key.starts_with("x-nvidia-cert-")),
+                "cert-chain claim carries certificates, not just verdicts"
+            );
+        }
+
         #[test]
         fn inventory_counts_nvidia_and_non_nvidia_gpus() {
             let root = tempfile::tempdir().unwrap();
@@ -1956,35 +1909,6 @@ mod gpu {
             assert_eq!(nvidia_gpu_count(nvidia).unwrap(), 2);
         }
 
-        #[test]
-        fn proxy_routes_ocsp_and_rim_and_selects_outpost_policy() {
-            let nonce = format!("test-nonce-{}", std::process::id());
-            let args = nvattest_args(&nonce, Some("http://10.0.2.2:8090/")).unwrap();
-            assert!(args
-                .windows(2)
-                .any(|args| args == ["--ocsp-url", "http://10.0.2.2:8090/ocsp"]));
-            assert!(args
-                .windows(2)
-                .any(|args| args == ["--rim-url", "http://10.0.2.2:8090"]));
-            assert!(args
-                .windows(2)
-                .any(|args| args == ["--relying-party-policy", TRUST_OUTPOST_POLICY]));
-
-            let direct = nvattest_args(&nonce, None).unwrap();
-            assert!(!direct.iter().any(|arg| arg == "--ocsp-url"));
-            assert!(!direct.iter().any(|arg| arg == "--relying-party-policy"));
-        }
-
-        #[test]
-        fn proxy_url_validation_is_fail_closed() {
-            let nonce = format!("test-nonce-{}", std::process::id());
-            assert!(nvattest_args(&nonce, Some("file:///tmp/proxy")).is_err());
-            assert!(nvattest_args(&nonce, Some("https://user@example.com")).is_err());
-            assert!(nvattest_args(&nonce, Some("https://example.com?q=1")).is_err());
-            assert!(nvattest_args(&nonce, Some("https://example.com/base")).is_err());
-            assert!(normalize_proxy_url(Some("  ")).unwrap().is_none());
-        }
-
         #[test]
         fn basic_policy_requires_cc_and_rejects_devtools_by_default() {
             let nonce = "44".repeat(32);

dstack/guest-agent/Cargo.toml

@@ -10,6 +10,7 @@ edition.workspace = true
 license.workspace = true
 
 [dependencies]
+nvattest.workspace = true
 rocket.workspace = true
 tracing.workspace = true
 tracing-subscriber.workspace = true