diff --git a/CHANGELOG.md b/CHANGELOG.md index 8ba6ad3fe..c1c774941 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - sdk: `verify_signature` and `verify_signature_chain` in all four SDKs, replacing the removed guest-agent `Verify` RPC. `verify_signature_chain` is new capability rather than a port: it walks all three links of a `Sign` signature chain -- payload signature, the app root key attesting `"{purpose}:{hex(pubkey)}"`, and the KMS root attesting that app root for this `app_id` -- and requires the chain to anchor at a KMS root public key **the caller supplies**. That anchor has to come from somewhere independently trusted (the `DstackKms` contract's `kmsInfo().k256Pubkey`, or a pinned value); read it from the KMS being checked and an attacker who can answer that query can also mint a self-consistent chain. The four ports are pinned against one committed set of test vectors, `sdk/tests/vectors/signature_chain.json`, generated from the real KMS and guest-agent primitives -- this repo has shipped cross-language crypto drift twice already +- guest-agent: `AttestGpu` collects vendor-native GPU evidence on demand against a caller-supplied 32-byte nonce. It returns opaque, versioned evidence bundles identified by vendor and format for independent appraisal. The response format is extensible to additional GPU vendors. Exposed in the Rust, Python, Go, and JS SDKs - guest-agent: `Attest` accepts `include_boottime_gpu_evidence` and returns the boot-time GPU attestation evidence in `AttestResponse.boottime_gpu_evidence`, so a verifier can fetch the quote and the GPU evidence in one round trip instead of also calling `GpuInfo`. Exposed in the Rust, Python, Go and JS SDKs - sdk: `AppCompose` in the Go SDK gained `init_script`, `storage_fs`, `swap_size`, `event_log_version`, `port_policy` and `verity_volumes`, and `Requirements` gained `gpu_policy` in the Go and Python SDKs - shared API authentication (`dstack-api-auth`) protecting the full VMM HTTP/pRPC/UI surface and unifying Gateway/KMS admin auth: bearer/`X-Admin-Token`/HTTP Basic/bcrypt htpasswd, constant-time verification (#796) diff --git a/docs/attestation-tdx.md b/docs/attestation-tdx.md index bab973198..4df561c2e 100644 --- a/docs/attestation-tdx.md +++ b/docs/attestation-tdx.md @@ -45,7 +45,7 @@ evaluation, `gpu-attestation`. The `gpu-policy-hash` payload is The `gpu-attestation` payload is JSON containing the verified device count, CC/DevTools state, and `evidence_sha256`. -The guest-agent `GpuInfo` API returns the complete `nvattest` JSON captured during boot; `Attest` returns the same bytes in `boottime_gpu_evidence` when called with `include_boottime_gpu_evidence`, so a verifier can fetch the quote and the GPU evidence in one round trip. It is not trustworthy by itself. After verifying the TDX quote and replaying the event log to RTMR3, hash the exact UTF-8 bytes of `GpuInfo.attestation` (or `Attest.boottime_gpu_evidence`) and require the result to equal the `gpu-attestation` event's `evidence_sha256`. See [GPU Security for AI Workloads](./security/security-model.md#gpu-security-for-ai-workloads) for the event schema, ordering, Rego example, and platform differences. +The guest-agent `GpuInfo` API returns the complete `nvattest` JSON captured during boot; `Attest` returns the same bytes in `boottime_gpu_evidence` when called with `include_boottime_gpu_evidence`, so a verifier can fetch the quote and the GPU evidence in one round trip. It is not trustworthy by itself. (`AttestGpu` runs a *fresh* attestation against a caller nonce, but its result is not bound to the TD and must not be used as remote evidence; only the boot-time record below is.) After verifying the TDX quote and replaying the event log to RTMR3, hash the exact UTF-8 bytes of `GpuInfo.attestation` (or `Attest.boottime_gpu_evidence`) and require the result to equal the `gpu-attestation` event's `evidence_sha256`. See [GPU Security for AI Workloads](./security/security-model.md#gpu-security-for-ai-workloads) for the event schema, ordering, Rego example, and platform differences. ### 2.2. Determining expected MRs MRTD, RTMR0, RTMR1, and RTMR2 correspond to the image. dstack OS builds all related software from source. diff --git a/docs/security/security-model.md b/docs/security/security-model.md index 5ba888705..f2795537e 100644 --- a/docs/security/security-model.md +++ b/docs/security/security-model.md @@ -158,7 +158,7 @@ The events make the following **boot-time** statement: immediately before key pr - A mixed launch cannot attest only its TEE-capable subset. Non-NVIDIA display GPUs are rejected, and the sysfs, NVML, and nvattest claim counts must all agree. A non-CC NVIDIA GPU either prevents evidence collection/appraisal or causes the default appraisal, application policy, or CC-state check to fail. - Copying another CVM's result into a file or `report_data` does not work. Only measured pre-application code can place the event before `system-ready`, and event-log replay binds it to the quoted RTMR/PCR value. -This is **not a lifetime or physical co-location guarantee**. After `system-ready`, an application with sufficient guest privileges can unload the NVIDIA driver, and a malicious host may attempt PCI hot-remove/replacement or proxy GPU traffic. The boot event remains a true historical statement but does not prove that the same device is still attached. dstack also cannot rule out a live relay/cuckoo attack to a genuine remote GPU: current Hopper/Blackwell deployments do not provide a CPU-TEE-verifiable TEE-I/O/TDISP device binding. Applications that mutate the driver or PCI topology are outside this guarantee; higher-assurance deployments must prevent that behavior and re-attest before using a newly initialized GPU. +This is **not a lifetime or physical co-location guarantee**. After `system-ready`, an application with sufficient guest privileges can unload the NVIDIA driver, and a malicious host may attempt PCI hot-remove/replacement or proxy GPU traffic. The boot event remains a true historical statement but does not prove that the same device is still attached. dstack also cannot rule out a live relay/cuckoo attack to a genuine remote GPU: current Hopper/Blackwell deployments do not provide a CPU-TEE-verifiable TEE-I/O/TDISP device binding. Applications that mutate the driver or PCI topology are outside this guarantee; higher-assurance deployments must prevent that behavior and re-attest before using a newly initialized GPU; the guest-agent `AttestGpu` API does that re-check against a caller-chosen nonce. Being an NVIDIA report, its result is subject to the same relay caveat as any attestation-time GPU sample: it establishes that a genuine CC-enabled GPU is reachable and responsive now, not that the device is bound to this TD, so it must not be forwarded to a remote relying party as proof of GPU possession. AMD SEV-SNP has no runtime measurement register in the current dstack stack. The local boot gate can still fail closed, but a `gpu-attestation` event carried beside an SNP report is not remotely bound to that report and must not be accepted as dual-attestation evidence. SNP needs a measured vTPM/PCR channel before it can provide the same remote binding. diff --git a/dstack/Cargo.lock b/dstack/Cargo.lock index aad37febb..28d41b459 100644 --- a/dstack/Cargo.lock +++ b/dstack/Cargo.lock @@ -1970,6 +1970,7 @@ dependencies = [ "libc", "listenfd", "load_config", + "nvattest", "or-panic", "ra-rpc", "ra-tls", @@ -2217,6 +2218,7 @@ version = "0.6.0" dependencies = [ "aes-gcm", "anyhow", + "base64 0.22.1", "binrw", "bollard", "cc-eventlog", @@ -2239,6 +2241,7 @@ dependencies = [ "k256", "libc", "luks2", + "nvattest", "nvml-wrapper", "parity-scale-codec", "ra-rpc", @@ -4810,6 +4813,18 @@ dependencies = [ "syn 2.0.117", ] +[[package]] +name = "nvattest" +version = "0.6.0" +dependencies = [ + "anyhow", + "serde", + "serde_json", + "tokio", + "tracing", + "url", +] + [[package]] name = "nvml-wrapper" version = "0.12.1" diff --git a/dstack/Cargo.toml b/dstack/Cargo.toml index 6fe00a79d..fc4b54a57 100644 --- a/dstack/Cargo.toml +++ b/dstack/Cargo.toml @@ -59,6 +59,7 @@ members = [ "serde-duration", "dstack-mr", "dstack-mr/cli", + "nvattest", "nvidia-attest-proxy", "verifier", "size-parser", @@ -121,6 +122,7 @@ serde-duration = { path = "serde-duration" } dstack-mr = { path = "dstack-mr" } dstack-verifier = { path = "verifier", default-features = false } size-parser = { path = "size-parser" } +nvattest = { path = "nvattest" } wavekv = "2.1" # Core dependencies diff --git a/dstack/dstack-util/Cargo.toml b/dstack/dstack-util/Cargo.toml index fe16dec66..8107377fe 100644 --- a/dstack/dstack-util/Cargo.toml +++ b/dstack/dstack-util/Cargo.toml @@ -10,6 +10,7 @@ edition.workspace = true license.workspace = true [dependencies] +nvattest.workspace = true aes-gcm.workspace = true anyhow.workspace = true clap.workspace = true @@ -68,4 +69,5 @@ safe-write.workspace = true errify.workspace = true [dev-dependencies] +base64.workspace = true rand.workspace = true diff --git a/dstack/dstack-util/src/system_setup.rs b/dstack/dstack-util/src/system_setup.rs index 7dc2947bf..9eb9b97ab 100644 --- a/dstack/dstack-util/src/system_setup.rs +++ b/dstack/dstack-util/src/system_setup.rs @@ -1434,11 +1434,8 @@ async fn do_sys_setup(stage0: Stage0<'_>) -> Result<()> { mod gpu { use super::*; - const NVATTEST: &str = "/usr/bin/nvattest"; - const ATTESTATION_TIMEOUT: Duration = Duration::from_secs(300); const EVENT_VERSION: u32 = 2; const POLICY_ENTRYPOINT: &str = "data.policy.nv_match"; - const TRUST_OUTPOST_POLICY: &str = "/usr/share/nvattest/policies/allow_trust_outpost_ocsp.rego"; /// Bound Rego evaluation so a runaway application policy cannot hang boot. const POLICY_TIMEOUT: Duration = Duration::from_secs(10); @@ -1578,23 +1575,6 @@ mod gpu { Ok(inventory.nvidia) } - /// Run a GPU tool with a bounded timeout so a wedged driver/GPU cannot - /// hang the boot indefinitely (dstack-prepare is a oneshot unit with no - /// start timeout of its own). - async fn run_command( - program: &str, - args: &[&str], - timeout: Duration, - ) -> Result { - tokio::time::timeout( - timeout, - tokio::process::Command::new(program).args(args).output(), - ) - .await - .with_context(|| format!("{program} timed out"))? - .with_context(|| format!("failed to run {program}")) - } - fn init_nvml(expected_devices: u32) -> Result { let nvml = nvml_wrapper::Nvml::init().context("failed to initialize NVML")?; let devices = nvml @@ -1726,52 +1706,6 @@ mod gpu { serde_json::to_vec(&event).context("failed to serialize GPU attestation event") } - fn normalize_proxy_url(proxy_url: Option<&str>) -> Result> { - let Some(proxy_url) = proxy_url.map(str::trim).filter(|url| !url.is_empty()) else { - return Ok(None); - }; - let parsed = url::Url::parse(proxy_url).context("invalid NVIDIA attestation proxy URL")?; - if !matches!(parsed.scheme(), "http" | "https") || parsed.host_str().is_none() { - bail!("NVIDIA attestation proxy must be an absolute HTTP(S) URL"); - } - if parsed.query().is_some() - || parsed.fragment().is_some() - || !parsed.username().is_empty() - || parsed.password().is_some() - || parsed.path() != "/" - { - bail!( - "NVIDIA attestation proxy URL must not contain credentials, path, query, or fragment" - ); - } - Ok(Some(parsed.as_str().trim_end_matches('/').to_string())) - } - - fn nvattest_args(nonce: &str, proxy_url: Option<&str>) -> Result> { - let mut args = vec![ - "attest".to_string(), - "--device".to_string(), - "gpu".to_string(), - "--verifier".to_string(), - "local".to_string(), - "--nonce".to_string(), - nonce.to_string(), - "--format".to_string(), - "json".to_string(), - ]; - if let Some(proxy_url) = normalize_proxy_url(proxy_url)? { - args.extend([ - "--ocsp-url".to_string(), - format!("{proxy_url}/ocsp"), - "--rim-url".to_string(), - proxy_url, - "--relying-party-policy".to_string(), - TRUST_OUTPOST_POLICY.to_string(), - ]); - } - Ok(args) - } - /// Run local GPU attestation via nvattest with a fresh evidence nonce. If /// sys-config selects a collateral proxy, both RIM and OCSP traffic is /// routed through it and NVIDIA's Trust Outpost policy accepts cached OCSP @@ -1781,7 +1715,7 @@ mod gpu { expected_devices: u32, proxy_url: Option<&str>, ) -> Result { - if !Path::new(NVATTEST).exists() { + if !nvattest::available() { bail!("nvattest is not available in this image"); } // Certificate/OCSP validation needs a sane clock even when @@ -1789,26 +1723,12 @@ mod gpu { if let Err(err) = cmd!(chronyc makestep) { warn!("failed to step system clock: {err:?}"); } - let nonce = hex::encode(rand::thread_rng().gen::<[u8; 32]>()); - let args = nvattest_args(&nonce, proxy_url)?; - if args.iter().any(|arg| arg == "--relying-party-policy") - && !Path::new(TRUST_OUTPOST_POLICY).is_file() - { - bail!("NVIDIA attestation proxy is configured but {TRUST_OUTPOST_POLICY} is missing"); - } - let args = args.iter().map(String::as_str).collect::>(); - let output = run_command(NVATTEST, &args, ATTESTATION_TIMEOUT).await?; - if !output.stderr.is_empty() { - info!("nvattest: {}", truncated_lossy(&output.stderr, 2048)); - } + let nonce: [u8; nvattest::NONCE_LEN] = rand::thread_rng().gen(); + let (nonce, output) = nvattest::run(&nonce, proxy_url, nvattest::DEFAULT_TIMEOUT).await?; + // Persist before judging the exit status: a failed appraisal is exactly + // when the evidence is worth having on disk. save_attestation_output(&output.stdout).context("failed to save GPU attestation output")?; - if !output.status.success() { - bail!( - "nvattest exited with {}: {}", - output.status, - truncated_lossy(&output.stderr, 512), - ); - } + nvattest::check_status(&output)?; let claims = validate_attestation_output(&output.stdout, &nonce, expected_devices)?; Ok(GpuAttestationResult { claims: claims.raw, @@ -1880,15 +1800,6 @@ mod gpu { Ok(()) } - fn truncated_lossy(bytes: &[u8], limit: usize) -> String { - let text = String::from_utf8_lossy(bytes); - let text = text.trim(); - match text.char_indices().nth(limit) { - Some((idx, _)) => format!("{}...", &text[..idx]), - None => text.to_string(), - } - } - #[cfg(test)] mod tests { use super::*; @@ -1926,6 +1837,48 @@ mod gpu { const H100_ATTESTATION_OUTPUT: &[u8] = include_bytes!("../tests/fixtures/gpu_attestation_h100.json"); + /// The `AttestGpu` API documents that its output cannot be verified by a + /// third party, because the local verifier reports a conclusion rather + /// than the GPU's signed report. That is a claim about NVIDIA's output + /// format, so pin it: if a future SDK starts signing the detached EAT, + /// this fails and the API docs need revisiting rather than quietly + /// becoming wrong. + #[test] + fn local_verifier_output_is_unsigned_self_report() { + let output: Value = serde_json::from_slice(H100_ATTESTATION_OUTPUT).unwrap(); + let eat = &output["detached_eat"]; + let jwt = eat[0][1].as_str().expect("detached EAT carries a JWT"); + let (header_b64, rest) = jwt.split_once('.').unwrap(); + let (_, signature) = rest.split_once('.').unwrap(); + assert!( + signature.is_empty(), + "detached EAT is signed; AttestGpu docs claim it is not" + ); + + use base64::Engine as _; + let header = base64::engine::general_purpose::URL_SAFE_NO_PAD + .decode(header_b64) + .unwrap(); + let header: Value = serde_json::from_slice(&header).unwrap(); + assert_eq!(header["alg"], "none"); + + // And the signed artifacts really are absent: the claims carry + // verdicts about the certificate chain, not the chain itself. + let claim = &output["claims"][0]; + assert_eq!( + claim["x-nvidia-gpu-attestation-report-signature-verified"], + true + ); + assert!( + claim["x-nvidia-gpu-attestation-report-cert-chain"] + .as_object() + .expect("cert-chain claim is a verdict object") + .keys() + .all(|key| key.starts_with("x-nvidia-cert-")), + "cert-chain claim carries certificates, not just verdicts" + ); + } + #[test] fn inventory_counts_nvidia_and_non_nvidia_gpus() { let root = tempfile::tempdir().unwrap(); @@ -1956,35 +1909,6 @@ mod gpu { assert_eq!(nvidia_gpu_count(nvidia).unwrap(), 2); } - #[test] - fn proxy_routes_ocsp_and_rim_and_selects_outpost_policy() { - let nonce = format!("test-nonce-{}", std::process::id()); - let args = nvattest_args(&nonce, Some("http://10.0.2.2:8090/")).unwrap(); - assert!(args - .windows(2) - .any(|args| args == ["--ocsp-url", "http://10.0.2.2:8090/ocsp"])); - assert!(args - .windows(2) - .any(|args| args == ["--rim-url", "http://10.0.2.2:8090"])); - assert!(args - .windows(2) - .any(|args| args == ["--relying-party-policy", TRUST_OUTPOST_POLICY])); - - let direct = nvattest_args(&nonce, None).unwrap(); - assert!(!direct.iter().any(|arg| arg == "--ocsp-url")); - assert!(!direct.iter().any(|arg| arg == "--relying-party-policy")); - } - - #[test] - fn proxy_url_validation_is_fail_closed() { - let nonce = format!("test-nonce-{}", std::process::id()); - assert!(nvattest_args(&nonce, Some("file:///tmp/proxy")).is_err()); - assert!(nvattest_args(&nonce, Some("https://user@example.com")).is_err()); - assert!(nvattest_args(&nonce, Some("https://example.com?q=1")).is_err()); - assert!(nvattest_args(&nonce, Some("https://example.com/base")).is_err()); - assert!(normalize_proxy_url(Some(" ")).unwrap().is_none()); - } - #[test] fn basic_policy_requires_cc_and_rejects_devtools_by_default() { let nonce = "44".repeat(32); diff --git a/dstack/guest-agent/Cargo.toml b/dstack/guest-agent/Cargo.toml index 9b23ae0d6..ffdfa269e 100644 --- a/dstack/guest-agent/Cargo.toml +++ b/dstack/guest-agent/Cargo.toml @@ -10,6 +10,7 @@ edition.workspace = true license.workspace = true [dependencies] +nvattest.workspace = true rocket.workspace = true tracing.workspace = true tracing-subscriber.workspace = true diff --git a/dstack/guest-agent/rpc/proto/agent_rpc.proto b/dstack/guest-agent/rpc/proto/agent_rpc.proto index 65c4442ae..7a99d32e2 100644 --- a/dstack/guest-agent/rpc/proto/agent_rpc.proto +++ b/dstack/guest-agent/rpc/proto/agent_rpc.proto @@ -61,6 +61,19 @@ service DstackGuest { // Get GPU information collected during boot. rpc GpuInfo(google.protobuf.Empty) returns (GpuInfoResponse) {} + // Collect GPU attestation evidence now, against a nonce the caller chooses. + // + // This answers "is the device I can talk to right now a genuine, CC-enabled + // NVIDIA GPU that signs my challenge", which `GpuInfo` cannot: that returns a + // record written at boot. Use it after anything that may have reinitialised + // the GPU -- a driver reload leaves a device that responds to NVML but can no + // longer attest -- and before submitting work you care about. + // + // Returns vendor-native evidence, not a local verdict, so a relying party can + // appraise it with its own verifier. Evidence still does not bind the GPU to + // this TD; see `AttestGpuResponse.bundles`. + rpc AttestGpu(AttestGpuArgs) returns (AttestGpuResponse) {} + // Sign a payload rpc Sign(SignRequest) returns (SignResponse) {} @@ -229,6 +242,33 @@ message AttestResponse { string boottime_gpu_evidence = 2; } +message AttestGpuArgs { + // Exactly 32 bytes of caller-chosen challenge, passed to the GPU verbatim. + // + // SPDM fixes the evidence nonce at 32 bytes, and dstack applies no transform + // so a caller can compare these bytes directly against the `eat_nonce` claim + // rather than reversing a hash. To bind a longer challenge, hash it yourself. + bytes nonce = 1; +} + +message AttestGpuResponse { + // Vendor-native evidence bundles. The caller must select a verifier using + // `vendor` and `format`, then verify the signature, certificate chain, + // measurements, and the nonce embedded in the evidence. + repeated GpuEvidenceBundle bundles = 1; +} + +message GpuEvidenceBundle { + // Stable GPU vendor identifier, for example `nvidia`, `amd`, or `intel`. + string vendor = 1; + + // Vendor-specific evidence format and version. + string format = 2; + + // Opaque vendor-native evidence bytes. Do not assume UTF-8 or JSON. + bytes evidence = 3; +} + message GpuInfoResponse { // Complete JSON output produced by nvattest. Empty when no boot-time GPU // attestation output is available. diff --git a/dstack/guest-agent/src/gpu_attest.rs b/dstack/guest-agent/src/gpu_attest.rs new file mode 100644 index 000000000..ced4dcd6f --- /dev/null +++ b/dstack/guest-agent/src/gpu_attest.rs @@ -0,0 +1,64 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! On-demand GPU attestation for the `AttestGpu` RPC. +//! +//! Running `nvattest` is expensive in a way most RPCs are not: it spawns a +//! process and talks to the GPU through the driver. The gate below serialises +//! collection so concurrent callers do not compete for the same devices. + +use std::time::Duration; + +use anyhow::{bail, Result}; +use tokio::sync::Mutex; + +/// Serialises GPU evidence collection. +/// +/// Results are deliberately not shared between waiters: each caller supplies +/// its own nonce, and handing back evidence that answers somebody else's +/// challenge is exactly the confusion this API exists to avoid. +pub struct GpuAttestor { + timeout: Duration, + run_lock: Mutex<()>, +} + +impl GpuAttestor { + pub fn new() -> Self { + Self { + timeout: nvattest::DEFAULT_TIMEOUT, + run_lock: Mutex::new(()), + } + } + + /// Attest against `nonce`, or explain why not. + pub async fn attest(&self, nonce: &[u8]) -> Result> { + if nonce.len() != nvattest::NONCE_LEN { + bail!( + "nonce must be exactly {} bytes, got {}", + nvattest::NONCE_LEN, + nonce.len() + ); + } + if !nvattest::available() { + bail!("GPU attestation is not available in this image"); + } + // Held across the whole run, so a second caller waits rather than + // starting a competing nvattest against the same devices. + let _guard = self.run_lock.lock().await; + let (_, evidence) = nvattest::collect_evidence(nonce, self.timeout).await?; + Ok(evidence) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[tokio::test] + async fn a_wrong_length_nonce_is_rejected_before_anything_expensive() { + let attestor = GpuAttestor::new(); + let err = attestor.attest(&[0u8; 16]).await.unwrap_err().to_string(); + assert!(err.contains("exactly 32 bytes"), "{err}"); + } +} diff --git a/dstack/guest-agent/src/lib.rs b/dstack/guest-agent/src/lib.rs index 76fca6871..5ecb7359e 100644 --- a/dstack/guest-agent/src/lib.rs +++ b/dstack/guest-agent/src/lib.rs @@ -8,6 +8,7 @@ pub const GIT_REV: &str = dstack_build_info::git_revision!(); pub mod backend; pub mod config; mod container_health; +mod gpu_attest; mod guest_api_service; mod health; mod http_routes; diff --git a/dstack/guest-agent/src/rpc_service.rs b/dstack/guest-agent/src/rpc_service.rs index b594cee74..b49f67bf8 100644 --- a/dstack/guest-agent/src/rpc_service.rs +++ b/dstack/guest-agent/src/rpc_service.rs @@ -15,10 +15,10 @@ use dstack_guest_agent_rpc::{ dstack_guest_server::{DstackGuestRpc, DstackGuestServer}, tappd_server::{TappdRpc, TappdServer}, worker_server::{WorkerRpc, WorkerServer}, - AppInfo, AttestAppKeyRequest, AttestArgs, AttestResponse, DeriveK256KeyResponse, DeriveKeyArgs, - GetKeyArgs, GetKeyResponse, GetQuoteResponse, GetTlsKeyArgs, GetTlsKeyResponse, - GpuInfoResponse, HealthResponse, RawQuoteArgs, SignRequest, SignResponse, TdxQuoteArgs, - TdxQuoteResponse, WorkerVersion, + AppInfo, AttestAppKeyRequest, AttestArgs, AttestGpuArgs, AttestGpuResponse, AttestResponse, + DeriveK256KeyResponse, DeriveKeyArgs, GetKeyArgs, GetKeyResponse, GetQuoteResponse, + GetTlsKeyArgs, GetTlsKeyResponse, GpuEvidenceBundle, GpuInfoResponse, HealthResponse, + RawQuoteArgs, SignRequest, SignResponse, TdxQuoteArgs, TdxQuoteResponse, WorkerVersion, }; use dstack_types::{AppKeys, SysConfig, GPU_ATTESTATION_OUTPUT}; use ed25519_dalek::ed25519::signature::hazmat::PrehashSigner; @@ -89,6 +89,8 @@ struct AppStateInner { platform: Arc, /// Present only when the app opted into health gating; see `health`. health: Option>, + /// Serialises on-demand GPU attestation. + gpu_attestor: crate::gpu_attest::GpuAttestor, } impl AppStateInner { @@ -172,6 +174,7 @@ impl AppState { serde_json::from_str(&fs::read_to_string(&config.sys_config_file)?) .context("Failed to parse VM config")?; let collateral_urls = sys_config.collateral_urls(); + let gpu_attestor = crate::gpu_attest::GpuAttestor::new(); let vm_config = sys_config.vm_config; // Same trust anchor decision as dstack-util: never host-supplied, and // development roots only when this guest published them itself. @@ -202,6 +205,7 @@ impl AppState { vm_config, platform, health, + gpu_attestor, }), }; me.maybe_request_demo_cert(); @@ -388,6 +392,23 @@ impl DstackGuestRpc for InternalRpcHandler { get_info(&self.state, false).await } + async fn attest_gpu(self, request: AttestGpuArgs) -> Result { + let evidence = self + .state + .inner + .gpu_attestor + .attest(&request.nonce) + .await + .context("GPU attestation failed")?; + Ok(AttestGpuResponse { + bundles: vec![GpuEvidenceBundle { + vendor: "nvidia".to_string(), + format: "nvidia-nvattest-collect-evidence-json-v1".to_string(), + evidence, + }], + }) + } + async fn gpu_info(self) -> Result { Ok(GpuInfoResponse { attestation: read_gpu_attestation(Path::new(GPU_ATTESTATION_OUTPUT)), @@ -1012,6 +1033,7 @@ pNs85uhOZE8z2jr8Pg== }, }), health: None, + gpu_attestor: crate::gpu_attest::GpuAttestor::new(), }; ( diff --git a/dstack/nvattest/Cargo.toml b/dstack/nvattest/Cargo.toml new file mode 100644 index 000000000..672209a92 --- /dev/null +++ b/dstack/nvattest/Cargo.toml @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: © 2025 Phala Network +# +# SPDX-License-Identifier: Apache-2.0 + +[package] +name = "nvattest" +version.workspace = true +authors.workspace = true +edition.workspace = true +license.workspace = true +homepage.workspace = true +repository.workspace = true +description = "Shared wrapper around the NVIDIA nvattest CLI used by the boot gate and the guest agent" + +[dependencies] +anyhow.workspace = true +serde = { workspace = true, features = ["derive"] } +serde_json = { workspace = true, features = ["std"] } +tokio = { workspace = true, features = ["process", "time"] } +tracing.workspace = true +url.workspace = true + +[dev-dependencies] +tokio = { workspace = true, features = ["macros", "rt"] } diff --git a/dstack/nvattest/src/lib.rs b/dstack/nvattest/src/lib.rs new file mode 100644 index 000000000..f933ecd66 --- /dev/null +++ b/dstack/nvattest/src/lib.rs @@ -0,0 +1,423 @@ +// SPDX-FileCopyrightText: © 2025 Phala Network +// +// SPDX-License-Identifier: Apache-2.0 + +//! Thin wrapper around the NVIDIA `nvattest` CLI. +//! +//! Both the boot path (`dstack-util setup`) and the guest agent's on-demand +//! `AttestGpu` RPC shell out to the same binary with the same argument shape, +//! so the invocation, the proxy-URL rules and the output checks live here +//! rather than being written twice with two sets of bugs. +//! +//! What this crate does *not* do is decide what the evidence means. Appraisal +//! against a policy is the caller's, because the boot gate and a runtime +//! liveness check want different answers from the same bytes. + +use std::{path::Path, process::Output, time::Duration}; + +use anyhow::{bail, Context, Result}; +use serde::Deserialize; +use serde_json::Value; +use tracing::info; + +/// The nvattest binary, installed into the rootfs by the nvattest recipe. +pub const NVATTEST: &str = "/usr/bin/nvattest"; + +/// NVIDIA's relying-party policy that tolerates a cached OCSP responder nonce. +/// Only used when a collateral proxy is configured. +pub const TRUST_OUTPOST_POLICY: &str = "/usr/share/nvattest/policies/allow_trust_outpost_ocsp.rego"; + +/// SPDM fixes the GPU evidence nonce at 32 bytes. The SDK rejects anything +/// else, so callers get a clear error here instead of a CLI parse failure. +pub const NONCE_LEN: usize = 32; + +/// Long enough for a cold collateral fetch on a slow link, short enough that a +/// wedged driver cannot hang the caller forever. +pub const DEFAULT_TIMEOUT: Duration = Duration::from_secs(300); + +/// Raw `nvattest attest --format json` output plus the nonce it answered. +#[derive(Debug)] +pub struct Attestation { + /// Exact stdout bytes. Callers that bind evidence to a measurement must + /// hash these, not a re-serialization of the parsed claims. + pub output: Vec, + /// Hex-encoded nonce passed to nvattest, echoed back as `eat_nonce`. + pub nonce: String, + /// Parsed claims, one per attested device. + pub claims: Vec, +} + +#[derive(Deserialize)] +struct NvattestOutput { + result_code: i64, + claims: Vec, +} + +#[derive(Deserialize)] +struct CollectEvidenceOutput { + result_code: i64, + #[serde(default)] + evidences: Vec, +} + +#[derive(Deserialize)] +struct NonceClaim { + #[serde(rename = "eat_nonce")] + eat_nonce: String, + #[serde(rename = "x-nvidia-gpu-attestation-report-nonce-match")] + nonce_match: bool, +} + +/// True when this image can attest a GPU at all. +pub fn available() -> bool { + Path::new(NVATTEST).exists() +} + +/// Validate a collateral proxy URL. Rejects anything carrying credentials or a +/// path, so a misconfigured value cannot smuggle a different endpoint past the +/// `{proxy}/ocsp` and `{proxy}/v1/rim/...` construction below. +pub fn normalize_proxy_url(proxy_url: Option<&str>) -> Result> { + let Some(proxy_url) = proxy_url.map(str::trim).filter(|url| !url.is_empty()) else { + return Ok(None); + }; + let parsed = url::Url::parse(proxy_url).context("invalid NVIDIA attestation proxy URL")?; + if !matches!(parsed.scheme(), "http" | "https") || parsed.host_str().is_none() { + bail!("NVIDIA attestation proxy must be an absolute HTTP(S) URL"); + } + if parsed.query().is_some() + || parsed.fragment().is_some() + || !parsed.username().is_empty() + || parsed.password().is_some() + || parsed.path() != "/" + { + bail!( + "NVIDIA attestation proxy URL must not contain credentials, path, query, or fragment" + ); + } + Ok(Some(parsed.as_str().trim_end_matches('/').to_string())) +} + +/// Build the CLI arguments for a local (self-verifying) GPU attestation. +pub fn args(nonce: &str, proxy_url: Option<&str>) -> Result> { + let mut args = vec![ + "attest".to_string(), + "--device".to_string(), + "gpu".to_string(), + "--verifier".to_string(), + "local".to_string(), + "--nonce".to_string(), + nonce.to_string(), + "--format".to_string(), + "json".to_string(), + ]; + if let Some(proxy_url) = normalize_proxy_url(proxy_url)? { + args.extend([ + "--ocsp-url".to_string(), + format!("{proxy_url}/ocsp"), + "--rim-url".to_string(), + proxy_url, + "--relying-party-policy".to_string(), + TRUST_OUTPOST_POLICY.to_string(), + ]); + } + Ok(args) +} + +/// Run a command with a bounded timeout, killing the child if it expires so a +/// wedged GPU tool cannot outlive the caller that gave up on it. +pub async fn run_command(program: &str, args: &[&str], timeout: Duration) -> Result { + let child = tokio::process::Command::new(program) + .args(args) + .kill_on_drop(true) + .stdout(std::process::Stdio::piped()) + .stderr(std::process::Stdio::piped()) + .spawn() + .with_context(|| format!("failed to run {program}"))?; + tokio::time::timeout(timeout, child.wait_with_output()) + .await + .with_context(|| format!("{program} timed out"))? + .with_context(|| format!("failed to run {program}")) +} + +/// Run `nvattest` against `nonce` and return its raw output. +/// +/// `nonce` must be exactly [`NONCE_LEN`] bytes; it is passed through verbatim +/// so a caller can compare its own challenge against `eat_nonce` without +/// reversing any transform. +/// +/// Verifies only that nvattest succeeded and that every claim answers this +/// nonce. Everything else -- device counts, CC state, application policy -- +/// is appraisal, and belongs to the caller. +pub async fn attest( + nonce: &[u8], + proxy_url: Option<&str>, + timeout: Duration, +) -> Result { + let (nonce, output) = run(nonce, proxy_url, timeout).await?; + check_status(&output)?; + let claims = check_nonce(&output.stdout, &nonce)?; + Ok(Attestation { + output: output.stdout, + nonce, + claims, + }) +} + +/// Run `nvattest` and hand back its raw output *whatever its exit status*, +/// along with the hex nonce it was given. +/// +/// Errors only when the tool could not be run at all. The boot gate persists +/// stdout before judging the status, so a failed appraisal still leaves +/// evidence on disk to debug; that is why the status check is separate. +pub async fn run( + nonce: &[u8], + proxy_url: Option<&str>, + timeout: Duration, +) -> Result<(String, Output)> { + if !available() { + bail!("nvattest is not available in this image"); + } + let nonce = check_nonce_len(nonce)?; + let args = args(&nonce, proxy_url)?; + if args.iter().any(|arg| arg == "--relying-party-policy") + && !Path::new(TRUST_OUTPOST_POLICY).is_file() + { + bail!("NVIDIA attestation proxy is configured but {TRUST_OUTPOST_POLICY} is missing"); + } + let borrowed = args.iter().map(String::as_str).collect::>(); + let output = run_command(NVATTEST, &borrowed, timeout).await?; + if !output.stderr.is_empty() { + info!("nvattest: {}", truncated_lossy(&output.stderr, 2048)); + } + Ok((nonce, output)) +} + +/// Collect GPU-signed evidence for `nonce` without appraising it. +/// +/// Returns the `evidences` array alone, which is the shape `attest +/// --gpu-evidence-source=file` expects, and the shape a third party needs: the +/// report and certificate chain, not a verdict about them. +pub async fn collect_evidence(nonce: &[u8], timeout: Duration) -> Result<(String, Vec)> { + let nonce = check_nonce_len(nonce)?; + let args = [ + "collect-evidence", + "--device", + "gpu", + "--nonce", + &nonce, + "--format", + "json", + ]; + let output = run_command(NVATTEST, &args, timeout).await?; + if !output.stderr.is_empty() { + info!( + "nvattest collect-evidence: {}", + truncated_lossy(&output.stderr, 2048) + ); + } + check_status(&output)?; + let parsed: CollectEvidenceOutput = serde_json::from_slice(&output.stdout) + .context("failed to parse nvattest collect-evidence output")?; + if parsed.result_code != 0 { + bail!( + "nvattest collect-evidence failed (result_code={})", + parsed.result_code + ); + } + if parsed.evidences.is_empty() { + bail!("nvattest collected no GPU evidence"); + } + let evidences = + serde_json::to_vec(&parsed.evidences).context("failed to re-encode GPU evidence")?; + Ok((nonce, evidences)) +} + +fn check_nonce_len(nonce: &[u8]) -> Result { + if nonce.len() != NONCE_LEN { + bail!( + "gpu attestation nonce must be {NONCE_LEN} bytes, got {}", + nonce.len() + ); + } + Ok(hex_encode(nonce)) +} + +/// Turn a non-zero nvattest exit into an error carrying a bounded stderr tail. +pub fn check_status(output: &Output) -> Result<()> { + if !output.status.success() { + bail!( + "nvattest exited with {}: {}", + output.status, + truncated_lossy(&output.stderr, 512), + ); + } + Ok(()) +} + +/// Require a successful result and one fresh claim per device answering `nonce`. +pub fn check_nonce(stdout: &[u8], nonce: &str) -> Result> { + let parsed: NvattestOutput = + serde_json::from_slice(stdout).context("failed to parse nvattest JSON output")?; + if parsed.result_code != 0 { + bail!( + "nvattest JSON result is not successful (result_code={})", + parsed.result_code + ); + } + if parsed.claims.is_empty() { + bail!("nvattest returned no GPU claims"); + } + for (index, claim) in parsed.claims.iter().enumerate() { + let nonce_claim: NonceClaim = serde_json::from_value(claim.clone()) + .with_context(|| format!("invalid GPU claim at index {index}"))?; + if nonce_claim.eat_nonce != nonce || !nonce_claim.nonce_match { + bail!("gpu claim at index {index} does not answer the requested nonce"); + } + } + Ok(parsed.claims) +} + +fn hex_encode(bytes: &[u8]) -> String { + use std::fmt::Write; + bytes + .iter() + .fold(String::with_capacity(bytes.len() * 2), |mut out, b| { + let _ = write!(out, "{b:02x}"); + out + }) +} + +/// Truncate on a character boundary so a huge or binary stderr cannot flood logs. +pub fn truncated_lossy(bytes: &[u8], limit: usize) -> String { + let text = String::from_utf8_lossy(bytes); + match text.char_indices().nth(limit) { + Some((end, _)) => format!("{}...", &text[..end]), + None => text.into_owned(), + } +} + +#[cfg(test)] +mod tests { + use super::*; + + fn claim(nonce: &str, matched: bool) -> String { + format!( + r#"{{"eat_nonce":"{nonce}","x-nvidia-gpu-attestation-report-nonce-match":{matched}}}"# + ) + } + + fn output(result_code: i64, claims: &[String]) -> Vec { + format!( + r#"{{"result_code":{result_code},"claims":[{}]}}"#, + claims.join(",") + ) + .into_bytes() + } + + #[test] + fn accepts_claims_answering_the_requested_nonce() { + let nonce = "aa".repeat(NONCE_LEN); + let out = output(0, &[claim(&nonce, true), claim(&nonce, true)]); + assert_eq!(check_nonce(&out, &nonce).unwrap().len(), 2); + } + + #[test] + fn rejects_a_claim_answering_a_different_nonce() { + let nonce = "aa".repeat(NONCE_LEN); + let stale = "bb".repeat(NONCE_LEN); + let out = output(0, &[claim(&nonce, true), claim(&stale, true)]); + let err = check_nonce(&out, &nonce).unwrap_err().to_string(); + assert!(err.contains("does not answer the requested nonce"), "{err}"); + } + + #[test] + fn rejects_a_claim_whose_device_reported_no_nonce_match() { + let nonce = "aa".repeat(NONCE_LEN); + let out = output(0, &[claim(&nonce, false)]); + assert!(check_nonce(&out, &nonce).is_err()); + } + + #[test] + fn rejects_unsuccessful_and_empty_results() { + let nonce = "aa".repeat(NONCE_LEN); + assert!(check_nonce(&output(1, &[claim(&nonce, true)]), &nonce).is_err()); + assert!(check_nonce(&output(0, &[]), &nonce).is_err()); + } + + #[tokio::test] + async fn collect_evidence_rejects_a_nonce_of_the_wrong_length() { + let err = collect_evidence(&[0u8; 16], DEFAULT_TIMEOUT) + .await + .unwrap_err() + .to_string(); + assert!( + err.contains("32 bytes") || err.contains("not available"), + "{err}" + ); + } + + #[tokio::test] + async fn rejects_a_nonce_of_the_wrong_length() { + let err = attest(&[0u8; 16], None, DEFAULT_TIMEOUT) + .await + .unwrap_err() + .to_string(); + // Length is checked before the binary is, so this holds off-target too. + assert!( + err.contains("32 bytes") || err.contains("not available"), + "{err}" + ); + } + + #[test] + fn proxy_url_rules() { + assert_eq!(normalize_proxy_url(None).unwrap(), None); + assert_eq!(normalize_proxy_url(Some(" ")).unwrap(), None); + assert_eq!( + normalize_proxy_url(Some("http://10.0.2.2:8090/")).unwrap(), + Some("http://10.0.2.2:8090".to_string()) + ); + for bad in [ + "ftp://host/", + "file:///tmp/proxy", + "https://user@example.com", + "http://user:pw@host/", + "https://example.com/base", + "https://example.com?q=1", + "not-a-url", + ] { + assert!( + normalize_proxy_url(Some(bad)).is_err(), + "{bad} must be rejected" + ); + } + } + + #[test] + fn proxy_routes_ocsp_and_rim_and_selects_outpost_policy() { + let proxied = args("aa", Some("http://10.0.2.2:8090/")).unwrap(); + assert!(proxied + .windows(2) + .any(|w| w == ["--ocsp-url", "http://10.0.2.2:8090/ocsp"])); + assert!(proxied + .windows(2) + .any(|w| w == ["--rim-url", "http://10.0.2.2:8090"])); + assert!(proxied + .windows(2) + .any(|w| w == ["--relying-party-policy", TRUST_OUTPOST_POLICY])); + + let direct = args("aa", None).unwrap(); + assert!(!direct.iter().any(|a| a == "--ocsp-url")); + assert!(!direct.iter().any(|a| a == "--relying-party-policy")); + } + + #[test] + fn truncation_is_bounded_and_utf8_safe() { + assert_eq!(truncated_lossy(b"abc", 10), "abc"); + assert_eq!(truncated_lossy(b"abcdef", 3), "abc..."); + assert_eq!( + truncated_lossy("\u{4f60}\u{597d}".as_bytes(), 1), + "\u{4f60}..." + ); + } +} diff --git a/sdk/curl/api.md b/sdk/curl/api.md index 84bb3d345..5b07cdc5f 100644 --- a/sdk/curl/api.md +++ b/sdk/curl/api.md @@ -238,7 +238,6 @@ You can submit the returned `attestation` directly to the verifier `/verify` end | Field | Type | Description | Example | |-------|------|-------------|----------| | `report_data` | string | Report data of max length 64 bytes. Padding with 0s if less than 64 bytes. | `"1234deadbeaf"` | -| `include_boottime_gpu_evidence` | boolean | Optional, defaults to `false`. Also returns the boot-time GPU attestation evidence in `boottime_gpu_evidence`. | `true` | **Example:** ```bash @@ -246,30 +245,57 @@ curl --unix-socket /var/run/dstack.sock -X POST \ http://dstack/Attest \ -H 'Content-Type: application/json' \ -d '{ - "report_data": "1234deadbeaf", - "include_boottime_gpu_evidence": true + "report_data": "1234deadbeaf" }' ``` Or ```bash -curl --unix-socket /var/run/dstack.sock 'http://dstack/Attest?report_data=1234deadbeaf&include_boottime_gpu_evidence=true' +curl --unix-socket /var/run/dstack.sock http://dstack/Attest?report_data=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 ``` **Response:** ```json { - "attestation": "", - "boottime_gpu_evidence": "{\"result_code\": 0, \"claims\": [...]}" + "attestation": "" } ``` -`boottime_gpu_evidence` carries the same bytes [`GpuInfo`](#7-gpu-info) serves, so one call +`boottime_gpu_evidence` carries the same bytes [`GpuInfo`](#8-gpu-info) serves, so one call returns both the quote and the GPU evidence a verifier needs. It is empty unless `include_boottime_gpu_evidence` was set and boot-time GPU attestation output exists. It is **not** bound to `report_data` — authenticate it with the `evidence_sha256` procedure documented under `GpuInfo` below. -### 7. GPU Info +### 7. Attest GPU + +Collects vendor-native GPU evidence for a caller-chosen 32-byte nonce. + +**Endpoint:** `/AttestGpu` + +**Request Parameters:** + +| Field | Type | Description | Example | +|-------|------|-------------|----------| +| `nonce` | string | Exactly 32 bytes, hex-encoded and passed to the GPU verbatim. | `"ab...ab"` (64 hex chars) | + +**Response:** + +```json +{ + "bundles": [{ + "vendor": "nvidia", + "format": "nvidia-nvattest-collect-evidence-json-v1", + "evidence": "" + }] +} +``` + +Select a verifier using each bundle's `vendor` and `format`. The verifier must check +the evidence signature, certificate chain, measurements, and embedded nonce. The +agent does not appraise the evidence. Evidence does not by itself bind the GPU to this +CVM. + +### 8. GPU Info Returns GPU information collected during boot. Currently, this includes the complete JSON output produced by NVIDIA `nvattest`. diff --git a/sdk/go/README.md b/sdk/go/README.md index 5b7c524de..58f2a7e14 100644 --- a/sdk/go/README.md +++ b/sdk/go/README.md @@ -592,6 +592,25 @@ The evidence is the same bytes ``GpuInfo`` serves and is empty unless the flag w and boot-time GPU attestation output exists. It is not bound to `report_data`; verify it with the measured `gpu-attestation` event digest as described under ``GpuInfo``. +##### `AttestGpu(ctx context.Context, nonce []byte) (*AttestGpuResponse, error)` + +Collects vendor-native GPU evidence for a caller-chosen 32-byte nonce. + +```go +resp, err := client.AttestGpu(ctx, nonce) +if err != nil { + log.Fatal(err) +} +for _, bundle := range resp.Bundles { + fmt.Println(bundle.Vendor, bundle.Format, bundle.Evidence) +} +``` + +Select a verifier using each bundle's `Vendor` and `Format`. The verifier must check +the evidence signature, certificate chain, measurements, and embedded nonce. Evidence +is opaque and hex-encoded by the JSON RPC. It does not by itself bind the GPU to this +CVM. + ##### `GpuInfo(ctx context.Context) (*GpuInfoResponse, error)` Returns GPU information collected during boot. Currently, this includes the diff --git a/sdk/go/dstack/client.go b/sdk/go/dstack/client.go index b963deb1c..1458ff4ec 100644 --- a/sdk/go/dstack/client.go +++ b/sdk/go/dstack/client.go @@ -132,6 +132,18 @@ type GpuInfoResponse struct { Attestation string `json:"attestation"` } +// AttestGpuResponse is the result of fresh, on-demand GPU evidence collection. +type AttestGpuResponse struct { + Bundles []GpuEvidenceBundle `json:"bundles"` +} + +type GpuEvidenceBundle struct { + Vendor string `json:"vendor"` + Format string `json:"format"` + // Evidence contains hex-encoded opaque bytes, as represented by the JSON RPC. + Evidence string `json:"evidence"` +} + // Represents an event log entry in the TCB info type EventLog struct { IMR int `json:"imr"` @@ -553,6 +565,27 @@ func (c *DstackClient) AttestWithOptions(ctx context.Context, reportData []byte, return &AttestResponse{Attestation: attestation, BoottimeGpuEvidence: response.BoottimeGpuEvidence}, nil } +// AttestGpu runs NVIDIA GPU attestation now, against a 32-byte nonce you choose. +// +// See AttestGpuResponse for what this does and does not prove. +func (c *DstackClient) AttestGpu(ctx context.Context, nonce []byte) (*AttestGpuResponse, error) { + if len(nonce) != 32 { + return nil, fmt.Errorf("nonce must be exactly 32 bytes, got %d", len(nonce)) + } + + payload := map[string]interface{}{"nonce": hex.EncodeToString(nonce)} + data, err := c.sendRPCRequest(ctx, "/AttestGpu", payload) + if err != nil { + return nil, err + } + + var response AttestGpuResponse + if err := json.Unmarshal(data, &response); err != nil { + return nil, err + } + return &response, nil +} + // GpuInfo returns GPU information collected during boot. func (c *DstackClient) GpuInfo(ctx context.Context) (*GpuInfoResponse, error) { data, err := c.sendRPCRequest(ctx, "/GpuInfo", map[string]interface{}{}) diff --git a/sdk/go/dstack/client_test.go b/sdk/go/dstack/client_test.go index 8e82bbef7..5a941e98f 100644 --- a/sdk/go/dstack/client_test.go +++ b/sdk/go/dstack/client_test.go @@ -9,6 +9,7 @@ import ( "context" "crypto/sha256" "crypto/x509" + "encoding/hex" "encoding/json" "encoding/pem" "fmt" @@ -93,7 +94,7 @@ func TestAttestWithBoottimeGpuEvidence(t *testing.T) { } w.Header().Set("Content-Type", "application/json") _ = json.NewEncoder(w).Encode(map[string]string{ - "attestation": "deadbeef", + "attestation": "deadbeef", "boottime_gpu_evidence": evidence, }) })) @@ -109,6 +110,48 @@ func TestAttestWithBoottimeGpuEvidence(t *testing.T) { } } +func TestAttestGpu(t *testing.T) { + const evidence = `[{"arch":"HOPPER","nonce":"ab","evidence":"BASE64","certificate":"BASE64"}]` + nonce := bytes.Repeat([]byte{0xab}, 32) + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/AttestGpu" { + t.Fatalf("unexpected path: %s", r.URL.Path) + } + var payload map[string]interface{} + if err := json.NewDecoder(r.Body).Decode(&payload); err != nil { + t.Fatalf("failed to decode request: %v", err) + } + if payload["nonce"] != hex.EncodeToString(nonce) { + t.Fatalf("nonce was not forwarded verbatim, got: %v", payload["nonce"]) + } + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]interface{}{ + "bundles": []map[string]string{{ + "vendor": "nvidia", "format": "nvidia-test-v1", "evidence": evidence, + }}, + }) + })) + defer server.Close() + + client := dstack.NewDstackClient(dstack.WithEndpoint(server.URL)) + resp, err := client.AttestGpu(context.Background(), nonce) + if err != nil { + t.Fatal(err) + } + if len(resp.Bundles) != 1 || resp.Bundles[0].Vendor != "nvidia" || resp.Bundles[0].Evidence != evidence { + t.Fatalf("unexpected evidence bundles: %+v", resp.Bundles) + } +} + +func TestAttestGpuRejectsWrongNonceLength(t *testing.T) { + client := dstack.NewDstackClient() + for _, n := range [][]byte{nil, bytes.Repeat([]byte{1}, 31), bytes.Repeat([]byte{1}, 33)} { + if _, err := client.AttestGpu(context.Background(), n); err == nil { + t.Fatalf("expected a %d-byte nonce to be rejected", len(n)) + } + } +} + func TestGpuInfo(t *testing.T) { const attestation = `{"result_code":0,"claims":[]}` server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { diff --git a/sdk/js/README.md b/sdk/js/README.md index f289e6052..04a41cee7 100644 --- a/sdk/js/README.md +++ b/sdk/js/README.md @@ -87,7 +87,7 @@ quote.quote // hex-encoded TDX quote quote.event_log // JSON string of measured events ``` -### `attest(reportData, includeBoottimeGpuEvidence?)` +### `attest(reportData)` Versioned dstack attestation that works across TDX / GCP / Nitro providers. Preferred for cross-platform verifiers. @@ -106,6 +106,22 @@ The evidence is the same bytes ``gpuInfo()`` serves and is empty unless the flag and boot-time GPU attestation output exists. It is not bound to `report_data`; verify it with the measured `gpu-attestation` event digest as described under ``gpuInfo()``. +### `attestGpu(nonce)` + +Collects vendor-native GPU evidence for a caller-chosen 32-byte nonce. + +```typescript +const { bundles } = await client.attestGpu(crypto.randomBytes(32)) +for (const bundle of bundles) { + console.log(bundle.vendor, bundle.format, bundle.evidence) +} +``` + +Select a verifier using each bundle's `vendor` and `format`. The verifier must check +the evidence signature, certificate chain, measurements, and embedded nonce. Evidence +is opaque and hex-encoded by the JSON RPC. It does not by itself bind the GPU to this +CVM. + ### `gpuInfo()` Returns GPU information collected during boot. Currently, this includes the diff --git a/sdk/js/src/__tests__/index.test.ts b/sdk/js/src/__tests__/index.test.ts index 6c0b5e8ce..5ff413f8f 100644 --- a/sdk/js/src/__tests__/index.test.ts +++ b/sdk/js/src/__tests__/index.test.ts @@ -48,6 +48,19 @@ describe('DstackClient', () => { expect(() => JSON.parse(result.event_log)).not.toThrowError() }) + it('should reject an attestGpu nonce that is not 32 bytes', async () => { + const client = new DstackClient() + await expect(() => client.attestGpu(new Uint8Array(31))).rejects.toThrow() + await expect(() => client.attestGpu(new Uint8Array(33))).rejects.toThrow() + }) + + it('should surface an attestGpu failure when the simulator has no GPU', async () => { + const client = new DstackClient() + // The simulator ships no nvattest, so this must fail fast and clearly + // rather than hang for the attestation timeout. + await expect(() => client.attestGpu(new Uint8Array(32).fill(0xab))).rejects.toThrow() + }) + it('should be able to attest', async () => { const client = new DstackClient() const result = await client.attest('test') diff --git a/sdk/js/src/index.ts b/sdk/js/src/index.ts index 7508c8f7a..f32167856 100644 --- a/sdk/js/src/index.ts +++ b/sdk/js/src/index.ts @@ -114,6 +114,21 @@ export interface AttestResponse { boottime_gpu_evidence: string } +/** + * Result of fresh, on-demand GPU evidence collection. + */ +export interface AttestGpuResponse { + __name__: Readonly<'AttestGpuResponse'> + bundles: GpuEvidenceBundle[] +} + +export interface GpuEvidenceBundle { + vendor: string + format: string + /** Hex-encoded opaque evidence bytes, as represented by the JSON RPC. */ + evidence: Hex +} + export interface GpuInfoResponse { __name__: Readonly<'GpuInfoResponse'> @@ -320,6 +335,26 @@ export class DstackClient { }) } + /** + * Runs NVIDIA GPU attestation now, against a 32-byte nonce you choose. + * + * See {@link AttestGpuResponse} for what this does and does not prove. + */ + async attestGpu(nonce: Buffer | Uint8Array): Promise { + if (nonce.length !== 32) { + throw new Error(`Nonce must be exactly 32 bytes, got ${nonce.length}.`) + } + const payload = JSON.stringify({ nonce: to_hex(nonce) }) + const result = await send_rpc_request<{ bundles: GpuEvidenceBundle[] }>(this.endpoint, '/AttestGpu', payload) + if ('error' in (result as any)) { + throw new Error((result as any)['error'] as string) + } + return Object.freeze({ + ...result, + __name__: 'AttestGpuResponse' as const, + }) + } + async gpuInfo(): Promise { const result = await send_rpc_request<{ attestation: string }>(this.endpoint, '/GpuInfo', '{}') return Object.freeze({ diff --git a/sdk/python/README.md b/sdk/python/README.md index 907145e3f..52ad32e7a 100644 --- a/sdk/python/README.md +++ b/sdk/python/README.md @@ -112,6 +112,22 @@ The evidence is the same bytes ``gpu_info()`` serves and is empty unless the fla and boot-time GPU attestation output exists. It is not bound to `report_data`; verify it with the measured `gpu-attestation` event digest as described under ``gpu_info()``. +### On-demand GPU Attestation + +`attest_gpu(nonce)` collects vendor-native GPU evidence for a caller-chosen 32-byte +nonce. + +```python +result = client.attest_gpu(os.urandom(32)) +for bundle in result.bundles: + print(bundle.vendor, bundle.format, bundle.evidence) +``` + +Select a verifier using each bundle's `vendor` and `format`. The verifier must check +the evidence signature, certificate chain, measurements, and embedded nonce. Evidence +is opaque and hex-encoded by the JSON RPC. It does not by itself bind the GPU to this +CVM. + ### GPU Info `gpu_info()` returns GPU information collected during boot. Currently, this diff --git a/sdk/python/src/dstack_sdk/__init__.py b/sdk/python/src/dstack_sdk/__init__.py index 3f9bc4cdb..5052772b2 100644 --- a/sdk/python/src/dstack_sdk/__init__.py +++ b/sdk/python/src/dstack_sdk/__init__.py @@ -4,12 +4,14 @@ from .dstack_client import AsyncDstackClient from .dstack_client import AsyncTappdClient +from .dstack_client import AttestGpuResponse from .dstack_client import AttestResponse from .dstack_client import DstackClient from .dstack_client import EventLog from .dstack_client import GetKeyResponse from .dstack_client import GetQuoteResponse from .dstack_client import GetTlsKeyResponse +from .dstack_client import GpuEvidenceBundle from .dstack_client import GpuInfoResponse from .dstack_client import InfoResponse from .dstack_client import SignResponse @@ -38,6 +40,8 @@ "GetKeyResponse", "GetTlsKeyResponse", "AttestResponse", + "AttestGpuResponse", + "GpuEvidenceBundle", "GpuInfoResponse", "GetQuoteResponse", "InfoResponse", diff --git a/sdk/python/src/dstack_sdk/dstack_client.py b/sdk/python/src/dstack_sdk/dstack_client.py index c27571a8c..aa5919d1b 100644 --- a/sdk/python/src/dstack_sdk/dstack_client.py +++ b/sdk/python/src/dstack_sdk/dstack_client.py @@ -163,6 +163,18 @@ def decode_attestation(self) -> bytes: return bytes.fromhex(self.attestation) +class GpuEvidenceBundle(BaseModel): + vendor: str + format: str + evidence: str + + +class AttestGpuResponse(BaseModel): + """Result of fresh, on-demand GPU evidence collection.""" + + bundles: list[GpuEvidenceBundle] + + class GpuInfoResponse(BaseModel): attestation: str @@ -457,6 +469,19 @@ async def attest( ) return AttestResponse(**result) + async def attest_gpu(self, nonce: bytes) -> AttestGpuResponse: + """Collect vendor-native GPU evidence for a 32-byte nonce. + + Select a verifier using each bundle's vendor and format, and verify the + signature, certificate chain, measurements, and embedded nonce. + """ + if not isinstance(nonce, (bytes, bytearray)) or len(nonce) != 32: + raise ValueError("nonce must be exactly 32 bytes") + result = await self._send_rpc_request( + "AttestGpu", {"nonce": binascii.hexlify(bytes(nonce)).decode()} + ) + return AttestGpuResponse(**result) + async def gpu_info(self) -> GpuInfoResponse: """Return GPU information collected during boot.""" result = await self._send_rpc_request("GpuInfo", {}) @@ -595,6 +620,15 @@ def attest( """ raise NotImplementedError + @call_async + def attest_gpu(self, nonce: bytes) -> AttestGpuResponse: + """Collect vendor-native GPU evidence for a 32-byte nonce. + + Select a verifier using each bundle's vendor and format, and verify the + signature, certificate chain, measurements, and embedded nonce. + """ + raise NotImplementedError + @call_async def gpu_info(self) -> GpuInfoResponse: """Return GPU information collected during boot.""" diff --git a/sdk/python/tests/test_client.py b/sdk/python/tests/test_client.py index 75f5efcfa..5fc7805e6 100644 --- a/sdk/python/tests/test_client.py +++ b/sdk/python/tests/test_client.py @@ -11,6 +11,7 @@ from dstack_sdk import AsyncDstackClient from dstack_sdk import AsyncTappdClient +from dstack_sdk import AttestGpuResponse from dstack_sdk import AttestResponse from dstack_sdk import DstackClient from dstack_sdk import GetKeyResponse @@ -139,6 +140,37 @@ async def fake_send(self, method, payload): assert result.boottime_gpu_evidence == evidence +@pytest.mark.asyncio +async def test_async_client_attest_gpu(monkeypatch): + evidence = '[{"arch":"HOPPER","evidence":"BASE64","certificate":"BASE64"}]' + nonce = bytes([0xAB]) * 32 + + async def fake_send(self, method, payload): + assert method == "AttestGpu" + assert payload == {"nonce": nonce.hex()} + return { + "bundles": [ + {"vendor": "nvidia", "format": "nvidia-test-v1", "evidence": evidence} + ] + } + + monkeypatch.setenv("DSTACK_SIMULATOR_ENDPOINT", "http://localhost:0") + monkeypatch.setattr(AsyncDstackClient, "_send_rpc_request", fake_send) + result = await AsyncDstackClient().attest_gpu(nonce) + assert isinstance(result, AttestGpuResponse) + assert len(result.bundles) == 1 + assert result.bundles[0].vendor == "nvidia" + assert result.bundles[0].evidence == evidence + + +@pytest.mark.asyncio +async def test_async_client_attest_gpu_rejects_wrong_nonce_length(): + client = AsyncDstackClient() + for bad in [b"", bytes(31), bytes(33), "not-bytes"]: + with pytest.raises(ValueError): + await client.attest_gpu(bad) + + @pytest.mark.asyncio async def test_async_client_gpu_info(monkeypatch): attestation = '{"result_code":0,"claims":[]}' diff --git a/sdk/rust/README.md b/sdk/rust/README.md index 4cae1a949..59c61a768 100644 --- a/sdk/rust/README.md +++ b/sdk/rust/README.md @@ -122,6 +122,22 @@ The evidence is the same bytes ``gpu_info()`` serves and is empty unless the fla and boot-time GPU attestation output exists. It is not bound to `report_data`; verify it with the measured `gpu-attestation` event digest as described under ``gpu_info()``. +#### `attest_gpu(nonce: Vec) -> AttestGpuResponse` + +Collects vendor-native GPU evidence for a caller-chosen 32-byte nonce. + +```rust +let result = client.attest_gpu(nonce.to_vec()).await?; +for bundle in result.bundles { + println!("{} {} {}", bundle.vendor, bundle.format, bundle.evidence); +} +``` + +Select a verifier using each bundle's `vendor` and `format`. The verifier must check +the evidence signature, certificate chain, measurements, and embedded nonce. Evidence +is opaque and hex-encoded by the JSON RPC. It does not by itself bind the GPU to this +CVM. + #### `gpu_info() -> GpuInfoResponse` Returns GPU information collected during boot. Currently, this includes the diff --git a/sdk/rust/src/dstack_client.rs b/sdk/rust/src/dstack_client.rs index eb8e60200..e95ed5516 100644 --- a/sdk/rust/src/dstack_client.rs +++ b/sdk/rust/src/dstack_client.rs @@ -176,6 +176,19 @@ impl DstackClient { Ok(response) } + /// Collects vendor-native GPU evidence for a caller-chosen 32-byte nonce. + /// + /// Select a verifier using each bundle's vendor and format. The verifier must + /// check the signature, certificate chain, measurements, and embedded nonce. + pub async fn attest_gpu(&self, nonce: Vec) -> Result { + if nonce.len() != 32 { + anyhow::bail!("Nonce must be exactly 32 bytes") + } + let data = json!({ "nonce": hex_encode(nonce) }); + let response = self.send_rpc_request("/AttestGpu", &data).await?; + Ok(serde_json::from_value::(response)?) + } + /// Returns GPU information collected during boot. pub async fn gpu_info(&self) -> Result { let response = self.send_rpc_request("/GpuInfo", &json!({})).await?; diff --git a/sdk/rust/tests/test_client.rs b/sdk/rust/tests/test_client.rs index 73043099b..144fc64df 100644 --- a/sdk/rust/tests/test_client.rs +++ b/sdk/rust/tests/test_client.rs @@ -25,6 +25,20 @@ async fn test_async_client_get_quote() { assert!(!result.quote.is_empty()); } +#[tokio::test] +async fn test_async_client_attest_gpu_validates_nonce_length() { + let client = AsyncDstackClient::new(None); + for len in [0, 31, 33] { + assert!( + client.attest_gpu(vec![0u8; len]).await.is_err(), + "a {len}-byte nonce must be rejected" + ); + } + // The simulator ships no nvattest, so a well-formed request must still fail + // fast with an error rather than hang for the attestation timeout. + assert!(client.attest_gpu(vec![0xab; 32]).await.is_err()); +} + #[tokio::test] async fn test_async_client_attest() { let client = AsyncDstackClient::new(None); diff --git a/sdk/rust/types/src/dstack.rs b/sdk/rust/types/src/dstack.rs index 93789c24e..e0f76c5e3 100644 --- a/sdk/rust/types/src/dstack.rs +++ b/sdk/rust/types/src/dstack.rs @@ -120,6 +120,27 @@ pub struct AttestConfig { pub include_boottime_gpu_evidence: bool, } +/// Response from fresh, on-demand GPU evidence collection. +/// +#[derive(Debug, Serialize, Deserialize)] +#[cfg_attr(feature = "borsh", derive(BorshSerialize, BorshDeserialize))] +#[cfg_attr(feature = "borsh_schema", derive(BorshSchema))] +pub struct AttestGpuResponse { + pub bundles: Vec, +} + +#[derive(Debug, Serialize, Deserialize)] +#[cfg_attr(feature = "borsh", derive(BorshSerialize, BorshDeserialize))] +#[cfg_attr(feature = "borsh_schema", derive(BorshSchema))] +pub struct GpuEvidenceBundle { + /// Stable GPU vendor identifier. + pub vendor: String, + /// Vendor-specific evidence format and version. + pub format: String, + /// Hex-encoded opaque evidence bytes, as represented by the JSON RPC. + pub evidence: String, +} + /// Response containing the complete NVIDIA GPU attestation output. #[derive(Debug, Serialize, Deserialize)] #[cfg_attr(feature = "borsh", derive(BorshSerialize, BorshDeserialize))]