Dstack-TEE · kvinwang · Aug 26, 2026 · Aug 25, 2026 · Aug 26, 2026 · Aug 26, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -35,6 +35,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 - certbot: a certificate covering both a name and its wildcard (`example.com` and `*.example.com`) could never be issued over dns-01. The two authorizations are answered under one `_acme-challenge.example.com`, each with its own TXT value, and the publish step cleared every TXT record at that name before writing its own -- so the second authorization deleted the record answering the first, and the order failed with `Correct value not found for DNS challenge`. Clearing leftovers from an aborted run is now done once per challenge name per issuance, and the records for one name accumulate instead of replacing each other; cleanup afterwards is unchanged, deleting each record this run created by id
+- certbot: editing `domains` in `certbot.toml` had no effect once a certificate existed. Issuance was skipped whenever `live/cert.pem` was present, whatever names it carried, and renewal read its name list back off that certificate rather than the configuration -- so an added or removed name never reached the CA, and the mismatch survived every renewal. The live certificate's DNS names are now compared against the configured list (as sets, case- and trailing-dot-insensitive) and a mismatch reissues, logging both lists. A reissue that fails does not take the renewal check down with it: a name the CA will not validate is reported on every cycle, while the certificate actually being served keeps renewing, and the failure is still what the run returns unless the renewal committed something of its own
+- certbot: `certbot cfg` attached every comment after `cf_api_url` to the wrong key, because an absent optional field shifts the generated document's keys out of step with the struct's. The template's documentation is now looked up by key name
 - gateway: a node removed via `RemoveNode` silently rejoined the cluster the next time it started, because every node re-registers its own sync address on boot. Once tombstone GC is collecting, that comeback is worse than an annoyance: a stale data directory diverges from every digest, and the divergence repair's full re-exchange resurrects records whose deletes the cluster already collected. Removal now writes a durable marker — a live record, so the GC can never eat it — that every gateway's sync endpoints enforce; a removed node's envelopes are refused until an operator re-admits it with `SetNodeUrl`. The refused node counts HTTP 403 sync rejections (`dstack_gateway_sync_rejected_total`), including removal lockouts and app-identity mismatches. Every gateway also exposes `dstack_gateway_node_last_seen_timestamp_seconds` per known node, so a long-offline gateway is a one-line alert instead of an ack-watermark puzzle
 - gateway: deleted KV records left a tombstone that nothing ever collected, so every deregistered CVM stayed on disk for the life of the deployment. Tombstones every peer has acknowledged are now dropped once every `tombstone_gc_writes` replicated writes (default 10000, zero disables); the trigger counts replicated writes rather than reading a clock, so nodes in a cluster collect in the same window without depending on time synchronization. A `SetTombstoneGcConfig` admin RPC stores an operator override in the KV itself, replicating one pace to every node
 - gateway: `Admin.RemoveCvm` now reports the outcome of the `inst/` tombstone alone. A failure to delete associated override or telemetry records is logged instead of failing the call, so a removal that did take effect is no longer reported as failed — which also aborted the local routing cleanup that follows it. Re-issuing a removal still sweeps up override and telemetry records orphaned by an earlier partial failure

diff --git a/docs/deployment.md b/docs/deployment.md
@@ -42,10 +42,10 @@ If you skip the KMS allowlist step, the VM may boot and the onboard UI may still
 1. Set up TDX host with dstack-vmm
 2. Deploy KMS as CVM (with auth server, capture its attestation info, and allowlist the KMS `mrAggregated` before bootstrap)
 3. Deploy Gateway as CVM
+4. [Zero Trust HTTPS](#4-zero-trust-https) - the gateway cannot serve an app over TLS until it holds a certificate for the domain
 
 **Optional Add-ons:**
 
-4. [Zero Trust HTTPS](#4-zero-trust-https-optional)
 5. [Certificate Transparency monitoring](#5-certificate-transparency-monitoring-optional)
 6. [Multi-node deployment](#6-multi-node-deployment-optional)
 7. [On-chain governance](./onchain-governance.md) - Smart contract-based authorization
@@ -355,29 +355,69 @@ Restart dstack-vmm to apply changes.
 
 ---
 
-### 4. Zero Trust HTTPS (Optional)
+### 4. Zero Trust HTTPS
 
-Generate TLS certificates inside the TEE with automatic CAA record management.
+The gateway issues its own certificates from inside the CVM. `dstack-gateway`
+links the `certbot` crate directly: it answers dns-01 challenges with the
+Cloudflare credential you give it, and keeps the ACME account key and every
+certificate in the CVM's WaveKV store. Neither key is ever written to the host,
+which is what makes the monitoring in step 5 worth running — every certificate
+the CT logs show for your domain should carry a public key the gateway
+published. Nothing on the host issues or holds these certificates; the `certbot`
+CLI under `dstack/certbot/cli` is a testing tool for the same crate and has no
+part in this path.
 
-Configure in `build-config.sh`:
+`bootstrap-cluster.sh` configures all of it, reading `CF_API_TOKEN`,
+`SRV_DOMAIN` and `ACME_STAGING` from the `.env` you filled in during step 3 and
+calling the gateway's admin API:
 
 ```bash
-GATEWAY_CERT=${CERTBOT_WORKDIR}/live/cert.pem
-GATEWAY_KEY=${CERTBOT_WORKDIR}/live/key.pem
-CF_API_TOKEN=<your-cloudflare-token>
-ACME_URL=https://acme-v02.api.letsencrypt.org/directory
+cd dstack/gateway/dstack-app/
+bash bootstrap-cluster.sh
 ```
 
-Run certbot:
+| RPC | What it sets |
+|---|---|
+| `SetCertbotConfig` | ACME directory URL and the renewal schedule |
+| `CreateDnsCredential` | the Cloudflare token used for dns-01, as the default credential |
+| `AddZtDomain` | a domain to keep a wildcard certificate for |
+
+Run it once per cluster. Additional nodes receive all three through cluster
+sync, so do not repeat it per node.
+
+Every name the gateway terminates TLS on needs its own ZT domain. The script
+adds `SRV_DOMAIN`; if app URLs sit one level deeper — `<id>-<port>.gateway.example.com`
+— then `gateway.example.com` needs an entry of its own, because a wildcard
+certificate for `*.example.com` does not cover subdomains of subdomains.
+
+Certificates are requested on the next renewal round rather than the moment a
+domain is added. Watch for them to arrive:
 
 ```bash
-RUST_LOG=info,certbot=debug ./certbot renew -c certbot.toml
+ADMIN_ADDR=127.0.0.1:9203  # GATEWAY_ADMIN_RPC_ADDR in .env
+curl -sf -H "Authorization: Bearer $ADMIN_API_TOKEN" \
+  "http://$ADMIN_ADDR/prpc/ListZtDomains" | jq '.domains[] | {domain: .config.domain, cert: .cert_status}'
 ```
 
-This will:
-- Create an ACME account
-- Set CAA DNS records on Cloudflare
-- Request and auto-renew certificates
+`has_cert: true` with a `not_after` roughly 90 days out means the domain is
+served. `POST /prpc/RenewCert` forces a round immediately instead of waiting
+for `renew_interval_secs`.
+
+Pin issuance to your own ACME account with `POST /prpc/SetCaa`, which writes
+CAA records naming Let's Encrypt and the gateway's account URI for every
+configured domain. Any other account is then refused by the CA rather than
+merely noticed after the fact by step 5.
+
+Start on Let's Encrypt staging (`ACME_STAGING=yes`), whose certificates are not
+browser-trusted but whose rate limits leave room for mistakes. Switching to
+production takes two calls: `SetCertbotConfig` with the production directory
+URL, then `RotateAcmeCredentials` to register an account there and re-pin every
+domain's CAA to it. Renewals refuse to run while the stored account and the
+configured ACME URL disagree, so do not skip the rotation.
+
+For the same flow driven by hand, one curl at a time, see the
+[Gateway service setup tutorial](./tutorials/gateway-service-setup.md#step-4-bootstrap-admin-api).
+For a multi-node cluster, see [Cluster deployment](../dstack/gateway/docs/cluster-deployment.md).
 
 ---
 

diff --git a/docs/dstack-gateway.md b/docs/dstack-gateway.md
@@ -2,41 +2,125 @@
 
 > **This guide is for self-hosted deployments** on your own TDX hardware. For cloud deployments, see [Quickstart](./quickstart.md).
 
-To set up dstack-gateway for production, you need a wildcard domain and SSL certificate.
+To set up dstack-gateway for production, you need a wildcard domain and a
+Cloudflare API token. You do not need to obtain a certificate yourself: the
+gateway links the `certbot` crate and runs ACME over dns-01 in its own process,
+keeping the ACME account key and every certificate in its WaveKV store. The
+`certbot` CLI under `dstack/certbot/cli` is a testing tool for the same crate
+and has no part in this path.
 
-## Step 1: Setup wildcard domain
+## Step 1: Set up the wildcard domain
 
 Set up a second-level wildcard domain using Cloudflare; make sure to disable proxy mode and use **DNS Only**.
 
 ![add-wildcard-domain](./assets/tproxy-add-wildcard-domain.jpg)
 
-## Step 2: Request a Wildcard Domain SSL Certificate with Certbot
+Then create an API token that can edit this zone's DNS records. The gateway uses
+it to publish the `_acme-challenge` TXT records that answer dns-01, and the CAA
+records that pin issuance to its own ACME account.
 
-You need to get a Cloudflare API Key and ensure the API can manage this domain.
+## Step 2: Configure `gateway.toml`
 
-Open your `certbot.toml`, and update these fields:
+Focus on these fields in the `core.proxy` section:
 
-- `acme_url`: change to `https://acme-v02.api.letsencrypt.org/directory`
-- `cf_api_token`: Obtain from Cloudflare
+- `base_domain`: the wildcard domain for the proxy
+- `listen_addr` & `listen_port`: listen on `0.0.0.0` and preferably `443` in production. If using another port, specify it in the URL (see [URL Format](#url-format))
 
-## Step 3: Run Certbot Manually and Get First SSL Certificates
+For example, if your base domain is `gateway.example.com`, app ID is `<app_id>`, listening on `80`, and dstack-gateway is on port 7777, the URL would be `https://<app_id>-80.gateway.example.com:7777`
+
+Leave `cert_chain` and `cert_key` unset. They load a certificate you already
+have from disk at startup, for the case where something else issues it; the
+gateway's own issuance does not use them and does not write them.
+
+Two more sections matter for certificates:
+
+```toml
+[core.admin]
+enabled = true
+address = "127.0.0.1:9016"
+auth_token = "<paste output of: openssl rand -hex 32>"
 
-```shell
-./certbot set-caa
-./certbot renew
+[core.sync]
+data_dir = "/var/lib/dstack-gateway/data"
 ```
 
-## Step 4: Update `gateway.toml`
+The admin API is where the ACME settings, the Cloudflare token and the domain
+list live — they are stored in the gateway's KV store and there is no file to
+put them in, so certificates cannot be issued without it. See
+[Admin API authentication](#admin-api-authentication) for the credential
+options. `data_dir` is where the ACME account key and the issued certificates
+are persisted; point it somewhere writable that survives restarts, or the
+gateway asks the CA for a fresh certificate every time it starts and will run
+into Let's Encrypt's rate limits. The section is named for cluster sync, but
+this store is used whether or not `enabled` is set.
+
+Start the gateway.
+
+## Step 3: Give the gateway its ACME configuration
+
+Open the admin dashboard at `http://<core.admin.address>` and fill in **Certbot
+Configuration**, **DNS Credentials** and **ZT-Domains**, or do the same over the
+admin API:
+
+```bash
+ADMIN_ADDR=127.0.0.1:9016
+AUTH=(-H "Authorization: Bearer $ADMIN_API_TOKEN")
+
+# Start on staging: its certificates are not browser-trusted, but its rate
+# limits leave room for mistakes.
+curl -sf -X POST "${AUTH[@]}" "http://$ADMIN_ADDR/prpc/SetCertbotConfig" \
+  -H "Content-Type: application/json" \
+  -d '{"acme_url":"https://acme-staging-v02.api.letsencrypt.org/directory",
+       "renew_interval_secs":3600,"renew_before_expiration_secs":864000,
+       "renew_timeout_secs":300}'
+
+curl -sf -X POST "${AUTH[@]}" "http://$ADMIN_ADDR/prpc/CreateDnsCredential" \
+  -H "Content-Type: application/json" \
+  -d '{"name":"cloudflare","provider_type":"cloudflare",
+       "cf_api_token":"'"$CF_API_TOKEN"'","set_as_default":true}'
+
+curl -sf -X POST "${AUTH[@]}" "http://$ADMIN_ADDR/prpc/AddZtDomain" \
+  -H "Content-Type: application/json" \
+  -d '{"domain":"gateway.example.com","port":443,"priority":100}'
+```
 
-Focus on these five fields in the `core.proxy` section:
+Add a ZT domain for every name the gateway terminates TLS on. An entry for
+`gateway.example.com` gets a certificate covering `*.gateway.example.com`, which
+is what app URLs live under — a certificate for `*.example.com` would not, since
+a wildcard does not span a further label. `port` is the port that domain is
+served on and `priority` breaks ties when more than one entry could be the
+default base domain.
 
-- `cert_chain` & `cert_key`: Point to the certificate paths from the previous step
-- `base_domain`: The wildcard domain for proxy
-- `listen_addr` & `listen_port`: Listen to `0.0.0.0` and preferably `443` in production. If using another port, specify it in the URL
+Certificates are requested on the next renewal round rather than the moment a
+domain is added. Watch for them to arrive:
 
-For example, if your base domain is `gateway.example.com`, app ID is `<app_id>`, listening on `80`, and dstack-gateway is on port 7777, the URL would be `https://<app_id>-80.gateway.example.com:7777`
+```bash
+curl -sf "${AUTH[@]}" "http://$ADMIN_ADDR/prpc/ListZtDomains" \
+  | jq '.domains[] | {domain: .config.domain, cert: .cert_status}'
+```
+
+`has_cert: true` with a `not_after` roughly 90 days out means the domain is
+served. `POST /prpc/RenewCert` forces a round immediately instead of waiting for
+`renew_interval_secs`.
+
+Pin issuance with `POST /prpc/SetCaa`, which writes CAA records naming Let's
+Encrypt and the gateway's ACME account URI for every configured domain, so no
+other account can have a certificate issued for them.
+
+Once the gateway serves traffic on staging certificates, switch to production:
+`SetCertbotConfig` with `https://acme-v02.api.letsencrypt.org/directory`, then
+`RotateAcmeCredentials` to register an account there and re-pin every domain's
+CAA to it. Renewals refuse to run while the stored account and the configured
+ACME URL disagree, so do not skip the rotation.
 
-### URL Format
+## Step 4: Adjust Configuration in `vmm.toml`
+
+Open `vmm.toml` and adjust dstack-gateway configuration in the `gateway` section:
+
+- `base_domain`: Same as `base_domain` from `gateway.toml`'s `core.proxy` section
+- `port`: Same as `listen_port` from `gateway.toml`'s `core.proxy` section
+
+## URL Format
 
 The gateway supports the following URL format:
 - `<app_id>[-<port>][<suffix>].<base_domain>`
@@ -57,13 +141,6 @@ Examples:
 
 Note: The `s` and `g` suffixes cannot be used together
 
-## Step 5: Adjust Configuration in `vmm.toml`
-
-Open `vmm.toml` and adjust dstack-gateway configuration in the `gateway` section:
-
-- `base_domain`: Same as `base_domain` from `gateway.toml`'s `core.proxy` section
-- `port`: Same as `listen_port` from `gateway.toml`'s `core.proxy` section
-
 ## Admin API authentication
 
 The gateway exposes a separate admin API (used for sync, WireGuard peer management, and other operator RPCs). Configure it in the `core.admin` section of `gateway.toml`:
@@ -73,19 +150,19 @@ The gateway exposes a separate admin API (used for sync, WireGuard peer manageme
 enabled = true
 address = "0.0.0.0:9016"
 # generate with: openssl rand -hex 32
-admin_token = "<paste output of: openssl rand -hex 32>"
+auth_token = "<paste output of: openssl rand -hex 32>"
 # alternatively, an Apache bcrypt htpasswd file (htpasswd -B -c admin.htpasswd admin)
 # htpasswd_file = "/etc/dstack/gateway-admin.htpasswd"
 insecure_no_auth = false
 ```
 
 - `enabled`: enable the admin API server.
 - `address`: bind address/port for the admin API.
-- `admin_token`: shared admin token. It can also be supplied via the environment variables `DSTACK_GATEWAY_ADMIN_TOKEN` or `ADMIN_API_TOKEN` instead of the config file.
-- `htpasswd_file`: path to an Apache bcrypt htpasswd file (create with `htpasswd -B -c admin.htpasswd admin`); only bcrypt entries are accepted. Can be used instead of, or alongside, `admin_token`.
+- `auth_token`: shared admin token. It can also be supplied via the environment variables `DSTACK_GATEWAY_ADMIN_TOKEN` or `ADMIN_API_TOKEN` instead of the config file. The older name `admin_token` is still accepted.
+- `htpasswd_file`: path to an Apache bcrypt htpasswd file (create with `htpasswd -B -c admin.htpasswd admin`); only bcrypt entries are accepted. Can be used instead of, or alongside, `auth_token`.
 - `insecure_no_auth`: development-only escape hatch that disables admin authentication. Never enable it on a network-reachable admin interface.
 
-The admin server is fail-closed: if it is enabled with no `admin_token` and no `htpasswd_file`, and `insecure_no_auth` is `false`, it refuses to start rather than exposing an unauthenticated admin API.
+The admin server is fail-closed: if it is enabled with no `auth_token` and no `htpasswd_file`, and `insecure_no_auth` is `false`, it refuses to start rather than exposing an unauthenticated admin API.
 
 Clients authenticate by sending `Authorization: Bearer <token>` or the `X-Admin-Token: <token>` header.
 
@@ -133,4 +210,3 @@ max(dstack_gateway_cluster_nodes_active) - min(dstack_gateway_cluster_nodes_acti
 | `dstack_gateway_kv_peer_buffered_logs` | Entries still buffered for a peer. Sustained growth means that peer stopped acknowledging and the two nodes are drifting apart. |
 | `dstack_gateway_cluster_cert_not_after_seconds` | Certificate expiry per domain; alert on `- time()` falling under the renewal window. Capped at 256 series — compare `dstack_gateway_cluster_cert_domains` to see whether the cap was hit. |
 | `dstack_gateway_kv_persist_failures_total` | Periodic snapshots are failing, so a restart replays a growing WAL. |
-| `dstack_gateway_kv_persist_failures_total` | Periodic snapshots are failing, so a restart replays a growing WAL. |