feat(certbot): issue certificates via dns-persist-01

CHANGELOG.md

@@ -25,6 +25,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - guest-agent: v1 `Attest` accepts `include_boottime_gpu_evidence` and returns the boot-time GPU attestation evidence in `AttestResponse.boottime_gpu_evidence`, so a verifier fetches the attestation and the GPU evidence in one round trip. It arrives as a `GpuEvidenceBundle` list -- the same shape `AttestGpu` returns, so a consumer writes one bundle parser and dispatches on `format`: `nvidia-nvattest-boottime-json-v1` is the record written at boot, `nvidia-nvattest-collect-evidence-json-v1` is collected on demand against a caller's nonce, and a verifier for one does not appraise the other. Absence is the empty list. The bundle's `evidence` is the nvattest output byte for byte as read from disk, because the only thing binding it to the boot is sha256 over precisely those bytes against the measured `gpu-attestation` event. `Attest` is also v1's sole CVM attestation entry point: the `VersionedAttestation` it returns already carries the TDX quote and event log, and unlike `GetQuote` it answers on every supported platform
 - sdk: `AppCompose` in the Go SDK gained `init_script`, `storage_fs`, `swap_size`, `event_log_version`, `port_policy` and `verity_volumes`, and `Requirements` gained `gpu_policy` in the Go and Python SDKs
 - shared API authentication (`dstack-api-auth`) protecting the full VMM HTTP/pRPC/UI surface and unifying Gateway/KMS admin auth: bearer/`X-Admin-Token`/HTTP Basic/bcrypt htpasswd, constant-time verification (#796)
+- certbot/gateway: opt-in `dns-persist-01` certificate validation ([draft-ietf-acme-dns-persist-01](https://datatracker.ietf.org/doc/html/draft-ietf-acme-dns-persist-01)), which issues without a DNS provider credential at all. `dns-01` needs write access to the zone on every order, so a gateway CVM holds a Cloudflare API token for the life of the deployment -- a token that rewrites the whole zone, not just `_acme-challenge`. Under `dns-persist-01` the zone owner publishes one `_validation-persist.<name>` TXT record naming the CA and the ACME account; nothing about it changes between orders, so certbot only ever reads DNS and the zone can be hosted anywhere, with no provider integration. Set `challenge = "dns-persist-01"` in `certbot.toml`, or `challenge` on a gateway ZT domain; the default stays `dns-01` and existing deployments are untouched. `certbot dns-records` prints the records to publish, `GetZtDomain`/`ListZtDomains` return them in `required_dns_records`, and the gateway logs them wherever it would otherwise have written DNS. CAA records carry `validationmethods=dns-persist-01` to match, so switching methods means republishing both. Two gateway operations change shape for such a domain: `SetCaa` skips it, having nothing to reconcile without write access, and `RotateAcmeCredentials` leaves it broken until the operator republishes -- the record still names the old account -- so the response now returns the new records in `required_dns_records`. A rotation that registers the account but fails to re-pin some domain's CAA now reports those domains in `repin_failed_domains` instead of failing the call: the account exists and the cluster is using it, so raising an error there reads as "nothing happened" and invites a retry that registers another account. **Experimental**: the draft is still changing, and Let's Encrypt serves the challenge on staging only pending an open working-group issue. Documented in `docs/certbot-dns-persist-01.md`.
+
+  Two settings changed shape for every deployment, not only `dns-persist-01` ones. `issuer_domain_name` now names the CA in the CAA records written for `dns-01` too -- its default is `letsencrypt.org`, so an untouched configuration writes exactly what it wrote before -- and it is validated where it is set, since it lands verbatim in an RFC 8659 `issue-value` and CAA is republished by deleting the old records first. The pre-order DNS self-check is now capped at half of `renew_timeout` rather than at its own configured value: both defaulted to 300s in the gateway, and the CLI wrapped a 300s wait in a 120s budget, so the timeout around the order always fired first and a missing record was reported as `certificate request timed out` instead of by name
 - gateway: `Admin.Status` reports `health_gating`, so an operator can see whether this node's health polling is switched on. With it off, instances that opted in sit at `unknown` forever and are all in rotation, which is otherwise indistinguishable on the dashboard from being held out pending a first answer
 - gateway: `Admin.SetInstanceReady` takes a CVM instance out of its app's load-balancing rotation without stopping it; instance-id routing stays open so the instance can still be investigated, and the setting survives re-registration
 - gateway: operator-set per-instance overrides now live under their own KV keys — `admin/<instance_id>/ready` and `admin/<instance_id>/port_policy` — instead of inside the instance record, so a CVM re-registration can no longer drop them and setting one cannot discard a peer's unsynced change to the other. An override left in an instance record by an earlier build is moved across on load

docs/certbot-dns-persist-01.md

@@ -0,0 +1,233 @@
+# Certificate issuance without a DNS credential (`dns-persist-01`)
+
+`dns-01` asks certbot to write a fresh `_acme-challenge` TXT record for every
+order, so whatever runs certbot holds a DNS API token with write access to the
+zone, forever. In dstack that token lives inside the gateway CVM. Attestation
+covers what the CVM is running, but a token is a token: anything that gets hold
+of it can rewrite the zone, including records that have nothing to do with
+certificates.
+
+`dns-persist-01` moves the proof out of the issuance loop. The zone owner
+publishes one record naming the CA and the ACME account allowed to issue:
+
+```dns
+_validation-persist.example.com. IN TXT "letsencrypt.org; accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/1234567890"
+```
+
+The account key proves who is asking, the record proves the zone owner agreed,
+and neither changes between orders. certbot reads DNS and never writes it, so
+the CVM holds no DNS credential and the zone can be hosted anywhere — no
+Cloudflare account, no API token, no provider integration.
+
+> **Experimental.** `dns-persist-01` is specified in
+> [draft-ietf-acme-dns-persist-01][draft], which is still changing: an open
+> working-group issue may add a client-key-derived value to the record, and
+> Let's Encrypt has said it will not deploy the challenge to production until
+> that is resolved. It is live on Let's Encrypt **staging** and in
+> [Pebble][pebble]. Treat the record format as unstable, and expect to
+> republish when the draft settles.
+
+[draft]: https://datatracker.ietf.org/doc/html/draft-ietf-acme-dns-persist-01
+[pebble]: https://github.com/letsencrypt/pebble
+
+## What the record means
+
+| Part | Effect |
+| --- | --- |
+| `letsencrypt.org` | Issuer Domain Name. A CA ignores records naming a different issuer, so one label can hold records for several CAs. |
+| `accounturi=` | The ACME account authorized to issue. Compared byte for byte — no case folding, no URI normalization. |
+| `policy=wildcard` | Extends the record to `*.example.com`. Without it the CA authorizes `example.com` alone and refuses wildcard orders. |
+| `persistUntil=` | Optional UNIX timestamp after which the CA stops accepting the record. |
+
+A wildcard order authorizes from its base name — `*.example.com` is validated
+against `_validation-persist.example.com`, not
+`_validation-persist.*.example.com` — so one record covers a name and its
+wildcard.
+
+Two things about the syntax bite in practice, because a CA rejects the whole
+record rather than ignoring the offending part: **no trailing semicolon**, and
+**no whitespace inside a value**. certbot renders records that satisfy both;
+copy them verbatim rather than retyping.
+
+The scope stops at the names above. Let's Encrypt does not walk up the tree, so
+a record on `example.com` does not authorize `sub.example.com` — give each base
+name its own record.
+
+## Standalone certbot
+
+`certbot` never writes DNS in this mode, so setup is: create the account, read
+the records off it, publish them, then issue.
+
+```toml
+# certbot.toml
+workdir = "/var/lib/certbot"
+acme_url = "https://acme-staging-v02.api.letsencrypt.org/directory"
+challenge = "dns-persist-01"
+issuer_domain_name = "letsencrypt.org"
+# auto_set_caa promises certbot keeps CAA in sync, which it cannot do without
+# write access. Leave it off and publish the CAA records below by hand.
+auto_set_caa = false
+domains = ["example.com", "*.example.com"]
+renew_interval = 3600
+renew_days_before = 10
+renew_timeout = 120
+max_dns_wait = 300
+```
+
+`cf_api_token` is unused and can be left out; certbot warns if one is set.
+
+```console
+$ certbot init -c certbot.toml
+INFO certbot::bot: creating new ACME account
+INFO certbot::bot: created new ACME account: https://acme-staging-v02.api.letsencrypt.org/acme/acct/1234567890
+
+$ certbot dns-records -c certbot.toml
+_validation-persist.example.com. IN TXT "letsencrypt.org; accounturi=https://acme-staging-v02.api.letsencrypt.org/acme/acct/1234567890; policy=wildcard"
+example.com. IN CAA 0 issue "letsencrypt.org;validationmethods=dns-persist-01;accounturi=https://acme-staging-v02.api.letsencrypt.org/acme/acct/1234567890"
+example.com. IN CAA 0 issuewild "letsencrypt.org;validationmethods=dns-persist-01;accounturi=https://acme-staging-v02.api.letsencrypt.org/acme/acct/1234567890"
+```
+
+Publish all three, wait for them to propagate, then issue:
+
+```console
+$ certbot renew --once -c certbot.toml
+INFO certbot::acme_client: requesting new certificates for example.com, *.example.com
+INFO certbot::bot: created new certificate
+```
+
+Renewals need nothing further. The record stays, and `certbot renew` reuses it
+for every order.
+
+The CAA records are optional but recommended — they stop any other account, at
+Let's Encrypt or elsewhere, from being issued for your name. Note the
+`validationmethods=dns-persist-01` in them: a CAA record left pinned to
+`dns-01` refuses every `dns-persist-01` order, so switching methods means
+updating CAA and the validation record together.
+
+### When issuance fails
+
+certbot checks its own resolver before starting an order, and says exactly what
+it expected to find:
+
+```
+WARN certbot::acme_client: no TXT record at _validation-persist.example.com matches the expected value: letsencrypt.org; accounturi=https://acme-staging-v02.api.letsencrypt.org/acme/acct/1234567890; policy=wildcard
+Error: order is invalid: API error: Checking DNS-PERSIST-01 challenge TXT record with issuer-domain-name "letsencrypt.org": accounturi mismatch: expected "https://acme-staging-v02.api.letsencrypt.org/acme/acct/1234567890", got "https://acme-staging-v02.api.letsencrypt.org/acme/acct/9876543210" (urn:ietf:params:acme:error:unauthorized)
+```
+
+The check is advisory and never blocks an order: certbot's resolver is not the
+CA's, and its expectation can be stricter than what the CA would accept. A
+warning with a successful issuance underneath it is a resolver difference, not
+a problem. A record that genuinely does not match costs the full `max_dns_wait`
+before the order is sent, because the check waits out its budget first.
+
+If the CA rejects the order, compare the published record against
+`certbot dns-records` character by character. The usual causes are a stale
+`accounturi` after the account was recreated, a missing `policy=wildcard` on a
+wildcard order, and an issuer domain name that does not match the CA.
+
+## dstack-gateway
+
+A ZT domain picks its method with the `challenge` field, which defaults to
+`dns-01` — existing domains are unaffected:
+
+```json
+{ "domain": "app.example.com", "port": 443, "challenge": "dns-persist-01" }
+```
+
+Such a domain needs no `dns_cred_id`, and the gateway CVM never receives a DNS
+credential for it. `GetZtDomain` and `ListZtDomains` return the records to
+publish in `required_dns_records`, and the gateway logs them whenever it cannot
+write DNS itself:
+
+```
+WARN cert[app.example.com]: publish this record by hand: _validation-persist.app.example.com. IN TXT "letsencrypt.org; accounturi=...; policy=wildcard"
+```
+
+The gateway issues for `*.{domain}` only, so each domain needs one validation
+record with `policy=wildcard`.
+
+### Setup order
+
+The records name the ACME account, so the account comes first:
+
+1. `SetCertbotConfig` — `acme_url`, and `issuer_domain_name` if the CA is not
+   Let's Encrypt.
+2. `RotateAcmeCredentials` — registers the account and returns its `account_uri`.
+   It needs no ZT domain: registration proves nothing about a domain. Every call
+   registers a *new* account, so call it once here; running it again later is a
+   rotation, with the consequences described below.
+3. `AddZtDomain` with `challenge: "dns-persist-01"`. No `dns_cred_id`: the
+   domain needs none, and the gateway CVM never receives one for it.
+4. Read the records — the dashboard shows them when the domain is added and
+   again under its **Records** button, and `GetZtDomain`, `ListZtDomains` and
+   the gateway log all carry them — then publish the `_validation-persist` TXT
+   record, plus the CAA records if you want issuance pinned to this account.
+5. `RenewZtDomainCert`. Renewals from here need nothing further.
+
+Adding another dns-persist-01 domain later starts at step 3: the account already
+exists, so the new domain's records are available as soon as it is. A domain's
+challenge is chosen when it is added — in the dashboard's ZT-Domain form as well
+as over the API — and carried forward by every edit. `UpdateZtDomain` replaces
+the whole record, so `challenge` is optional there and omitting it means "leave
+it as it is": a caller that predates the field — a cached dashboard bundle, a
+script, an older SDK — cannot downgrade a `dns-persist-01` domain to `dns-01` by
+editing something else, which its hand-published CAA would then refuse. Send the
+field explicitly to switch a domain over.
+
+### Naming the CA
+
+For a non-production ACME server, set `issuer_domain_name` in the global certbot
+config — `Admin.SetCertbotConfig`, or the field of that name in the dashboard's
+Certbot Configuration — to whatever that server puts in `issuer-domain-names`,
+`pebble.letsencrypt.org` for Pebble. Empty means `letsencrypt.org`.
+
+It also names the CA in the CAA records certbot writes for `dns-01` domains, so
+one setting covers both challenges rather than pinning CAA to Let's Encrypt
+while orders go somewhere else. `acme_url` is one setting for the whole
+deployment; this follows it.
+
+The value is checked when it is set. It ends up verbatim inside an RFC 8659
+`issue-value`, and publishing CAA deletes the records it replaces before writing
+the new ones, so a name carrying a space or a `;` would leave the zone holding a
+malformed `issue` property with nothing valid behind it — CAA that forbids every
+issuer. Anything that is not a DNS name is refused by `SetCertbotConfig`, and by
+`certbot` at startup.
+
+Publishing CAA installs a temporary `;` guard, deletes the records it replaces,
+writes the new ones and drops the guard. A run interrupted part-way leaves the
+guard behind, which denies every issuer until a later run finishes the job — so
+each run now sweeps stale guards before installing its own, and a rerun is
+enough to recover.
+
+### How long the check waits
+
+The self-check before each order waits for the record to become visible, but
+never longer than half of `renew_timeout`, the budget for the whole renewal.
+Past that the outer timeout fires first and the renewal ends as `certificate
+request timed out` with no mention of the record it could not find — the
+opposite of what the check is for. Both values defaulted to 300s in the gateway
+(and the CLI wrapped a 300s wait in a 120s budget), so this was the default
+behaviour rather than an edge case.
+
+Two operations behave differently on these domains:
+
+- **`SetCaa`** skips them. There is nothing to reconcile without write access;
+  the records are logged instead, and the summary reports how many were left to
+  the operator.
+- **`RotateAcmeCredentials`** is not self-service. Rotation moves the cluster
+  to a new ACME account, and every `_validation-persist` record still names the
+  old one, so orders for those domains fail until the operator republishes. The
+  response returns the new records in `required_dns_records` and the gateway
+  logs them; publish before the next renewal comes due.
+
+  A rotation that registers the account but cannot re-pin every `dns-01`
+  domain's CAA still succeeds — the account exists and the cluster is on it —
+  and names those domains in `repin_failed_domains`. Finish with `SetCaa`
+  rather than another rotation, which would register yet another account
+  against a rate-limited quota. The dashboard reports both lists in one dialog
+  after a rotation, with a **Run SetCaa** button for the second.
+
+## Related
+
+- [dstack-gateway](dstack-gateway.md) — gateway architecture and TLS termination
+- [deployment.md](deployment.md#4-zero-trust-https-optional) — the `dns-01` setup this replaces