feat(certbot): issue certificates via dns-persist-01 - #1132
Open
kvinwang wants to merge 37 commits into
Open
Conversation
kvinwang
changed the title
dns-persist-01
kvinwang
force-pushed
the
feat/certbot-dns-persist-01
branch
2 times, most recently
from
August 25, 2026 12:11
9c9da5e to
f2b1e3c
Compare
This was referenced Aug 25, 2026
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
kvinwang
force-pushed
the
feat/certbot-dns-persist-01
branch
from
August 26, 2026 03:31
f190fb2 to
a05a812
Compare
kvinwang
force-pushed
the
feat/certbot-dns-persist-01
branch
from
August 26, 2026 04:02
a05a812 to
89538f2
Compare
dns-01 rewrites the zone on every order, so certbot holds a DNS provider credential for the life of the deployment. dns-persist-01 (draft-ietf-acme-dns-persist-01) proves control with a `_validation-persist` TXT record published once, out of band: the record names the CA and the ACME account, nothing about it changes between orders, and certbot only ever reads DNS. `ValidationMethod` replaces the `Dns01Client` field on `AcmeClient`, so the dns-01-only state -- the provider client and the TXT TTL -- lives in the one variant that has any use for it, and dns-persist-01 cannot be constructed holding a credential it would never call. `check_dns` widens from an exact match on the key authorization to matching whatever the challenge expects, and the cleanup pass skips records certbot did not create: a persistent record is the operator's and outlives every order. The self-check stays advisory for both methods. Our resolver is not the CA's, and under dns-persist-01 our expectation can be stricter than the CA's -- the challenge's `issuer-domain-names` are not exposed by instant-acme -- so a record we cannot see is named in a warning and the order proceeds. CAA content now names the challenge in use, because a record pinned to `validationmethods=dns-01` refuses every dns-persist-01 order. The dns-01 string is unchanged byte for byte, pinned by a test, so records published by earlier releases keep matching. `required_dns_records` renders the whole one-time setup -- validation record plus CAA -- from an account URI rather than a live client, so a caller holding only stored credentials can render it without a round trip to the CA. The record grammar is an RFC 8659 issue-value, and the parser mirrors what CAs run down to the parts that reject rather than ignore (trailing semicolon, repeated tag, whitespace in a value), so certbot never renders a record the CA would refuse or accepts one it would. The gateway keeps its dns-01 behaviour; the call sites move to the new constructor unchanged.
Under dns-persist-01 certbot cannot write the records it needs, so the records are the setup. `certbot dns-records` prints them as zone-file lines for the configured domains, ready to paste into any provider, once `certbot init` has registered the account they name. `challenge` and `issuer_domain_name` join certbot.toml, and `cf_api_token` becomes optional -- a token left configured alongside dns-persist-01 is warned about rather than silently ignored. `auto_set_caa` is refused outright with dns-persist-01: it promises certbot keeps CAA in sync, and without write access nothing can keep that promise.
A gateway CVM running dns-01 holds a Cloudflare token with write access to the operator's whole zone. Attestation covers what the CVM runs, not what becomes of a secret it holds, so that token is the widest credential in the deployment and it exists only to write one TXT record per order. dns-persist-01 removes it: control comes from a `_validation-persist` record the operator publishes once, and the CVM never gets DNS write access at all. `ZtDomainConfig.challenge` picks the method per domain and defaults to dns-01, so records written before the field existed decode as the method those deployments were using -- pinned by a test over both the named and the legacy positional msgpack encodings. Such a domain needs no DNS credential, and `validation_for` never looks one up for it. `GetZtDomain` and `ListZtDomains` return the records to publish in `required_dns_records`, rendered from the stored account URI with no ACME round trip so the listing endpoints stay cheap; it comes back empty rather than failing when no account exists yet. Two operations cannot be self-service for such a domain, and say so rather than failing silently: - `SetCaa` skips it and logs the records instead. There is nothing to reconcile without write access, and one such domain must not make the RPC unusable for the dns-01 domains beside it; the summary reports how many were left to the operator. - `RotateAcmeCredentials` moves the cluster to a new account while every `_validation-persist` record still names the old one, so orders for those domains fail until the operator republishes. The response now carries the new records in `required_dns_records`, rendered after the switch so they name the account the cluster actually moved to, and `domains_updated` counts only the domains whose CAA was re-pinned.
… its first byte
The scan walked bytes and rendered the offender with `*byte as char`, which
reinterprets one byte of a multi-byte character as a Latin-1 codepoint. So
`lé.org` was rejected for a `Ã` -- a character that appears nowhere in the
value the same message quotes back through `{configured:?}`, leaving the
message contradicting itself about the one thing it exists to report.
Scanning `chars()` costs nothing here: the length bound stays on bytes, where
DNS's 63 is octets, and by the time the first/last check indexes the label the
scan has already established it is ASCII.
…t tried The failure count was reported over `total`, which counts every domain the rotation covered -- including the dns-persist-01 ones that `continue` before a re-pin is ever attempted, having no CAA the gateway can write. Five domains, three of them dns-persist-01 and the other two failing, logged "failed to re-pin CAA for 2/5 domains", which says three were re-pinned when none were. The response said `domains_updated: 0` for the same rotation, so the log and the RPC contradicted each other. The three numbers now derive from one `RepinTally` rather than from three separate reads of the loop's leftovers, which is what let them disagree. That also makes the arithmetic testable without an ACME server: reaching the partial-failure path end to end needs registration to succeed and a CAA write to fail, but the contract worth pinning is the tally, and it is now pure. `saturating_sub` because this runs after the account is published -- an underflow panic here would abandon a rotation the cluster has already committed to.
A rotation that could not re-pin every domain's CAA reported it through `showToast`, which removes itself after three seconds and which the DNS-records modal opened immediately afterwards covers. Everything else the handler does reads as success -- the account URI cell flips to the new account, the domain table reloads -- so once the toast expired the page held no trace that re-pinning failed, for the operation whose whole point is that the operator must not simply run it again. Both lists are the same question, "what is left to do before this rotation is finished", so they share one dialog. A dns-01-only cluster has the re-pin half without the records half, so either alone opens it. The re-pin section carries a Run SetCaa button, which is also the first time that operation is reachable from the dashboard at all -- until now the docs pointed at a remedy the UI could not perform. Rotation is reported rather than raised because repeating it registers another account; SetCaa is the opposite, idempotent and raising on partial failure, so it is safe to offer as a button and reaching its success case means every domain took.
`challenge` was a plain `string`, so the server could not tell "omitted" from "dns-01". `UpdateZtDomain` replaces the whole record, so any caller that does not know the field -- a cached dashboard bundle, an operator's script, an older SDK -- downgraded a dns-persist-01 domain to dns-01 cluster-wide by editing something unrelated. The domain's hand-published CAA names `validationmethods=dns-persist-01`, so every order after that is refused by the CA until someone rewrites the zone. The compat layer cannot cover this: `carry_unknown_fields` restores fields the writing *binary* does not declare, and this binary declares `challenge`. Making it `optional` is what `issuer_domain_name` already does for the same reason, and `merge_certbot_config` already assigns only on `Some`. An add has nothing to preserve, so absence there is still the historical default; an explicit empty string still reads as dns-01, which is what a proto3 zero value carries. Fixing it at the protocol rather than in the bundled dashboard also covers the callers the dashboard fix could not reach -- it ships with the server, every other client does not.
`cf_api_token` became `#[serde(default)]` so a dns-persist-01 config can omit
it, which it has no use for. That also stopped a dns-01 config from failing
deserialization when it forgets the token: it reached Cloudflare instead and
came back with
failed to list zones: {"code":6111,"message":"Invalid format for
Authorization header"}
naming a header the operator never wrote, where it used to say
`missing field 'cf_api_token'`. Still a startup failure either way, since
resolving the zone is authenticated -- but the diagnosis had to be recovered.
Checked in the dns-01 arm, symmetric with the `auto_set_caa` bail two arms
below, and it names the alternative: a deployment that does not want to hold a
provider credential has one.
kvinwang
force-pushed
the
feat/certbot-dns-persist-01
branch
from
August 26, 2026 06:14
89538f2 to
9e30e53
Compare
`set_caa_records` installs `;` guards, deletes the records they replace, writes the new ones and drops the guards. A run interrupted anywhere in the middle leaves the guards behind, and a `;` issue-value denies every issuer -- the gateway's own error text tells the operator to rerun until it succeeds. But the rerun's first call re-adds a byte-identical `0 issue ";"`. A provider that rejects exact duplicates -- Cloudflare does for TXT -- fails the rerun at step one, and the zone never recovers from the advice it was given. So a rerun reuses the record it finds instead of adding a second one. The obvious alternative, sweeping the stale guards first, fixes the same thing and opens a worse hole: between the delete and the add the name has no issue or issuewild record at all, and absent CAA is not "denied" but "any CA may issue" (RFC 8659 §3). The trade is not symmetric -- a stranded guard is a renewal that fails until someone reruns, loudly and recoverably, while a certificate obtained during a fail-open window outlives the window and has to be found and revoked. Adoption keeps the deny-all continuous, which is the only reason to write a guard in the first place. Guards beyond the two adopted are ordinary issue records as far as step 2 is concerned, so they are removed there as before.
`certbot cfg` labelled every key after the first absent one with its neighbour's comment, because it walked the document by position and indexed `FIELD_DOCS` by the same counter while `toml_edit` drops `None` options. `max_dns_wait` came out as "Renew timeout in seconds", next to the one value whose relationship with `renew_timeout` decides whether a missing record is reported by name -- in the file `certbot cfg` exists to hand a new operator. Fixed upstream in da36db8 with no test. These are that test: one holding every rendered key to the comment `get_field_docs` returns for it, so a future drift is caught wherever it lands rather than at the one key that drifted, and one naming `max_dns_wait` because it is the key that made the bug visible. Both fail against the positional lookup and pass against the fix.
The challenge had no end-to-end coverage: unit tests pin the record grammar and the tallies, but not that a domain reaches the CA without a DNS credential. Phase 10 asserts what this harness can actually decide -- the domain is accepted holding no credential, the record an operator must publish is rendered on the base name with `policy=wildcard`, the challenge survives an update that omits it, and nothing for that domain ever reaches the provider API. Issuance itself is not asserted, because the pinned Pebble image cannot serve it: its binary carries `dns-01`, `http-01`, `tls-alpn-01` and `dns-account-01`, and neither `dns-persist-01` nor `_validation-persist`. A run confirms this -- the order is created, the authorization fetched, and no matching challenge is found. Closing that gap needs a Pebble built with the draft, and the comment on `PERSIST_DOMAIN` says so rather than leaving the next reader to find out. The domain is kept out of `CERT_DOMAINS` so the certificate phases do not expect one for it, and the add deletes first, since the KV store outlives a run and an "already exists" rejection would mask whether the add is accepted.
The phase added a moment ago stopped short of the CA, because the pinned Pebble image could not serve the challenge -- its binary carried dns-01, http-01, tls-alpn-01 and dns-account-01, and neither "dns-persist-01" nor "_validation-persist". Upstream Pebble has implemented the draft since (va.validateDNSPersist01, parsing the record as an RFC 8659 issue-value), so the image is rebuilt from upstream v2.10.1 with only the `-http` flag this harness has always needed, and the tag is pinned rather than floating on :latest. The flag now lives at github.com/kvinwang/pebble branch http-flag instead of in an unrecorded working tree. So the challenge is covered end to end: selected out of the authorization, posted ready, finalized, and the certificate fetched -- for a domain the gateway holds no credential for. The DNS answer is the only part standing in, since nothing publishes the record here and Pebble runs with PEBBLE_VA_ALWAYS_VALID. certbot warns and proceeds by design, because the CA's DNS view is not this node's. Two orderings the suite has to respect, both learned by getting them wrong: - Every record names the ACME account, so `required_dns_records` is empty until one exists, and in this harness issuing is what creates it. The record assertion therefore runs after issuance, not before. - `renew_timeout` bounds the DNS self-check at half its value. Left at the default the order would sit for 150s waiting on a record that never arrives, so the harness sets 60s and the poll outlasts it. `issuer_domain_name` is set to pebble.letsencrypt.org: a record naming a CA the challenge does not list is ignored, and Pebble does not answer to letsencrypt.org.
The mock has served TXT on UDP/53 since it was written, and both e2e suites point Pebble's -dnsserver at it -- yet both also set PEBBLE_VA_ALWAYS_VALID, because the wiring never worked. Pebble sets `dnsClient.Net = "tcp"` whenever it is given a custom resolver (va/va.go), so it never sent a UDP query to answer. The DNS side of the harness looked complete and validated nothing. TCP is the same responder behind RFC 1035 §4.2.2 framing: a two-byte length in front of each message, in both directions. `dns_response` is unchanged and serves both paths, so a record is answered identically however it is asked for.
… around it The phase issued a certificate but proved nothing about the record: Pebble ran with PEBBLE_VA_ALWAYS_VALID, so it accepted a challenge it never looked up. Everything the record's grammar has to satisfy -- the `; ` separator, the issuer name the CA answers to, `policy=wildcard` covering the base name -- was pinned only by unit tests written against a reading of Boulder. Now the harness publishes the record the way a zone owner would, once, before the first order, taking it verbatim from what the gateway reports; Pebble looks it up and parses it as an RFC 8659 issue-value. A record this implementation renders and a CA implementation rejects now fails the suite. That is possible because the mock answers TCP, and worth doing because the gateway itself never writes the record -- which is the claim the last assertion checks, scoped to the `_acme-challenge` name certbot would have written rather than to the zone being empty. The mock is built from the repo instead of pulled as kvin/mock-cf-dns-api, which was HTTP-only and could not have served this. Both suites now build the same image.
There were two. `tools/mock-cf-dns-api` was 882 lines of Flask and gunicorn behind a published `kvin/mock-cf-dns-api:latest`, and after the gateway suite switched to a repo-built mock, nothing used it. The survivor is 354 lines of stdlib with no dependencies, and it is the only one that answers DNS, which is what lets an ACME server validate rather than be told to skip. Nothing is lost: certbot's Cloudflare client calls four endpoints -- list zones, list records, create, delete -- and the survivor implements exactly those. The dropped extras (PUT, single-record GET, a log buffer, a web UI) had no caller. Moved to tools/ because it is no longer one suite's fixture: both build it, and the gateway suite was reaching into the other suite's directory through four levels of `..` to find it.
CodeQL flags the DNS listeners for binding every interface. In a container that is the only address that answers -- peers reach this by an address on a private compose network that the process cannot know in advance -- so the default stays, but it is now one documented function rather than a literal repeated at each listener, and MOCK_CF_BIND narrows it for anyone running the mock outside a container, where a test double that trusts every caller should not be on every interface.
The phase proved a certificate issues from a published record. It could not
prove the record had to be that record: the only failing case it exercised was
the domain having no record at all.
Three cases the CA now decides, because it looks the record up rather than
being run with PEBBLE_VA_ALWAYS_VALID:
- A record correct in every respect except `policy=wildcard` does not answer
the order. The gateway only ever orders `*.{domain}`, so every authorization
it answers is a wildcard one, and the draft has a CA accept those only from
a record carrying the parameter.
- Adding that one parameter, and nothing else, issues. Without this the case
above could pass for any reason at all.
- A record naming a different account does not authorize. If it did, the
record would authorize anybody who found it and the challenge would prove
nothing.
Plus the two gateway operations that change shape for a domain the gateway
cannot write: `SetCaa` skips it rather than failing the whole run or writing
to a provider it has no credential for, and `RotateAcmeCredentials` returns
the records to republish, naming the account it just registered -- the old
records name the account it replaced, and nothing else reports them.
Each refusal gets its own domain. `DeleteZtDomain` keeps the certificate on
purpose, so a domain that has already issued reports that certificate after
being re-added, and a refusal would read as a pass. The assertions are
inverted rather than matched against an error string, because what is being
claimed is that no certificate appears.
The CAA assertion splits the response into records before matching, rather
than looking for `"type"` and `"name"` adjacent in one pattern: the mock
serializes with sorted keys, so `"type"` never follows `"name"`, and a pattern
spanning the two can only fail to match -- which in a negative assertion is a
test that passes whatever the gateway did.
kvinwang
force-pushed
the
feat/certbot-dns-persist-01
branch
from
August 26, 2026 08:35
9e30e53 to
5ed087c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Every dstack certificate today is issued with
dns-01, which requires a fresh_acme-challengeTXT record per order. That means whatever runs certbot holds aCloudflare API token with write access to the operator's zone, permanently —
and in the gateway's case that token lives inside a CVM:
Attestation covers what the CVM is running. It does not cover what becomes of a
long-lived secret the CVM holds. So the widest credential in a dstack deployment
— one that can rewrite
MX,A, anything — exists solely to write one TXTrecord per order and delete it again. It also forces the base domain onto a
provider certbot has an integration for; today that list is exactly
{Cloudflare}.Fix
Add
dns-persist-01(draft-ietf-acme-dns-persist-01, CA/Browser Forumballot SC-088v3) as an opt-in validation method. The zone owner publishes one
record naming the CA and the ACME account:
The account key proves who is asking, the record proves the zone owner agreed,
and nothing about it changes between orders. certbot only ever reads DNS, so the
CVM holds no DNS credential at all and the zone can be hosted anywhere.
Shape of the change
ValidationMethodreplaces theDns01Clientfield onAcmeClient, so thedns-01-only state — the provider client and the TXT TTL — lives in the one
variant that uses it, and
dns-persist-01cannot be constructed holding acredential it would never call:
Opt in per deployment (
challenge = "dns-persist-01"incertbot.toml) or pergateway ZT domain (
ZtDomainConfig.challenge). The default staysdns-01everywhere and existing deployments are untouched; a stored
ZtDomainConfigwritten before the field existed decodes as
dns-01, pinned by a test over boththe named and the legacy positional msgpack encodings.
Because certbot cannot write the records under this method, the records are
the setup, and they are surfaced everywhere an operator would look:
certbot dns-recordsprints them,GetZtDomain/ListZtDomainsreturn them inrequired_dns_records, and the gateway logs them wherever it would otherwisehave written DNS.
Details worth a reviewer's attention
CAA has to move with the method. The value was hardcoded to
validationmethods=dns-01; a record left pinned to that refuses everydns-persist-01 order (and vice versa). It now names the method in use. The
dns-01 string is byte-for-byte unchanged, pinned by a test, so records published
by earlier releases keep matching.
The self-check stays advisory. Our resolver is not the CA's, and under
dns-persist-01 our expectation can be stricter than the CA's —
instant-acmedoes not expose the challenge's
issuer-domain-names, so we compare against theconfigured one. A record we cannot see is named in a warning and the order
proceeds; the check can never fail an issuance that would have worked.
Two gateway operations cannot be self-service, and say so instead of failing
silently:
SetCaaskips such a domain and logs the records. One dns-persist-01 domainmust not make the RPC unusable for the dns-01 domains beside it.
RotateAcmeCredentialsis the sharp edge: it moves the cluster to a newaccount while every
_validation-persistrecord still names the old one, soorders for those domains fail until the operator republishes. The response now
carries the new records in
required_dns_records, rendered after the switchso they name the account the cluster actually moved to, and
domains_updatedcounts only domains whose CAA was re-pinned.
The record grammar rejects rather than ignores. It is an RFC 8659
issue-value, and a CA fails the whole record on a trailing semicolon, arepeated tag, or whitespace inside a value. The parser here mirrors the one CAs
run (Boulder's
va/dns_persist.go), so certbot never renders a record the CAwould refuse nor accepts one it would.
Verification
End-to-end against Let's Encrypt staging, with a real Cloudflare-hosted
zone (
kvin.wang) and no DNS credential in the certbot config at all -- thethree records were published by hand from what
dns-recordsprinted:The single record answered both authorizations at the real CA, which is the
claim this design rests on: the debug log shows the base-name authorization
expecting
letsencrypt.org; accounturi=...and the wildcard one expecting thesame with
; policy=wildcard, both satisfied by the one published record.After issuance and a forced renewal, the zone holds exactly the three records
that were published by hand -- no
_acme-challenge, nothing added, nothingremoved. certbot never wrote DNS, which is the point of the mode.
Failure path at the real CA. A second account (fresh workdir, so a fresh
accounturi) against the same published record:Staging's rejection is more specific than Pebble's, so the doc now quotes this
one; it also notes that a genuinely mismatched record costs the full
max_dns_waitbefore the order is sent, since the advisory check waits out itsbudget first.
End-to-end against Pebble v2.10.1, which implements the draft
(
va.validateDNSPersist01,_validation-persist,policy=wildcard,persistUntil), withpebble-challtestsrvas the zone. Full flow, no DNScredential anywhere in the config:
One record covers the bare name and the wildcard — which is the behaviour the
CA implements, not an assumption: a wildcard authorization strips the
*.andlooks up the base name.
Failure paths, same harness. With a record naming a different account, the
warning names exactly what was expected and the CA rejects the order:
auto_set_caa = true,set-caa, and a leftovercf_api_tokeneach producetheir own message rather than a silent skip:
dns-01 regression, live. The same binary, same Pebble, against the repo's
mock-cf-dnsCloudflare API fromtest-suites/full-stack-compose— a wildcardcertificate still issues through the provider-API path:
Unit tests. 20 in
dns_persistcovering the grammar against the draft's ownexample and each rule a CA enforces (case-insensitive tags, byte-exact
accounturi, wildcard policy in both directions, trailing semicolon, duplicatetag, whitespace in a value,
persistUntilexpiry, several records at onelabel); 6 in
acme_clientpinning the rendered records, including thebyte-identical dns-01 CAA value; 1 in
gateway/src/kvpinning theZtDomainConfigdecode.Rebased onto
nextafter #1129 (theinstant-acmeupgrade this needed),#1130 (authoritative-nameserver DNS checks) and #1133 (the ACME follow-ups)
landed. Both merges kept the incoming structure whole:
check_dnsresolves each challenge's authoritative nameservers andfalls back to the system resolver. Only the "does this answer match" step
widens, from an exact comparison to whatever the challenge expects. Its zone
walk now drops any leading underscore label rather than
_acme-challengespecifically, so
_validation-persisttakes the same path instead of burninga round trip on a name that cannot carry NS records.
challenge_domainhelper — which exists becauseAuthorizedIdentifier'sDisplayrenders the wildcard prefix and wouldpublish at
_acme-challenge.*.example.com— is generalised over the challengekind rather than duplicated. That trap is identical for
_validation-persist,so its test now asserts both methods keep the bare name.
Every e2e run below is on the rebased tree.
dns-01 regression, live. The same binary, same Pebble, against the repo's
mock-cf-dnsCloudflare API fromtest-suites/full-stack-compose— a wildcardcertificate still issues through the provider-API path:
Build/lint.
cargo fmt --all --check,cargo clippy -- -D warnings -D clippy::expect_used -D clippy::unwrap_used --allow unused_variables(the CIinvocation),
cargo clippy -p certbot -p certbot-cli -p dstack-gateway --all-targets -- -D warnings,cargo test -p certbot -p dstack-gateway(31 + 290 passing),
prek run --all-files, andreuse lintall pass.Not covered: a live run against Let's Encrypt staging. It offers the
challenge today, but with
accounturiomitted from the challenge object(Boulder's
AccountURIPrefixis unset there), so the account URI comes from ourown
kideither way — which is what this implementation does, sinceinstant-acmeexposes neither field. Pebble is the closer match to the currentdraft.
Review follow-ups
Five findings from review, each verified against the CA's own implementation
rather than taken on description:
The self-check read records more strictly than Boulder does. RFC 8659's
grammar is
parameter = tag *WSP "=" *WSP valueand Boulder trims each halfseparately (
va/dns_persist.go); this trimmed the parameter as a whole, soaccounturi = <uri>parsed its tag as"accounturi "and reported themandatory parameter missing. Boulder also enforces uniqueness only on tags it
recognizes, since the draft has it ignore unknown ones outright, and folds the
issuer name (lowercase, root dot dropped) before deciding whether a record is
its own. All three now match, with the empty-
accounturicase Boulder rejectsadded too.
The cost was not theoretical. Against Let's Encrypt staging, with the published
record padded around its separators -- legal, and issued from -- before and
after:
Issuance succeeded either way -- the check is advisory -- but every renewal
burned the full wait and logged two warnings naming a record that was in fact
correct.
The gateway's
issuer_domain_namecould not be set. It was read by thedns-persist-01 path and documented as the knob for a non-Let's-Encrypt ACME
server, but no proto field, RPC or config path ever wrote it, so it was
permanently empty and the gateway was hardwired to
letsencrypt.org-- againstPebble, every order would have failed with no way to correct it. It is now a
field on
SetCertbotConfigRequest/CertbotConfigResponseand in thedashboard's Certbot Configuration.
A dashboard edit silently downgraded a domain to dns-01.
UpdateZtDomainreplaces the whole record and an empty
challengedecodes asdns-01, but theedit form never round-tripped the field: changing a priority would have moved a
dns-persist-01 domain onto the dns-01 branch, which then fails on the DNS
credential such a deployment deliberately does not have. The form now carries
the challenge forward, the add form offers it (so the UI can create these
domains at all), and the listing shows which challenge a domain uses instead of
naming a credential a dns-persist-01 domain never has.
The advisory wait could not stay advisory.
DNS_PERSIST_MAX_DNS_WAITwas300s, the same as the default
renew_timeoutthat wraps the whole order, andthe wait is measured only after the order and its authorizations are fetched --
so the outer timeout always fired first, aborting with "certificate request
timed out" instead of proceeding to the CA, and the warning an operator is told
to look for was never logged. Fixed for both challenges in the round below, by
deriving the wait from
renew_timeoutinstead of picking a constant.issuer_domain_namewas ignored on dns-01. The key documents itself asnaming the CA "in dns-persist-01 and CAA records", but dns-01 hardcoded
letsencrypt.org, so an operator running dns-01 against another CA withauto_set_caaon would have published CAA forbidding the very CA in use. Bothmethods now read the configured name, whose default is
letsencrypt.org, so anuntouched configuration writes exactly what it wrote before.
Bootstrapping without a domain
RotateAcmeCredentialsused to refuse to run until a ZT domain existed, whichput a fresh
dns-persist-01deployment in a loop: the record an operator has topublish names the ACME account, the gateway only registered one lazily on its
first issuance attempt, and that attempt could not succeed before the record was
published. The way out was to trigger a renewal, let it fail, and read the
records off the account the failure had registered.
Registration asserts nothing about a domain -- it is a POST to
newAccount,with no DNS access and no challenge involved -- so the requirement was an
artifact of
AcmeClient::new_accounttaking aValidationMethodin order toreturn a client.
AcmeClient::register_accountnow returns just the account(
AcmeAccount { credentials, account_uri }),new_accountis that plus aload, and rotation registers before it looks at any domain. A cluster with nodomain registers and stops there; one with domains re-pins their CAA exactly as
before, and the "first domain reuses the registration client" special case is
gone with it -- every domain now takes the same path.
The setup order in the doc is correspondingly straight:
SetCertbotConfig→RotateAcmeCredentials→AddZtDomain→ publish the records it reports →RenewZtDomainCert.Verified against Let's Encrypt staging through the CLI, which registers over the
same path: a fresh workdir registers
acct/329645064anddns-recordsimmediately renders the record naming it, while the existing account still
renews (
DNS:*.persist01.kvin.wang, DNS:persist01.kvin.wang), so theregister-then-loadsplit did not disturb the account it replaces. Thegateway's own no-domain path is covered by a unit test pointed at an unroutable
directory: it fails at the ACME server rather than on a missing domain, which is
what proves the requirement is gone.
Second review round
Two reviews came back on the fixes above; between them they found that one of
those fixes was half a fix, and that another had traded a dormant bug for a
sharper one. Both were right.
The DNS wait now derives from the order budget instead of being a constant.
120sfixed thedns-persist-01arm and leftdns-01— the arm nearly everydeployment uses — with the same 300s-inside-300s collision, and it was a
hardcoded assumption about
renew_timeout, which an operator can lower from thedashboard to 30s. It is now one rule for both:
advisory_dns_wait(configured, renew_timeout)caps the wait at half the budget, so the check always reachesits "proceed anyway" exit and the warning naming the missing record.
The CLI has the same shape and worse defaults —
renew_timeout120s wrapping a300s
max_dns_wait— which neither review looked at, so the clamp lives in thecertbotcrate and both callers use it. Against Let's Encrypt staging, sameconfig (
renew_timeout = 120,max_dns_wait = 300), same missing record:Before, the operator gets
requesting cert timeoutand no idea why. After, theyget the record that was missing and the CA's own verdict.
issuer_domain_nameis validated where it is set. Making it reachdns-01CAA turned an inert config key into a load-bearing one, and nothing checked it:
set_caa_recordsinstalls a;guard, deletes the existing issue/issuewildrecords, then writes the new content — so
"lets encrypt.org", or an explicit""on the CLI path (which passed the empty string straight through where thegateway mapped it to
letsencrypt.org), would leave the zone holding amalformed or empty-issuer
issueproperty and nothing valid behind it. That isCAA denying every issuer, published by certbot itself.
resolve_issuer_domain_namenow does both jobs for every caller: empty meansletsencrypt.org, and a value that is not a DNS name is refused —SetCertbotConfigrejects it, andcertbotrefuses to start with it.The corrupt-record repair path can no longer forget it.
merge_certbot_configrequired four fields to replace an unreadable record and then started from the
defaults, so a deployment on Pebble that repaired its config silently reverted to
letsencrypt.org— the exact hazardacme_urlis in that required set for.issuer_domain_namejoins it.renew_timeout_secs = 0is refused too, since theDNS wait is derived from it.
The dashboard no longer hides the setup.
showAddZtDomainModal()blankedevery field except the challenge select, so a second add silently inherited the
first one's choice; it resets now. And
AddZtDomainreturnsrequired_dns_recordswhich the page discarded — an operator pickingdns-persist-01got "ZT-Domain added" and no hint that nothing would be issueduntil they published a record. The records are now shown after the add, and
again behind a Records button on each
dns-persist-01row, since rotationinvalidates them.
Minor: the KV field's doc comment still said the setting was read only by
dns-persist-01, andset_certbot_config's audit log listed every field exceptthe one just changed. Both fixed. The setup-order doc also lost the
issuer-name guidance when it was rewritten in
4b47a448; it is back, with thenew rules.
cargo test -p certbot -p dstack-gateway: 43 + 296.Third review round
The round above introduced one regression and left three edges; all four are
closed here, verified against Pebble.
The records dialog fired for
dns-01domains, withdns-persist-01copy.Mine, from
f162c07d5b.required_dns_recordsis non-empty fordns-01too --required_dns_recordsskips the TXT record but still returns the two CAA lines-- so adding an ordinary
dns-01domain popped a dialog saying the gateway"holds no DNS credential for it and cannot write these itself", every clause of
which is false there, ending with an instruction to hand-publish records the
gateway writes itself. Gated on the challenge, like the
Recordsbutton in thesame commit already was.
The dashboard now has the button its own message points at. The empty-records
path tells an operator to register an ACME account first, and the documented
setup order makes that step 2 -- but
RotateAcmeCredentialshad no controlanywhere in the dashboard, so a dashboard-only operator hit a dead end. Added to
the Certbot Configuration section, behind a confirmation that says what rotation
costs, and it shows the records the response returns.
new_accountcould drop an account it had just registered. Splittingregister_accountout leftnew_accountas register-then-load, andloadfetches the ACME directory -- so a transient failure in that window returned
ErrafternewAccounthad succeeded, losing credentials neither caller hadpersisted yet and spending one of Let's Encrypt's 10 registrations per IP per 3
hours. The two halves are now taken from one call:
registerreturns the liveaccount and the credentials together,
new_accountbuilds the client from whatit already holds, and
register_accountencodes the same credentials forcallers that want only those. No second request, no window.
The DNS wait overshot the budget it had just been clamped to.
check_dnsslept then checked its deadline, so it overran by up to a full backoff step
(32s). The clamp reserves half of
renew_timeout, which the overshoot could eatwhole: at
renew_timeout = 64the wait is 32s, but the sleep checkpoints landat 31.75s, so the loop slept again and reached its exit at ~63.8s -- past the
order's budget. On Pebble, same config, same missing record:
The deadline is now checked before sleeping and the sleep is cut to what is left
of it, so the exit lands on the cap rather than past it and the order still has
its budget. The clamp also logs when it lowers a configured value, which it
previously did silently -- an operator raising
max_dns_waitand seeing nochange had nothing to read.
The issuer name validator was looser than the grammar it protects. It
accepted
_letsencrypt.org,-letsencrypt.org,letsencrypt-.organd labels ofany length, none of which match RFC 8659's
label = (ALPHA/DIGIT) *( *("-") (ALPHA/DIGIT) )-- so a strict CA reads theissueproperty as malformed, whichis the state the validator exists to keep out of a zone, reached through a value
it waved through. Now held to that grammar, with the 63-octet DNS label bound.
Full
dns-persist-01flow re-verified on Pebble after the registration change:account registered, records printed, published with padded separators (the
parser fix's case), certificate issued for
DNS:p3.e2e.test, DNS:*.p3.e2e.test.cargo test -p certbot -p dstack-gateway: 44 + 296.Fourth review round
Every finding this round was in the dashboard button the previous round added,
which is a fair verdict on where new surface area costs the most.
A partial rotation is now reported, not raised.
do_rotate_acme_credentialsbailed when a domain's CAA could not be re-pinned — after registering the
account and publishing the credentials. The dashboard rendered that as "Failed
to register ACME account", so the obvious next move is clicking again, which
registers another account against a rate-limited quota and rewrites CAA a second
time: exactly what the bail's own text warned against. It also meant
required_dns_recordsnever reached the caller on the one path where therecords matter most.
RotateAcmeCredentialsResponsegainsrepin_failed_domains; the rotation returns success, names the domains, and thelog keeps the full "rerun SetCaa, do not rotate again" line at
error!.The Account URI no longer goes stale. It is server-rendered, and rotation
only refreshed the ZT-domain table — so the one authoritative place an operator
reads the account kept showing the replaced one for the rest of the session,
while the dialog beside it showed records naming the new one. The cell is
updated from the response.
The confirmation says what the button actually does. It warned about
republishing
dns-persist-01records — true — and said nothing about the twothings that can hurt: it registers a rate-limited account, and it rewrites the
CAA of every
dns-01domain by installing a;guard and deleting the existingrecords first, so a provider failure part-way through can leave a domain with
CAA that blocks all issuance until a later
SetCaasucceeds.Polish. The new DNS-budget line fired on every issuance in every deployment
— the stock defaults have both values at 300s, so
configured > cappedisalways true — and was worded as if an operator setting had been overridden; it
is now
debug!and phrased as the budget it is. And the issuer-name validator'sfive rejection reasons had been merged into one message that named no character:
my_ca.example.comwas refused without mentioning the underscore, which is thecharacter this round newly started rejecting and the one most likely to be
reached for. Each reason says what it rejected again, with a test on the
underscore case.
cargo test -p certbot -p dstack-gateway: 45 + 296.Maturity
Deliberately opt-in and documented as experimental. draft-01 is the latest
revision, and Let's Encrypt has stated it will not deploy to production
until an open working-group issue about client-computed record
content is resolved — that change would alter the record format. Nothing here
runs unless an operator sets
challenge, and the default path is untouched.Documented in
docs/certbot-dns-persist-01.md.