v1.5.4: finish the User-Agent override, add per-operator HTTP/1.1 transport - #13
Merged
Merged
Conversation
Some CT log operators (e.g. Geomys) apply a more generous rate limit tier to clients that include a contact email in their User-Agent.
- a blank CERTSTREAM_USER_AGENT falls back to the default instead of
sending an empty header; `CERTSTREAM_USER_AGENT=` in a compose file or
.env reads back as Ok(""), which is the opposite of what an operator
setting a contact address is asking for
- the override now also reaches the TLS-pinned Apple catalog client, so
every outbound request carries the same identity; the default string
lives in one const instead of two copies
- ct_log.force_http1_operators gives the listed operators a dedicated
HTTP/1.1 client. DigiCert throttles per TCP connection and serves
several logs from one host, so under HTTP/2 all of their watchers share
one connection's quota. The per-operator token bucket still gates every
fetch, so this spreads the same request rate across more connections
rather than raising it (see SSLMate/certspotter#126)
- operator_rate_limits and force_http1_operators are both looked up by
canonicalized operator name; keys that match no discovered log were
silently inert and are now named at startup
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on [#12](#12) by [@ineffyble](https://github.com/ineffyble). Their commit is included here unchanged, so merging this PR will also close #12.
User-Agent
[@ineffyble](https://github.com/ineffyble)'s
CERTSTREAM_USER_AGENT/ct_log.user_agentchange is included as-is, with a few follow-ups on top.A blank value could previously make it all the way onto the wire. For example,
CERTSTREAM_USER_AGENT=in a compose file or.envis read asOk(""), andHeaderValue::try_from("")accepts it. That meant validation passed, and reqwest ended up sending an emptyUser-Agent:header not very useful for someone trying to provide a contact address.Blank or whitespace-only values now fall back to the default User-Agent, with a warning logged at startup.
The override is also now used by the TLS-pinned Apple catalog client, which previously had its own hardcoded UA. This means all outbound requests use the same identity. I also moved the default User-Agent string into a single
constinstead of keeping two copies.The YAML example in
docs/docs.htmlhad1.5.3hardcoded, so I made it versionless to avoid it going stale on the next release.Per-operator HTTP/1.1
DigiCert appears to throttle per TCP connection rather than per IP.
Both
wyvern.ct.digicert.comandsphinx.ct.digicert.comnegotiate h2 (verified withopenssl s_client -alpn h2,http/1.1). Wyvern alone serves2026h1,2026h2, and2027h1, which means that under HTTP/2 reqwest can multiplex all three watchers over a single connection, causing them to share that connection's quota.Operators listed here get a dedicated
http1_onlyclient. Everyone else continues using the shared HTTP/2 client.With HTTP/1.1, each in-flight fetch uses its own connection, so the requests are no longer multiplexed onto a single connection.
pool_max_idle_per_host(fetch_concurrency) only controls how many connections stay warm between polls; it does not cap the number of concurrent connections. The docs now make that distinction instead of suggesting an "N× rate limit" multiplier.This does not increase the outbound request rate. The per-operator token bucket still runs before every fetch in
static_ct.rsandwatcher.rs, so with the default 500 ms interval the operator still sees 2 requests/sec. The only difference is how those requests are distributed across connections.Environment variable equivalent:
This is the same general observation behind certspotter's
digicerthackbranch ([SSLMate/certspotter#126](SSLMate/certspotter#126)), although their approach disables keep-alives entirely.Keep-alive remains enabled here. The setting is called
force_http1_operatorsbecause forcing HTTP/1.1 is exactly what it does.Unmatched operator names
operator_rate_limitskeys were not previously checked against the operator names discovered from the CT log catalog.That meant a config value like
digicertcould silently do nothing if the catalog reported the operator asDigiCert, Inc.. From the user's point of view, an unmatched setting looks exactly like a working one until you measure the behavior.Startup now reports unmatched names for both
ct_log.operator_rate_limitsand the newforce_http1_operatorslist:The matching behavior itself has not changed. This is warning-only.
Verification
cargo clippy --all-targets -- -D warningsis clean.All 258 lib and 262 bin unit tests pass, with the bin count up from 251 on
main.The 11 new tests cover:
DigiCertandDigiCert, Inc.Integration and snapshot suites are unchanged.
--validate-configagainstconfig.example.yamlalso parses both new fields successfully.One thing I have not measured is the actual throughput improvement against DigiCert. The h2 negotiation and connection sharing are verified, but the resulting rate-limit headroom is not. The docs and release notes intentionally avoid claiming a specific multiplier.