Add CERTSTREAM_USER_AGENT env var to override HTTP User-Agent

[ineffyble]

Some CT log operators (e.g. Geomys) apply a more generous rate limit tier to clients that include a contact email in their User-Agent. This adds a config option to override the default User-Agent header, allowing operators to include their email address or other relevant information.

Written by GLM 5.2, reviewed and endorsed by Effy Elden (ineffyble).

[reloading01]

Thanks for this good catch on the rate limit tiers. I hadn’t run into that before. I pulled your commit into #13 as-is, so merging that should close this issue too.

I added three small follow-ups on top of it:

First, an empty CERTSTREAM_USER_AGENT value could make it all the way into the request. For example, CERTSTREAM_USER_AGENT= in a compose file or .env comes through as Ok(""), and HeaderValue::try_from("") considers that valid. So reqwest would end up sending an empty User-Agent header, which isn’t really what someone trying to set a contact address would expect. Empty or whitespace-only values now fall back to the default, with a startup log explaining that.

I also wired the override into build_apple_pinned_client. That still had its own hardcoded User-Agent, so now catalog fetches use the same identity as the log fetches. While I was there, I moved the default User-Agent string into a single const.

The YAML example in docs/docs.html also had 1.5.3 hardcoded. I made that versionless so it doesn’t go stale on the next release.

On the Geomys/rate limit point, #13 also adds ct_log.force_http1_operators. DigiCert appears to throttle per TCP connection rather than per IP, and wyvern.ct.digicert.com negotiates HTTP/2 while serving 2026h1, 2026h2, and 2027h1. That means all three watchers can end up multiplexed over the same connection and sharing the same quota.

For operators listed in that setting, the client switches to http1_only, so concurrent fetches use separate connections. The per-operator token bucket still applies to every request, so this doesn’t increase the overall request rate it just distributes the same rate across connections. That matches the same behavior noted in certspotter’s digicerthack branch.

Both changes are going out in v1.5.4.

[ineffyble]

I actually had a patch for the digicerthack locally too, was testing it more before submitting 😅

Thank you!