Skip to content

v1.5.4: finish the User-Agent override, add per-operator HTTP/1.1 transport - #13

Merged
reloading01 merged 3 commits into
mainfrom
feat/ua-override-and-http1-transport
Aug 22, 2026
Merged

reloading01 merged 3 commits into
mainfrom
feat/ua-override-and-http1-transport

Conversation

@reloading01

@reloading01 reloading01 commented Aug 22, 2026

Copy link
Copy Markdown
Owner

Builds on [#12](#12) by [@ineffyble](https://github.com/ineffyble). Their commit is included here unchanged, so merging this PR will also close #12.

User-Agent

[@ineffyble](https://github.com/ineffyble)'s CERTSTREAM_USER_AGENT / ct_log.user_agent change is included as-is, with a few follow-ups on top.

A blank value could previously make it all the way onto the wire. For example, CERTSTREAM_USER_AGENT= in a compose file or .env is read as Ok(""), and HeaderValue::try_from("") accepts it. That meant validation passed, and reqwest ended up sending an empty User-Agent: header not very useful for someone trying to provide a contact address.

Blank or whitespace-only values now fall back to the default User-Agent, with a warning logged at startup.

The override is also now used by the TLS-pinned Apple catalog client, which previously had its own hardcoded UA. This means all outbound requests use the same identity. I also moved the default User-Agent string into a single const instead of keeping two copies.

The YAML example in docs/docs.html had 1.5.3 hardcoded, so I made it versionless to avoid it going stale on the next release.

Per-operator HTTP/1.1

DigiCert appears to throttle per TCP connection rather than per IP.

Both wyvern.ct.digicert.com and sphinx.ct.digicert.com negotiate h2 (verified with openssl s_client -alpn h2,http/1.1). Wyvern alone serves 2026h1, 2026h2, and 2027h1, which means that under HTTP/2 reqwest can multiplex all three watchers over a single connection, causing them to share that connection's quota.

ct_log:
  force_http1_operators:
    - DigiCert

Operators listed here get a dedicated http1_only client. Everyone else continues using the shared HTTP/2 client.

With HTTP/1.1, each in-flight fetch uses its own connection, so the requests are no longer multiplexed onto a single connection. pool_max_idle_per_host (fetch_concurrency) only controls how many connections stay warm between polls; it does not cap the number of concurrent connections. The docs now make that distinction instead of suggesting an "N× rate limit" multiplier.

This does not increase the outbound request rate. The per-operator token bucket still runs before every fetch in static_ct.rs and watcher.rs, so with the default 500 ms interval the operator still sees 2 requests/sec. The only difference is how those requests are distributed across connections.

Environment variable equivalent:

CERTSTREAM_CT_LOG_FORCE_HTTP1_OPERATORS=DigiCert,Geomys

This is the same general observation behind certspotter's digicerthack branch ([SSLMate/certspotter#126](SSLMate/certspotter#126)), although their approach disables keep-alives entirely.

Keep-alive remains enabled here. The setting is called force_http1_operators because forcing HTTP/1.1 is exactly what it does.

Unmatched operator names

operator_rate_limits keys were not previously checked against the operator names discovered from the CT log catalog.

That meant a config value like digicert could silently do nothing if the catalog reported the operator as DigiCert, Inc.. From the user's point of view, an unmatched setting looks exactly like a working one until you measure the behavior.

Startup now reports unmatched names for both ct_log.operator_rate_limits and the new force_http1_operators list:

WARN configured operator names match no discovered CT log operator and have no effect field="ct_log.operator_rate_limits" unmatched=["digicert inc"]

The matching behavior itself has not changed. This is warning-only.

Verification

cargo clippy --all-targets -- -D warnings is clean.

All 258 lib and 262 bin unit tests pass, with the bin count up from 251 on main.

The 11 new tests cover:

  • blank and whitespace-only User-Agent fallback
  • trimming
  • validation accepting blank values so they can fall back
  • comma-separated list parsing
  • YAML deserialization
  • canonical operator-name matching
  • the intentional distinction between DigiCert and DigiCert, Inc.
  • client selection for operators not in the HTTP/1.1 list
  • unmatched operator detection

Integration and snapshot suites are unchanged.

--validate-config against config.example.yaml also parses both new fields successfully.

One thing I have not measured is the actual throughput improvement against DigiCert. The h2 negotiation and connection sharing are verified, but the resulting rate-limit headroom is not. The docs and release notes intentionally avoid claiming a specific multiplier.

ineffyble and others added 3 commits August 23, 2026 00:23
Some CT log operators (e.g. Geomys) apply a more generous rate limit
tier to clients that include a contact email in their User-Agent.
- a blank CERTSTREAM_USER_AGENT falls back to the default instead of
  sending an empty header; `CERTSTREAM_USER_AGENT=` in a compose file or
  .env reads back as Ok(""), which is the opposite of what an operator
  setting a contact address is asking for
- the override now also reaches the TLS-pinned Apple catalog client, so
  every outbound request carries the same identity; the default string
  lives in one const instead of two copies
- ct_log.force_http1_operators gives the listed operators a dedicated
  HTTP/1.1 client. DigiCert throttles per TCP connection and serves
  several logs from one host, so under HTTP/2 all of their watchers share
  one connection's quota. The per-operator token bucket still gates every
  fetch, so this spreads the same request rate across more connections
  rather than raising it (see SSLMate/certspotter#126)
- operator_rate_limits and force_http1_operators are both looked up by
  canonicalized operator name; keys that match no discovered log were
  silently inert and are now named at startup
@reloading01
reloading01 merged commit 0f4d0b3 into main Aug 22, 2026
1 check passed
@reloading01
reloading01 deleted the feat/ua-override-and-http1-transport branch August 27, 2026 08:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants