Skip to content

deps: switch reqwest from native-tls to rustls with aws-lc-rs - #39415

Merged
jasonhernandez merged 1 commit into
MaterializeInc:mainfrom
jasonhernandez:jason/reqwest-rustls
Oct 1, 2026
Merged

jasonhernandez merged 1 commit into
MaterializeInc:mainfrom
jasonhernandez:jason/reqwest-rustls

Conversation

@jasonhernandez

@jasonhernandez jasonhernandez commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Move reqwest 0.13 off OpenSSL (native-tls) onto rustls with the aws-lc-rs provider, as part of removing OpenSSL and ring from the tree. Prior art: #35947.

Description

  • reqwest uses its rustls feature: aws-lc-rs plus rustls-platform-verifier, which trusts the system store on Linux and macOS.
  • sentry switches from transport to reqwest + rustls. transport enables reqwest's native-tls, and reqwest defaults to native-tls whenever it is compiled in, so every client would otherwise stay on OpenSSL.
  • mz-ccsr's Identity holds the PEM key and certificates and passes them to rustls. Schema registry connections already take PEM (SSL KEY, SSL CERTIFICATE). The PKCS#12 archive Identity used to build from them was only an internal step for native-tls, so the SQL surface is unchanged. Identity checks that the key matches the certificate, is zeroized on drop, and redacts Debug. It is built from secrets at connect time and never persisted.
  • testdrive's --cert still takes a PKCS#12 keystore and converts it to PEM itself. No CI job passes --cert.
  • New mz_ccsr::tls::TlsError replaces the native-tls and OpenSSL variants of CsrConnectError. mz-ccsr and mz-storage-types drop native-tls and openssl.
  • Still on OpenSSL after this PR: reqwest 0.12 (forced by the iceberg fork's defaults and duckdb), and the MySQL source's PEM to PKCS#12 conversion for mysql_async.

Behavior changes

  • On Linux, building a client fails fast if no system CA bundle is present.
  • The system trust store is read each time a client is built, which for ccsr means once per schema registry connection.
  • Schema registry TLS errors use rustls text, for example invalid peer certificate: UnknownIssuer. The kafka-auth .td files are updated.
  • A key that does not match its certificate is rejected with keys may not be consistent: KeyMismatch instead of an OpenSSL PKCS#12 error.
  • webpki is stricter than OpenSSL about server certificates. A self-signed CA:TRUE certificate given as its own SSL CERTIFICATE AUTHORITY fails with CaUsedAsEndEntity. Follow-up ccsr: trust a server certificate that exactly matches the configured CA #39459 accepts it by exact match. A certificate with only a CN and no subjectAltName now fails with a name error and must be reissued with a SAN. The same stricter validation applies to COPY FROM URLs and the OIDC issuer fetch, with no exact-match fallback.

Release notes

Schema registry connections and COPY FROM URLs now reject server certificates that identify the host only by common name, without a subjectAltName. If a server uses such a certificate, reissue it with a subjectAltName that matches its hostname before upgrading.

Verification

Locally: bin/fmt, workspace cargo check, clippy on the touched crates, cargo-deny, lint-cargo, and ccsr and testdrive unit tests, including a new test_pem_identity.

CI: PR CI covers kafka-auth (schema registry ssl, ssl-basic, mssl, mssl-basic), testdrive, and the environmentd auth/server and balancerd server tests. Nightly (ci-nightly) covers testdrive-old-kafka-src-syntax, cloudtest and orchestratord.

🤖 Generated with Claude Code

Build reqwest 0.13 with its `rustls` feature (aws-lc-rs provider, platform
verifier) instead of `native-tls-vendored`, and switch sentry from
`transport` to `reqwest` + `rustls`. Sentry's `transport` feature enables
`reqwest/native-tls-no-alpn`, and reqwest defaults to native-tls whenever it
is compiled in, so leaving it would keep every reqwest 0.13 client on
OpenSSL.

The schema registry client (mz-ccsr) used native-tls-only APIs:

* `Identity` now holds a PEM key and certificate chain instead of a
  PKCS #12 archive. `Identity::from_pem` checks that the key matches the
  leaf certificate up front, and the buffer is zeroized on drop. `Debug`
  no longer prints the key.
* `Identity::from_pkcs12_der` is removed. testdrive, its only caller,
  converts its PKCS #12 keystore to PEM with OpenSSL.
* `Certificate` parsing uses rustls instead of native-tls.

mz-ccsr no longer depends on native-tls, openssl or mz-tls-util, and
mz-storage-types drops its native-tls and openssl dependencies. The
kafka-auth schema registry tests expect the rustls error strings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jasonhernandez
jasonhernandez marked this pull request as ready for review September 30, 2026 22:07
@jasonhernandez
jasonhernandez requested review from a team as code owners September 30, 2026 22:07
@jasonhernandez jasonhernandez added the ci-nightly PR CI control: also trigger Nightly label Sep 30, 2026
@def-

def- commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

QA LLM Review

1. MEDIUM -- Existing schema registry connections break on upgrade when the registry's certificate is one OpenSSL accepts but webpki rejects

Cargo.toml:470

Schema registry TLS moves from OpenSSL to rustls/webpki, which is stricter about server certificates. A registry whose certificate has no subjectAltName (CN only), or which uses a self-signed CA:TRUE certificate supplied as SSL CERTIFICATE AUTHORITY, works today and fails after upgrade. Nothing flags this at upgrade time. Existing Avro sources halt and Kafka sinks fail to start.

Details

I checked this against reqwest 0.13.5, rustls 0.23.45 and rustls-webpki 0.103.15, the versions this PR locks. I used add_root_certificate + resolve, the same setup as ccsr::ClientConfig::build, against openssl s_server, and switched between tls_backend_native() and tls_backend_rustls():

  • Leaf signed by a private CA, CN=sr.example.com, no SAN. Native: 200 OK. Rustls: invalid peer certificate: certificate not valid for name "sr.example.com"; certificate is not valid for any names (according to its subjectAltName extension). OpenSSL falls back to the CN when there are no DNS SANs, and webpki does not. keytool -genkeypair -dname CN=... without -ext SAN=... produces this kind of certificate, which is common for self-hosted Schema Registry keystores.
  • Self-signed cert from a plain openssl req -x509 (adds CA:TRUE), given as its own CA. Native: 200 OK. Rustls: invalid peer certificate: Other(OtherError(CaUsedAsEndEntity)).

The kafka-auth fixtures (test/test-certs/create-certs.sh) always issue SAN-bearing leaves from a separate CA, so CI does not exercise either case. At runtime CsrConnection::connect is called from the source decoder (src/storage/src/decode.rs:364) and the Kafka sink (src/storage/src/sink/kafka.rs:1423). After a restart the writer-schema cache is empty, so the first fetch fails and the source goes into a halting loop. Librdkafka still uses OpenSSL, so a Kafka connection that uses the same PKI keeps working, which makes the failure confusing. The same stricter validation now also applies to COPY FROM URLs and to the OIDC issuer fetch (src/authenticator/src/oidc.rs:308).

This change is not listed under "Behavior changes". At minimum it needs a release note. A pre-upgrade check or a per-connection fallback would avoid breaking these connections silently.

@antiguru antiguru left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thank you!

Comment thread src/ccsr/src/tls.rs
@jasonhernandez
jasonhernandez merged commit 1fdfee4 into MaterializeInc:main Oct 1, 2026
342 of 347 checks passed
@jasonhernandez
jasonhernandez deleted the jason/reqwest-rustls branch October 1, 2026 16:00
@jasonhernandez
jasonhernandez restored the jason/reqwest-rustls branch October 1, 2026 16:00
@jasonhernandez

Copy link
Copy Markdown
Contributor Author

Addressed: self-signed CA:TRUE certificates given as their own CA are accepted by exact match in #39459. CN-only certificates stay rejected and are now in Behavior changes and Release notes, along with COPY FROM and OIDC.

jasonhernandez added a commit that referenced this pull request Oct 2, 2026
…CA (#39459)

### Motivation

This addresses the QA review finding on #39415. Under rustls, a
self-signed `CA:TRUE` schema registry certificate supplied as its own
`SSL CERTIFICATE AUTHORITY` fails with `CaUsedAsEndEntity`, though
OpenSSL accepted it. Follows #39415.

### Description

- When a ccsr client has configured root certificates, mz-ccsr builds
the rustls config (aws-lc-rs) and hands it to reqwest with
`tls_backend_preconfigured`.
- A server certificate byte-for-byte identical to a configured root
skips the chain, basic-constraints and key-usage checks, but its
validity period and server name are still checked. The name is checked
against SANs, falling back to any subject CN only when the pinned cert
has no DNS or IP SAN. This is stricter than OpenSSL, which also falls
back when only IP SANs are present, matches wildcard CNs, and decodes
other CN string types. Validity is read with `x509-cert` because
rustls-webpki exposes no validity accessor.
- Every other certificate goes to `rustls-platform-verifier`, with the
same native and extra roots reqwest uses. Handshake signatures are
always verified.
- The client identity moves into the rustls config.
- Without configured roots, reqwest's default path is unchanged.
Non-pinned CN-only leaves without a SAN are still rejected.

### Verification

New `test_tls_server_verification` runs against a local TLS server. It
accepts a SAN leaf and exact-match self-signed CAs matched by SAN or by
any of several CNs, and rejects an exact match with the wrong SAN or CN,
or with only an IP SAN, an expired or not-yet-valid exact match, a
different self-signed cert, and a CN-only leaf. New
`test_tls_client_identity_with_roots` covers mTLS with configured roots.
Each check was confirmed to make its test fail when removed. A local
differential run against `openssl verify -verify_hostname` (3.6.4 and
4.0.2) over 18 certificate shapes found no case we accept that OpenSSL
rejects.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Jason Hernandez <7144515+jasonhernandez@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-nightly PR CI control: also trigger Nightly

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants