Skip to content

fix(ENGHLP-1703): keep TLS when DIODE_SKIP_TLS_VERIFY is set - #114

Open
jajeffries wants to merge 7 commits into
developfrom
fix/enghlp-1703-skip-tls-verify-parity
Open

jajeffries wants to merge 7 commits into
developfrom
fix/enghlp-1703-skip-tls-verify-parity

Conversation

@jajeffries

@jajeffries jajeffries commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Align the Python Diode SDK with the Go SDK for DIODE_SKIP_TLS_VERIFY: the connection stays on grpcs/https and only certificate validation is skipped.
  • Fixes deployments behind Traefik or other proxies that present self-signed certificates, where the previous behavior could downgrade to plaintext (h2c) and surface as 404 or protocol errors.
  • Preserves encryption on the wire while allowing lab and staging setups that cannot use publicly trusted certs.

What changed

  • parse_target now returns separate is_plaintext and tls_verify flags instead of treating skip-verify as plaintext.
  • _open_grpc_channel uses TLS credentials with an optional peer certificate pin when verification is disabled; auth token URL scheme follows is_plaintext.
  • Added skip_tls_verify constructor parameter and documented DIODE_SKIP_TLS_VERIFY in the README.

How tested

  • pytest tests/test_client.py -q (133 passed)
  • ruff check netboxlabs/diode/sdk/client.py tests/test_client.py

Linear

Made with Cursor

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

github-actions Bot commented Sep 18, 2026

Copy link
Copy Markdown

Coverage

Coverage Report
FileStmtsMissCoverMissing
/opt/hostedtoolcache/Python/3.10.21/x64/lib/python3.10/site-packages/netboxlabs/diode/sdk
   chunking.py33197%112
   client.py6658687%146, 156–180, 182, 192–220, 234–236, 242, 277, 368–369, 388–390, 393–396, 670, 738, 743, 747, 836–837, 865–867, 897, 906, 910, 932, 953, 955, 1029, 1060, 1068, 1130–1131, 1140, 1173, 1200, 1205–1206, 1235, 1240–1243
   exceptions.py44393%69, 82–83
TOTAL7579088% 

Tests Skipped Failures Errors Time
225 0 💤 0 ❌ 0 🔥 2.090s ⏱️

@nbl-ai-review

nbl-ai-review Bot commented Sep 18, 2026

Copy link
Copy Markdown

AI Code Review — risk tier: Full · dimensions run: correctness, security

Advisory only. A human owns the merge decision.

Summary — Net positive. This head reverts the nested-TLS wrap on the proxy CONNECT socket, fixing the high-severity bug that broke the skip-verify probe entirely when combined with an https:// proxy. No new correctness issues in the delta.

Resolved since last review

  • netboxlabs/diode/sdk/client.py — Nested SSLContext.wrap_socket() calls on the proxy CONNECT socket, which broke the tunnel for DIODE_SKIP_TLS_VERIFY + https:// proxy, are fixed — the proxy leg is now plain TCP (mirroring grpc.http_proxy), with a single TLS handshake to the target on the tunneled socket.

Previously raised — waived, not re-raised

  • netboxlabs/diode/sdk/client.py:156 — CONNECT tunnel to an https:// proxy is sent in the clear again. This is a known, explicit trade-off of the fix above: wrapping the proxy socket in TLS a second time is what caused the nested-TLS regression, and the author has documented in the PR conversation that they are intentionally not adding a separate TLS session to the proxy leg (matching how grpc.http_proxy handles HTTPS_PROXY for the real channel). No credentials are sent over this leg — only the CONNECT target host:port is exposed to an on-path observer between client and proxy. Not re-raised per the recorded waiver; flag again only if a future change increases its severity (e.g. if credentials are ever added to the CONNECT request).

🤖 AI Code Review · run · prompts: code-review/v1 · models: claude-sonnet-5/medium · context: description · 6 comment(s) · 1 prior AI review(s) · 2/2 thread(s) resolved · usage: $1.41 · 2559k in (95% cached) · 12.5k out

Previous review · 2026-09-18 · 46c67dd

AI Code Review — risk tier: Full · dimensions run: correctness, security

Advisory only. A human owns the merge decision.

Summary — Net positive overall: this head fixes the two low-severity items from the last review (bracketed IPv6-literal probe targets, plaintext CONNECT to an https:// proxy). However, the https-proxy fix itself introduces a new bug — nesting a second TLS handshake on top of the already-TLS proxy socket — which breaks the skip-verify probe whenever DIODE_SKIP_TLS_VERIFY is combined with an https:// proxy, i.e. exactly the case the fix was meant to support.

Findings

  • netboxlabs/diode/sdk/client.py:167 and :205 — [High] [correctness] Nested ssl.SSLContext.wrap_socket() calls break the proxy tunnel (TLS-in-TLS not supported this way) — _connect_socket wraps the proxy connection in TLS when proxy_url is https:// (line 167) and returns that SSLSocket after completing the CONNECT handshake over it. _fetch_peer_leaf_certificate then calls context.wrap_socket(raw_sock, ...) on that same object (line 205) to do a second, inner TLS handshake with the real target. SSLContext.wrap_socket() takes sock.fileno() and detaches the outer socket, discarding its live TLS session and writing the inner ClientHello as raw bytes directly on the underlying fd — the proxy's outer TLS session can't parse that as valid application data and will typically reset or alert. All three probe attempts fail the same way, so _skip_verify_channel_credentials exhausts retries and DiodeClient.__init__/DiodeOTLPClient.__init__ always raises DiodeConfigError for DIODE_SKIP_TLS_VERIFY=true + an https:// proxy. Suggested fix: don't nest TLS wraps — either implement true TLS-in-TLS via a BIO-based shim (e.g. ssl.MemoryBIO) that keeps the outer SSLSocket's send()/recv() in the loop for the inner handshake bytes, or drop the inner TLS-wrap of the proxy socket in _connect_socket (matching the real gRPC channel's grpc.http_proxy, which does a plain CONNECT regardless of the proxy URL scheme) and rely on the second handshake only for the tunnel to the ultimate target.

Resolved since last review

  • Skip-verify probe cannot connect to bracketed IPv6-literal targets — _authority_host_port now strips [/] from the host before connecting.
  • CONNECT tunnel to an https:// proxy sent over a plaintext socket — the proxy socket is now wrapped in TLS before CONNECT (though this introduces the new nested-TLS bug above).

🤖 AI Code Review · run · prompts: code-review/v1 · models: claude-sonnet-5/medium · context: description · 5 comment(s) · 1 prior AI review(s) · 1/1 thread(s) resolved · usage: $1.61 · 2494k in (94% cached) · 26.5k out

Previous review · 2026-09-18 · 0c40e2e

AI Code Review — risk tier: Full · dimensions run: correctness, security

Advisory only. A human owns the merge decision.

Summary — Net positive: this follow-up commit wraps the remaining unwrapped ssl.SSLError/OSError from the TLS-probe handshake in DiodeConfigError, closing the last open correctness finding from the prior review. No new bugs of note in this delta; two low-severity, narrow-scope items remain for awareness.

Findings

  • netboxlabs/diode/sdk/client.py:142 — [Low] [correctness] Skip-verify probe cannot connect to bracketed IPv6-literal targets — _connect_socket splits authority via rsplit(":", 1), leaving brackets on an IPv6-literal host (e.g. [::1]), which socket.create_connection cannot resolve, causing grpcs://[::1]:port with skip-verify to always fail. Suggestion: strip surrounding [/] from the host before connecting.
  • netboxlabs/diode/sdk/client.py:152 — [Low] [security] CONNECT tunnel to an https:// proxy is sent over a plaintext socket — _connect_socket never wraps the socket in TLS before sending the HTTP CONNECT request, even when the configured proxy URL uses https://, exposing the CONNECT request in the clear and deviating from what the scheme implies. Suggestion: wrap sock in an SSL context (verifying the proxy's cert normally) when parsed_proxy.scheme == "https", or explicitly document/reject https:// proxy URLs.

Resolved since last review

  • Unwrapped low-level network/TLS exceptions from the cert-fetch path (netboxlabs/diode/sdk/client.py) — now caught and wrapped in DiodeConfigError.

🤖 AI Code Review · run · prompts: code-review/v1 · models: claude-sonnet-5/medium · context: description · 4 comment(s) · 1 prior AI review(s) · 1/1 thread(s) resolved · usage: $1.99 · 3293k in (94% cached) · 27k out

Previous review · 2026-09-18 · ec6399f

AI Code Review — risk tier: Full · dimensions run: correctness, security

Advisory only. A human owns the merge decision.

Summary — Net positive: the skip-verify probe now pins TLS 1.2 minimum (fixing the CodeQL flag) and keeps the gRPC channel encrypted instead of falling back to plaintext. One of the three previously-raised correctness issues is only partially addressed — the TLS handshake step in the probe can still leak a raw, unwrapped exception on a transient network failure.

Previously raised — still open

  • netboxlabs/diode/sdk/client.py:193 — [Medium] correctness: Unwrapped low-level network/TLS exceptions from the cert-fetch path. Partially fixed — _connect_socket now wraps OSError and the proxy CONNECT path wraps failures in DiodeConfigError, but _fetch_peer_leaf_certificate's TLS handshake step (context.wrap_socket(...) / tls_sock.getpeercert(...)) still only catches ssl.SSLError. A TimeoutError/ConnectionResetError/BrokenPipeError during the 2nd or 3rd probe attempt — a realistic transient blip against the load-balanced/self-signed backends this PR targets — propagates as a raw OSError out of DiodeClient.__init__/DiodeOTLPClient.__init__ instead of a clean DiodeConfigError. Suggest widening the except clause to except (ssl.SSLError, OSError) as exc: raise DiodeConfigError(...) from exc, matching _connect_socket's handling.

Resolved since last review

  • Peer-cert probe vs. real gRPC channel cert mismatch (client.py, previously flagged around line 173) — now runs up to three probe handshakes and pins a combined bundle of distinct leaf certs, with SNI fallback to the authority host.
  • Reliance on the private CPython API ssl._ssl._test_decode_cert (previously flagged around line 124) — removed; server name is now taken from SSLSocket.getpeercert() during the probe handshake.
  • CodeQL: insecure TLS 1.0/1.1 permitted on the probe context — fixed via an inline SSLContext(PROTOCOL_TLS_CLIENT) with minimum_version = TLSv1_2.

🤖 AI Code Review · run · prompts: code-review/v1 · models: claude-sonnet-5/medium · context: description · 3 comment(s) · 1 prior AI review(s) · 1/1 thread(s) resolved · usage: $1.11 · 1047k in (83% cached) · 12.2k out

Comment thread netboxlabs/diode/sdk/client.py Fixed
Probe up to three times and pin all distinct peer leaf certs for gRPC
skip-verify, wrap connect/TLS failures in DiodeConfigError, derive SNI
override from getpeercert(), and require TLS 1.2+ on the probe handshake.

Co-authored-by: Cursor <cursoragent@cursor.com>
@jajeffries

Copy link
Copy Markdown
Contributor Author

AI review follow-up (pushed d621ee3)

  • Peer-cert probe vs gRPC channel: _skip_verify_channel_credentials now runs up to three probe handshakes and passes a combined root_certificates bundle (all distinct leaf PEMs) to gRPC, so a load-balanced peer that rotates certs between connections is more likely to match the real channel. When probe runs return different SNI names, grpc.ssl_target_name_override falls back to the authority host.
  • Unwrapped network/TLS errors: _connect_socket and _fetch_peer_leaf_certificate wrap OSError / ssl.SSLError in DiodeConfigError with context (proxy CONNECT failures unchanged).
  • Private CPython cert decoder: removed ssl._ssl._test_decode_cert; server name comes from SSLSocket.getpeercert() during the probe handshake.
  • CodeQL (TLS 1.0/1.1 on probe): probe uses SSLContext(PROTOCOL_TLS_CLIENT) with minimum_version = TLSv1_2 instead of create_default_context().

Verification: pytest tests/test_client.py -q (136 passed), ruff check netboxlabs/diode/sdk/client.py tests/test_client.py.

Inline PROTOCOL_TLS_CLIENT setup at wrap_socket so CodeQL sees TLS 1.2+
on the peer-cert probe handshake (addresses PR review discussion r4046196340).

Co-authored-by: Cursor <cursoragent@cursor.com>
@jajeffries

Copy link
Copy Markdown
Contributor Author

/ai-review

Widen _fetch_peer_leaf_certificate handshake handling to catch OSError
alongside ssl.SSLError so transient network failures during skip-verify
probes surface as DiodeConfigError instead of raw exceptions.

Co-authored-by: Cursor <cursoragent@cursor.com>
@jajeffries

Copy link
Copy Markdown
Contributor Author

AI review follow-up (open finding: unwrapped TLS probe errors)

Pushed fix for the remaining Medium item on _fetch_peer_leaf_certificate: the TLS handshake block now catches (ssl.SSLError, OSError) and wraps both in DiodeConfigError, matching _connect_socket.

Verification: pytest tests/test_client.py -q (136 passed), ruff check netboxlabs/diode/sdk/client.py tests/test_client.py.

/ai-review

Parse bracketed IPv6 authorities for the peer-cert probe and wrap
HTTPS_PROXY CONNECT in TLS before sending the tunnel request.

Co-authored-by: Cursor <cursoragent@cursor.com>
@jajeffries

Copy link
Copy Markdown
Contributor Author

AI review follow-up (low findings on 0c40e2ee)

  • IPv6 literals: _authority_host_port strips [/] so grpcs://[::1]:port skip-verify probes connect correctly.
  • HTTPS proxy: _connect_socket wraps the proxy TCP socket in TLS when the proxy URL uses https:// before issuing CONNECT.

Verification: pytest tests/test_client.py -q (139 passed), ruff check netboxlabs/diode/sdk/client.py tests/test_client.py.

/ai-review

Comment thread netboxlabs/diode/sdk/client.py Fixed
jajeffries and others added 2 commits September 18, 2026 13:32
Use PROTOCOL_TLS_CLIENT with minimum_version TLSv1_2 when wrapping
the skip-verify probe's HTTPS_PROXY socket so CodeQL matches the
peer handshake path.

Co-authored-by: Cursor <cursoragent@cursor.com>
The peer-cert probe issues CONNECT over a plain TCP socket to the
proxy, matching grpc.http_proxy, so the target TLS handshake is not
stacked on an existing proxy SSLSocket.

Co-authored-by: Cursor <cursoragent@cursor.com>
@jajeffries

Copy link
Copy Markdown
Contributor Author

AI review follow-up (nested TLS on 46c67dd)

Reverted TLS-wrapping the proxy socket before CONNECT. The skip-verify probe now mirrors grpc.http_proxy: plain TCP to the proxy, CONNECT, then a single TLS handshake to the target on the tunneled socket.

Waiver (low security, HTTPS proxy CONNECT): We are not adding a separate TLS session to the proxy for this probe path. That matches how gRPC uses HTTPS_PROXY for the real channel; a TLS-to-proxy CONNECT tunnel would also require TLS-in-TLS through the probe, which breaks when combined with the target handshake (see AI review on 46c67dd).

Verification: pytest tests/test_client.py -q (138 passed), ruff check netboxlabs/diode/sdk/client.py tests/test_client.py.

/ai-review

@nbl-ai-review

nbl-ai-review Bot commented Sep 18, 2026

Copy link
Copy Markdown

🔁 AI Code Review updated for 197c0b75see the review · 2 resolved · 0 open (run)

@leoparente

Copy link
Copy Markdown
Contributor

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 18, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-18T15:34:56.472658Z 197c0b7 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 197c0b75a5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

f"No peer certificate returned from {authority}"
)
pem = ssl.DER_cert_to_PEM_cert(der_cert).encode()
server_name = _tls_server_name_from_peercert(tls_sock.getpeercert(), host)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Decode the peer certificate before choosing the TLS override

When skip verification is used against a server whose certificate name differs from the target, such as a default Traefik self-signed certificate, verify_mode = ssl.CERT_NONE causes the non-binary SSLSocket.getpeercert() call to return an empty dictionary. This line therefore falls back to the target host, and gRPC still checks that hostname against the pinned certificate, causing the channel to fail even though certificate verification was disabled. Decode the already-fetched DER/PEM certificate or use credentials that genuinely disable hostname verification.

Useful? React with 👍 / 👎.

Comment on lines +231 to +233
for _ in range(_SKIP_VERIFY_PEER_PROBE_ATTEMPTS):
try:
pem, server_name = _fetch_peer_leaf_certificate(authority, proxy_url)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid probabilistic leaf pinning across load-balanced peers

When the target resolves to multiple backends with independently signed leaf certificates, these three probes can repeatedly select only a subset of the backends. The subsequent grpc.secure_channel connection may then reach an unsampled backend whose certificate is absent from root_certificates, producing CERTIFICATE_VERIFY_FAILED despite skip-verify being enabled. A fixed number of preflight probes cannot reliably emulate disabled verification; use a stable issuer or a channel credential mechanism that does not require sampling peer leaves.

Useful? React with 👍 / 👎.

Comment on lines +165 to +169
connect_request = (
f"CONNECT {host}:{port} HTTP/1.1\r\n"
f"Host: {host}:{port}\r\n\r\n"
)
sock.sendall(connect_request.encode())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Establish TLS before sending CONNECT to an HTTPS proxy

When the configured proxy URL itself uses https://, this code opens a raw TCP socket and immediately sends a plaintext HTTP CONNECT request. An HTTPS proxy expects a TLS handshake first, so the skip-verify certificate probe fails before the gRPC channel can be created, even though _validate_proxy_url accepts HTTPS proxy URLs. Either establish a TLS transport to the proxy before tunneling or reject this unsupported proxy scheme.

Useful? React with 👍 / 👎.

@leoparente

Copy link
Copy Markdown
Contributor

P1: skip_tls_verify still appears to enforce hostname verification

I don't think this fully matches Go's InsecureSkipVerify semantics yet.

In _fetch_peer_leaf_certificate() we do:

context.check_hostname = False
context.verify_mode = ssl.CERT_NONE

tls_sock = context.wrap_socket(raw_sock, server_hostname=host)

der_cert = tls_sock.getpeercert(binary_form=True)
server_name = _tls_server_name_from_peercert(
    tls_sock.getpeercert(),
    host,
)

The problem is that Python returns an empty decoded dict from getpeercert() when the certificate was not validated (CERT_NONE). So in the real skip-verify path _tls_server_name_from_peercert() will normally fall back to host.

We then pin the peer leaf cert as a trusted root and set:

("grpc.ssl_target_name_override", host)

which means gRPC can still reject the connection if the certificate SAN/CN doesn't match the target hostname/IP.

For example:

target: grpcs://10.0.0.20:443
cert SAN: diode.internal
DIODE_SKIP_TLS_VERIFY=true

Go's InsecureSkipVerify: true should accept this, while this implementation looks like it would still fail hostname validation.

Can we add an integration test with a real self-signed TLS server where the target hostname deliberately does not match the cert SAN, and make sure skip_tls_verify=True succeeds? We probably need to derive the SAN/CN from the DER certificate directly (or otherwise truly disable gRPC hostname verification) rather than relying on the decoded getpeercert() result.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants