Fix X509 reference leak in SslStream extra chain certificates on Unix

[glima]

Fixes #132350.

CryptoNative_SslAddExtraChainCert and CryptoNative_SslCtxAddExtraChainCert have opposite ownership semantics, but the two managed helpers that drive them are character-for-character identical:

// Interop.SslCtx.cs:47                        // Interop.Ssl.cs:312
SafeX509Handle dup = Crypto.X509UpRef(...);    SafeX509Handle dup = Crypto.X509UpRef(...);
SslCtxAddExtraChainCert(ctx, dup);             SslAddExtraChainCert(ssl, dup);
dup.SetHandleAsInvalid();                      dup.SetHandleAsInvalid();

shim	call	semantics
CryptoNative_SslCtxAddExtraChainCert	SSL_CTX_add_extra_chain_cert	add0 — takes ownership
CryptoNative_SslAddExtraChainCert	SSL_ctrl(ssl, SSL_CTRL_CHAIN_CERT, **1**, x509)	add1 — up-refs for itself

X509UpRef + SetHandleAsInvalid is the add0 contract. Against add1 two references are taken and neither is returned, so one X509 leaks per intermediate per SSL handle. SetHandleAsInvalid also disarms the SafeHandle, so nothing is finalized either — the native X509, its X509_PUBKEY and the EVP_PKEY cached inside it live until process exit, invisible to the GC and to dotnet-gcdump.

This makes the SSL shim add0, matching its SSL_CTX sibling and the contract the caller already implements. CryptoNative_SslAddExtraChainCert has exactly one caller, Interop.Ssl.AddExtraChainCertificates, so the change is self-consistent. CryptoNative_SslCtxAddExtraChainCert is genuinely add0 and is untouched.

Verification

Because the change compiles to a single immediate, I validated it by patching that one byte in a shipped libSystem.Security.Cryptography.Native.OpenSsl.so (mov $0x1,%edx -> mov $0x0,%edx) so the two runtimes under test differ by exactly one byte and nothing else. Each arm resolved through its own private DOTNET_ROOT, verified from /proc/<pid>/maps.

Counting every X509 by pointer (create at refcount 1, X509_up_ref +1, X509_free -1, entry dropped at 0) over 800 client connections presenting a client certificate with one intermediate, .NET 10.0.10 / OpenSSL 3.5.7:

runtime	mode	created	destroyed	leaked
stock	fresh cert context per connection	1600	800	800
patched	fresh cert context per connection	1600	1600	0
stock	shared cert context	800	800	0
patched	shared cert context	800	800	0

On the shared-context arm nothing accumulates because the same native certificate is reused, but the surplus reference is still directly visible:

runtime	X509_up_ref on the intermediate over 800 connections
stock	1599 (2/connection: managed X509UpRef + ssl_cert_add1_chain_cert)
patched	799 (1/connection)

A self-contained repro that needs no tracing or binary patching is in the issue.

Not done

• I have not built dotnet/runtime or run the test suite; the validation above is against the shipped binary.
• No regression test is included. A natural one would be a client-side SslStream test that presents a certificate with intermediates over N connections and asserts the native certificate count does not grow, but that needs a hook the test suite does not have today. Happy to add one if you can point me at the right pattern.

Origin

Found while diagnosing an OOM in the infrastructure behind an Azure PostgreSQL product: ~0.59 leaked certificates/s, ~14 MiB/h of native growth, linear across 19 hours, in a container with a flat ~9 MB managed heap. A forced gen2 gcdump showed 10 live X509Certificate2 against ~15,300 outstanding native certificates, which is what made it hard to attribute.

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @bartonjs, @vcsjones, @dotnet/area-system-security
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR changes the OpenSSL shim for adding “extra chain” certificates to an SSL* so it uses add0 ownership semantics (OpenSSL takes ownership of the caller’s reference) instead of add1 semantics (OpenSSL up-refs). This aligns the native helper with the existing managed call pattern (X509UpRef + SetHandleAsInvalid) and prevents a per-connection X509 reference leak on Unix when intermediates are added per SSL handle.

Changes:

• Switch CryptoNative_SslAddExtraChainCert from SSL_CTRL_CHAIN_CERT with larg=1 to larg=0.
• Add an in-code comment documenting the intended ownership contract and why larg=0 is required.