SslStream on Unix leaks one X509 per intermediate per connection: AddExtraChainCertificates assumes add0 but CryptoNative_SslAddExtraChainCert is add1

[glima]

Description

Interop.Ssl.AddExtraChainCertificates leaks one OpenSSL X509 reference per intermediate certificate per SslStream connection on Unix.

There are two overloads of this helper, and they are character-for-character identical in how they handle ownership:

// src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.SslCtx.cs:47
internal static bool AddExtraChainCertificates(SafeSslContextHandle ctx, ReadOnlyCollection<X509Certificate2> chain)
{
    ...
    SafeX509Handle dupCertHandle = Crypto.X509UpRef(chain[i].Handle);
    if (!SslCtxAddExtraChainCert(ctx, dupCertHandle)) { ...; return false; }
    dupCertHandle.SetHandleAsInvalid(); // ownership has been transferred to sslHandle; ...
}

// src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Ssl.cs:312
internal static bool AddExtraChainCertificates(SafeSslHandle ssl, ReadOnlyCollection<X509Certificate2> chain)
{
    ...
    SafeX509Handle dupCertHandle = Crypto.X509UpRef(chain[i].Handle);
    if (!SslAddExtraChainCert(ssl, dupCertHandle)) { ...; return false; }
    dupCertHandle.SetHandleAsInvalid(); // ownership has been transferred to sslHandle; ...
}

But the two native shims they call do not have the same ownership semantics:

shim	pal_ssl.c	OpenSSL semantics
CryptoNative_SslCtxAddExtraChainCert	SSL_CTX_add_extra_chain_cert(ctx, x509) (L957)	add0 — takes ownership of the caller's reference
CryptoNative_SslAddExtraChainCert	SSL_ctrl(ssl, SSL_CTRL_CHAIN_CERT, **1**, x509) (L974)	add1 — larg == 1 selects ssl_cert_add1_chain_cert, which up-refs for itself

X509UpRef + SetHandleAsInvalid is the correct pattern for add0. Against add1 it takes two references and returns neither: libssl owns one for its chain list, and the managed one is abandoned rather than released.

Because SetHandleAsInvalid() deliberately disarms the SafeHandle, there is no finalizer for it either. The native X509, its X509_PUBKEY and the EVP_PKEY cached inside it live until the process exits, and are invisible to the GC, to dotnet-gcdump, and to any managed heap analysis.

The compiled form of the two shims side by side:

CryptoNative_SslAddExtraChainCert          CryptoNative_SslCtxAddExtraChainCert
  ba 01 00 00 00  mov $0x1,%edx             be 0e 00 00 00  mov $0xe,%esi
  4c 89 f7        mov %r14,%rdi             31 d2           xor %edx,%edx
  be 59 00 00 00  mov $0x59,%esi            48 89 d9        mov %rbx,%rcx
  e8 ..           call SSL_ctrl             e8 ..           call SSL_CTX_ctrl
         ^ larg = 1, cmd = 0x59 (CHAIN_CERT)       ^ larg = 0, cmd = 0x0e (EXTRA_CHAIN_CERT)

Impact

Any TLS client that presents a client certificate with intermediates and does not reuse an SslStreamCertificateContext leaks one certificate per connection. SslClientAuthenticationOptions.ClientCertificates is exactly this case: SslStream builds a fresh SslStreamCertificateContext per connection, which rebuilds the chain and materialises new native intermediate objects each time.

We hit this in production in the infrastructure behind an Azure PostgreSQL product: ~0.59 leaked certificates/s, ~14 MiB/h of native growth, linear across 19 hours, ending in an OOM kill of a 512 MiB container. The managed heap was flat at ~9 MB the whole time and a forced gen2 gcdump showed only 10 live X509Certificate2 while ~15,300 native certificates were outstanding, which is what made it hard to find.

Reproduction

Self-contained, no external tooling. The intermediate carries a 16 KB dummy extension purely so a single leaked certificate is visible in RSS without a native profiler — the leak is one X509 per connection at any certificate size.

dotnet run -c Release -- leak 2000
dotnet run -c Release -- ok   2000

Observed on .NET 10.0.10, Fedora, OpenSSL 3.5.7 (two runs each):

mode=leak  connections=2000   RSS  54.1 -> 109.9 MiB   growth 55.8 MiB   (28.6 KiB/connection)
mode=ok    connections=2000   RSS  51.4 ->  68.2 MiB   growth 16.8 MiB   ( 8.6 KiB/connection)
mode=leak  connections=2000   RSS  53.3 -> 103.8 MiB   growth 50.4 MiB   (25.8 KiB/connection)
mode=ok    connections=2000   RSS  51.4 ->  68.4 MiB   growth 17.0 MiB   ( 8.7 KiB/connection)

The delta between the arms is ~17 KiB/connection, which is the size of the intermediate. ok reuses one SslStreamCertificateContext, so the same native certificate is up-reffed every time and nothing accumulates.

Program.cs (net8.0+)

// Self-contained repro for the X509 reference leak in Interop.Ssl.AddExtraChainCertificates.
//
//   dotnet run -c Release -- leak 2000     # SslStream leaks one intermediate per connection
//   dotnet run -c Release -- ok   2000     # control: same work, one shared context
//
// No external tooling: the PKI is generated in-process and the TLS server runs on
// loopback in the same process.
//
// The intermediate carries a large dummy extension so that each leaked certificate is
// ~16 KB rather than ~1 KB. That is only to make the leak visible in RSS without a
// native profiler - the leak is one X509 object per connection at any certificate size.

using System.Diagnostics;
using System.Net;
using System.Net.Security;
using System.Net.Sockets;
using System.Security.Authentication;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;

string mode = args.Length > 0 ? args[0] : "leak";
int iterations = args.Length > 1 ? int.Parse(args[1]) : 2000;

// ---------------------------------------------------------------- build a small PKI
DateTimeOffset now = DateTimeOffset.UtcNow;
DateTimeOffset caFrom = now.AddDays(-1), caTo = now.AddYears(5);
DateTimeOffset eeFrom = now.AddDays(-1), eeTo = now.AddYears(1);

using var rootKey = RSA.Create(2048);
var rootReq = new CertificateRequest("CN=repro-root", rootKey, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
rootReq.CertificateExtensions.Add(new X509BasicConstraintsExtension(true, false, 0, true));
X509Certificate2 root = rootReq.CreateSelfSigned(caFrom, caTo);

using var interKey = RSA.Create(2048);
var interReq = new CertificateRequest("CN=repro-intermediate", interKey, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
interReq.CertificateExtensions.Add(new X509BasicConstraintsExtension(true, false, 0, true));
interReq.CertificateExtensions.Add(new X509KeyUsageExtension(X509KeyUsageFlags.KeyCertSign | X509KeyUsageFlags.CrlSign, true));
// padding, purely so one leaked certificate is big enough to see in RSS
interReq.CertificateExtensions.Add(new X509Extension("1.3.6.1.4.1.99999.1", new byte[16 * 1024], false));
X509Certificate2 intermediatePub = interReq.Create(root, caFrom, caTo.AddDays(-1), new byte[] { 1 });
X509Certificate2 intermediate = intermediatePub.CopyWithPrivateKey(interKey);
byte[] intermediateDer = intermediatePub.RawData;

X509Certificate2 IssueLeaf(string cn, int bits, X509Certificate2 issuer)
{
    using RSA key = RSA.Create(bits);
    var req = new CertificateRequest($"CN={cn}", key, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
    req.CertificateExtensions.Add(new X509BasicConstraintsExtension(false, false, 0, true));
    var sans = new SubjectAlternativeNameBuilder();
    sans.AddDnsName("localhost");
    req.CertificateExtensions.Add(sans.Build());
    X509Certificate2 pub = req.Create(issuer, eeFrom, eeTo, Guid.NewGuid().ToByteArray());
    X509Certificate2 withKey = pub.CopyWithPrivateKey(key);
    // round-trip through PKCS#12 so SslStream can use the private key on Unix
    return X509CertificateLoader.LoadPkcs12(withKey.Export(X509ContentType.Pkcs12), null, X509KeyStorageFlags.Exportable);
}

X509Certificate2 serverCert = IssueLeaf("localhost", 2048, intermediate);
X509Certificate2 clientCert = IssueLeaf("repro-client", 4096, intermediate);

// ---------------------------------------------------------------- loopback TLS server
// The server runs in a CHILD PROCESS. In-process it retains the peer chain per connection,
// which adds growth to both arms and hides the signal.
if (mode == "server")
{
    var srvListener = new TcpListener(IPAddress.Loopback, 0);
    srvListener.Start();
    Console.WriteLine(((IPEndPoint)srvListener.LocalEndpoint).Port);
    Console.Out.Flush();

    while (true)
    {
        TcpClient sc = await srvListener.AcceptTcpClientAsync();
        _ = Task.Run(async () =>
        {
            using (sc)
            using (var s = new SslStream(sc.GetStream(), false))
            {
                try
                {
                    await s.AuthenticateAsServerAsync(new SslServerAuthenticationOptions
                    {
                        ServerCertificate = serverCert,
                        ClientCertificateRequired = true,
                        RemoteCertificateValidationCallback = (_, _, _, _) => true,
                        EnabledSslProtocols = SslProtocols.Tls12 | SslProtocols.Tls13,
                        CertificateRevocationCheckMode = X509RevocationMode.NoCheck,
                    });
                    await s.WriteAsync(new byte[] { 1 });
                }
                catch (Exception ex)
                {
                    Console.Error.WriteLine("server: " + ex.GetBaseException().Message);
                }
            }
        });
    }
}

// Spawn the server as a child process. Both sides use accept-all validation callbacks, so
// the child generating its own PKI is fine.
var psi = new ProcessStartInfo { RedirectStandardOutput = true };
string exe = Environment.ProcessPath!;
string dll = System.Reflection.Assembly.GetEntryAssembly()!.Location;
psi.FileName = exe;
if (Path.GetFileNameWithoutExtension(exe).Equals("dotnet", StringComparison.OrdinalIgnoreCase))
{
    psi.ArgumentList.Add(dll);
}

psi.ArgumentList.Add("server");
using Process server = Process.Start(psi)!;
int port = int.Parse(server.StandardOutput.ReadLine()!);

// A shared context, used by the "ok" arm.
SslStreamCertificateContext sharedContext = SslStreamCertificateContext.Create(
    clientCert, new X509Certificate2Collection(X509CertificateLoader.LoadCertificate(intermediateDer)));

async Task Connect()
{
    // "leak": a fresh context each connection. This is what SslStream itself does when you
    // set SslClientAuthenticationOptions.ClientCertificates rather than reusing a context -
    // it rebuilds the chain per connection and materialises new intermediate objects.
    SslStreamCertificateContext ctx = mode == "ok"
        ? sharedContext
        : SslStreamCertificateContext.Create(
              clientCert, new X509Certificate2Collection(X509CertificateLoader.LoadCertificate(intermediateDer)));

    using var tcp = new TcpClient();
    await tcp.ConnectAsync(IPAddress.Loopback, port);
    using var ssl = new SslStream(tcp.GetStream(), false);
    await ssl.AuthenticateAsClientAsync(new SslClientAuthenticationOptions
    {
        TargetHost = "localhost",
        ClientCertificateContext = ctx,
        RemoteCertificateValidationCallback = (_, _, _, _) => true,
        EnabledSslProtocols = SslProtocols.Tls12 | SslProtocols.Tls13,
        CertificateRevocationCheckMode = X509RevocationMode.NoCheck,
    });
    byte[] b = new byte[1];
    await ssl.ReadAsync(b);
}

static long Rss()
{
    foreach (string l in File.ReadLines("/proc/self/status"))
        if (l.StartsWith("VmRSS:", StringComparison.Ordinal))
            return long.Parse(l.Split(':')[1].Trim().Split(' ')[0]) * 1024;
    return -1;
}

static void Settle()
{
    for (int i = 0; i < 3; i++)
    {
        GC.Collect(2, GCCollectionMode.Forced, true, true);
        GC.WaitForPendingFinalizers();
    }
    GC.Collect(2, GCCollectionMode.Forced, true, true);
}

for (int i = 0; i < 50; i++) await Connect();   // warm up
Settle();

long before = Rss();
var sw = Stopwatch.StartNew();
for (int i = 0; i < iterations; i++) await Connect();
sw.Stop();
Settle();
long after = Rss();

Console.WriteLine($"mode={mode}  connections={iterations}  elapsed={sw.Elapsed.TotalSeconds:F1}s");
Console.WriteLine($"  RSS {before / 1048576.0:F1} -> {after / 1048576.0:F1} MiB   growth {(after - before) / 1048576.0:F1} MiB"
                  + $"   ({(after - before) / (double)iterations / 1024.0:F1} KiB/connection)");
Console.WriteLine($"  managed heap after settle: {GC.GetTotalMemory(false) / 1048576.0:F1} MiB");
server.Kill();

Direct object counts

RSS is indirect, so we also counted the certificates. Tracking every X509 by pointer (X509_new_ex/d2i_X509/X509_dup create at refcount 1, X509_up_ref +1, X509_free -1, entry deleted at 0) over 800 connections, comparing a stock runtime against one where the single larg immediate was patched from 1 to 0 — the two libSystem.Security.Cryptography.Native.OpenSsl.so files differ by exactly one byte:

runtime	mode	X509 created	destroyed	leaked
stock	context per connection	1600	800	800
patched	context per connection	1600	1600	0
stock	shared context	800	800	0
patched	shared context	800	800	0

And on the shared-context arm, where the same native certificate is reused so nothing accumulates, the surplus reference is still directly visible:

runtime	X509_up_ref on the intermediate, 800 connections
stock	1599 (2 per connection: managed X509UpRef + ssl_cert_add1_chain_cert)
patched	799 (1 per connection)

Suggested fix

Make the SSL shim match its SSL_CTX sibling, which is what the identical managed helpers already assume:

--- a/src/native/libs/System.Security.Cryptography.Native/pal_ssl.c
+++ b/src/native/libs/System.Security.Cryptography.Native/pal_ssl.c
@@ int32_t CryptoNative_SslAddExtraChainCert(SSL* ssl, X509* x509)
-    if (SSL_ctrl(ssl, SSL_CTRL_CHAIN_CERT, 1,(void*)x509) == 1)
+    if (SSL_ctrl(ssl, SSL_CTRL_CHAIN_CERT, 0,(void*)x509) == 1)

CryptoNative_SslAddExtraChainCert has exactly one caller, Interop.Ssl.AddExtraChainCertificates, which already implements the add0 contract, so this is self-consistent.

Alternatively, keep add1 and release the managed reference instead — smaller behavioural change, does not touch a shared native entry point:

--- a/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Ssl.cs
+++ b/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.Ssl.cs
@@ internal static bool AddExtraChainCertificates(SafeSslHandle ssl, ReadOnlyCollection<X509Certificate2> chain)
-                dupCertHandle.SetHandleAsInvalid(); // ownership has been transferred to sslHandle; do not free via this safe handle
+                // SSL_add1_chain_cert took its own reference, so release ours.
+                dupCertHandle.Dispose();

Only the first was tested. CryptoNative_SslCtxAddExtraChainCert is genuinely add0 and must not be changed.

Workaround

Reuse a single SslStreamCertificateContext and set SslClientAuthenticationOptions.ClientCertificateContext instead of ClientCertificates. The intermediates are then created once, so the surplus reference lands on one long-lived object and nothing accumulates. This is the ok arm above.

Configuration

• .NET 10.0.10 (also present in 10.0.11, and the code is unchanged back through .NET 8)
• linux-x64, OpenSSL 3.5.7; originally found on a different distro with OpenSSL 3.3.7
• Client-side SslStream with a client certificate that has at least one intermediate

Regression?

No — the code is unchanged as far back as I checked. It is long-standing.

[dotnet-policy-service]

Tagging subscribers to this area: @bartonjs, @vcsjones, @dotnet/area-system-security
See info in area-owners.md if you want to be subscribed.