Something is out of order

[RoLex]

Hello and thanks for this great component!

I currently have problem on Linux with your precompiled SSL libraries.

It seems that SSL handshake is successful, but when TCP server start to exchange regular protocol data, SSL fails:

10:19:02:689 SSL_read error 167773170 error:0A0003F2:SSL routines::ssl/tls alert unexpected message
// ^ here the client exchanges protocol data
10:19:02:689 epoll_wait, socket=12, type=ctAccept, local=(192.168.1.220:9393), remote=(192.168.1.38:22084), EPOLLERR=8, EPOLLHUP=16

This happens only on Linux using EPOLL, Windows and IOCP works without issues, also using your precompiled libraries of version 3.5.0.

My guess is that something is out of order in threads/locks when using EPOLL. There are several open issues considering EPOLL and thread/locking.

Please help!

[RoLex]

Here is what I think the problem is: openssl/openssl#20771

openssl s_client

This OpenSSL testing client confirms that there are two post-handshake session tickets arrived, but socket thinks that those are part of protocol data.

[RoLex]

Here is what I think the problem is: openssl/openssl#20771

openssl s_client

This OpenSSL testing client confirms that there are two post-handshake session tickets arrived, but socket thinks that those are part of protocol data.

Probably not this is the issue, but very old existing issue within EPOLL class: #11

[RoLex]

I think I found a solution. If there is a need to send protocol data within a TCrossDataEvent, there is a need to synchronize sending operation like this:

procedure OnReceived(const AComp: TObject; const AConn: ICrossConnection; const AData: Pointer; const ASize: Integer);
var
  LByte: TBytes;
  LConn: ICrossConnection;
begin
  LByte := 'ABC'; // implement send buffer
  LConn := AConn;

  TThread.Queue(nil,
    procedure
    begin
      LConn.SendBytes(LByte);
    end
  );
end;

There is a huge lack of demos so people can understand which parts of the cross socket needs to be synchronized or locked.

[RoLex]

But this solution does not work when there are many connections, when one of the other connections want to broadcast protocol data to other connections.

Question: How to send data to multiple SSL connections on TCrossDataEvent event from one connection?

Client speaks with server alone.
Client speaks with server > server speaks with other clients.

[RoLex]

I've traced the full send path. There are two distinct, concrete bugs in Net.CrossSslSocket.OpenSSL.pas — both in the SSL send path — and they line up exactly with "out of order" + "corrupted." The epoll unit itself is fine; the problem is how the SSL layer hands ciphertext to the epoll send queue.

Background: the send queue is zero-copy

In Net.CrossSocket.Epoll.pas:922, Send stores the buffer by pointer, not by copy:

LSendItem.Data := ABuf;   // no copy
LSendItem.Size := ALen;

The bytes are only read later, when EPOLLOUT fires in _HandleWrite. TCrossConnectionBase.DirectSend (Net.CrossSocket.Base.pas:1706) preserves this — it stores LBuffer := ABuffer and defers. So the contract is: the buffer must stay valid and unchanged until the send callback fires.

Bug 1 — use-after-free of the encrypted buffer

In _SslSendInner Net.CrossSslSocket.OpenSSL.pas:603:

LEncryptedData := _BIO_read_all;   // local TBytes
...
_Send(@LEncryptedData[0], Length(LEncryptedData),
  procedure(const AConnection: ICrossConnection; const ASuccess: Boolean)
  begin
    ...  // <-- does NOT reference LEncryptedData
  end);
Exit;

This calls the pointer overload of _Send (line 431), which goes straight to inherited DirectSend with no retention. The closure never references LEncryptedData, so Delphi/FPC does not capture it. When _SslSendInner exits, LEncryptedData's refcount hits 0 and the array is freed — but the epoll queue still holds @LEncryptedData[0]. By the time _HandleWrite actually sends, it reads freed/reused memory → garbage TLS records on the wire.

Compare the TBytes overload at line 437: it deliberately does LBytes := ABytes and nils it inside the closure — i.e. it retains the buffer for exactly this reason. TriggerConnected/TriggerReceived correctly use that overload (_Send(LHandshakeData)); _SslSendInner does not.

Bug 2 — ciphertext reordering under concurrent sends

The lock (FLock) is released at line 593 before the ciphertext is enqueued at line 606. So the atomic unit is just SSL_write+BIO_read, not SSL_write+BIO_read+enqueue.

Your own comment at line 484 states the required invariant: concurrent sends must queue to preserve FIFO. But the gate in _SslSend line 511 only diverts to the queue when (FPendingWrites.Count > 0) or FPumping. A plain in-flight _SslSendInner sets neither flag. So the first concurrent pair both fall through to _SslSendInner:

T1: FLock → SSL_write(A) → enc_A → unlock
T2: FLock → SSL_write(B) → enc_B → unlock
T2: _Send(enc_B)   ← enqueued first
T1: _Send(enc_A)   ← enqueued second

Wire order enc_B, enc_A → the peer's record layer desyncs → "bad record mac" / decrypt failure. This happens whenever two threads send on one connection, or when an app thread calls SendBytes while an IO thread is in _PumpPendingWrites (both reachable; EPOLLONESHOT only serializes a connection's epoll events, not app-thread sends).

SSL_MODE_ENABLE_PARTIAL_WRITE is not set, line 822 only sets AUTO_RETRY, so SSL_write is all-or-nothing — the partial-write CPS branch is effectively dead, which keeps the fix simple.

Recommended fix

Both collapse into one change in the LEncryptedData <> nil block of _SslSendInner: enqueue the ciphertext while still holding FLock, using the retaining TBytes overload.

• Switch _Send(@LEncryptedData[0], ...) → _Send(LEncryptedData, ...) (fixes the UAF).
• Keep FLock held across that _Send call (fixes ordering — the second sender blocks on the lock until the first has enqueued its ciphertext, so wire order = SSL_write order).

This is deadlock-safe on epoll: _Send only enqueues (the actual socket write and the callback are deferred to _HandleWrite, which invokes callbacks outside FEpLock — line 591, and no epoll path acquires FLock while holding FEpLock. So the new FLock → FEpLock order has no reverse path. The pump/parking logic stays untouched.

It does mean restructuring that loop body from try/finally to manual unlock so the unlock lands after _Send.