feat(stream): support TLS passthrough on the stream proxy

[AlinsRan]

Description

Adds TLS passthrough to the stream proxy: a stream_proxy.tcp listen can now forward the encrypted stream to the upstream untouched while still picking that upstream from the SNI.

Until now "route by SNI" and "don't decrypt" were mutually exclusive in the stream subsystem — the only SNI source was a handshake the worker performed itself, so any SNI-matched stream route implied terminating TLS at the gateway. A backend that owns its own certificate, or an operator who does not want private keys on the gateway, had no way to get both.

Motivation: Gateway API TLSRoute passthrough

Gateway API defines TLSRoute as

a Gateway API type for specifying routing behavior of TLS requests from a client to an API object, i.e. Service. It allows to route traffic to specific backend based on the Server Name Indication (SNI) hostname provided during the TLS handshake.

and pairs it with a listener in Passthrough mode, described in the TLS configuration guide:

In Passthrough TLS mode, TLS settings do not take effect because the TLS session from the client is not terminated at the Gateway, but rather passes through the Gateway, encrypted.

That combination is what the stream proxy could not express: picking a backend from the SNI required terminating the handshake first. This change makes both hold at once — the SNI is taken from the prereaded ClientHello, and the session reaches the backend untouched.

Three modes per listen

apisix:
  proxy_mode: "http&stream"
  stream_proxy:
    tcp:
      - addr: 9100
        tls: true                # terminate (unchanged)
      - addr: 9101
        tls_passthrough: true    # pass everything through
      - addr: 9102
        tls: true                # mixed: the matched stream_route decides
        tls_passthrough: true

A passthrough listen is rendered without ssl, with ssl_preread on, and apisix/ssl.lua reads the SNI captured by ngx_stream_ssl_preread_module instead of ngx.ssl.server_name().

On a mixed port each connection is terminated or passed through according to tls_passthrough on the stream_route it matches (new field, default false). That comes from etcd, so moving a service between the two modes needs no gateway config change or restart.

Why the port mode itself is static

ssl_preread on and listen ... ssl are config-time directives with no runtime switch, and on one server they are incompatible — measured: the handshake completes, the preread phase then never sees a ClientHello and the connection stalls. So a port is structurally either terminate or preread.

What can be dynamic is the choice per connection. The two behaviours move into internal servers reachable over unix sockets; the preread phase of the public listen matches the route and sets the proxy_pass target:

server {
    listen 9102;                      # no ssl
    ssl_preread on;
    proxy_protocol on;                # carries the client across the internal hop
    access_log off;
    set $stream_tls_target "";
    preread_by_lua_block { apisix.stream_tls_route_phase("…_terminate_2", "…_passthrough_2") }
    proxy_pass $stream_tls_target;
}
server {                              # internal: terminates
    listen unix:…/stls-t2.sock ssl proxy_protocol;
    set_real_ip_from unix:;
    ssl_certificate_by_lua_block { apisix.ssl_phase() }
    preread_by_lua_block { apisix.stream_preread_phase(nil, true) }
    proxy_pass apisix_backend;
}
server {                              # internal: passes through
    listen unix:…/stls-p2.sock proxy_protocol;
    set_real_ip_from unix:;
    ssl_preread on;                   # the ClientHello was inspected, never consumed
    preread_by_lua_block { apisix.stream_preread_phase(true, true) }
    proxy_pass apisix_backend;
}

Plugins and the log phase run once, on whichever internal server owns the connection; the outer block matches the route and does nothing else. The client address crosses the hop in a PROXY protocol header and is restored with set_real_ip_from unix:; $server_addr/$server_port are restored from $proxy_protocol_server_addr/_port, so a stream_route carrying server_addr/server_port matches the same on both sides of the hop.

Only mixed ports pay that hop. tls: true and tls_passthrough: true ports keep their direct path.

Consequences of not terminating

These are inherent to passthrough, not implementation limits:

• payload-inspecting stream plugins (mqtt-proxy, xrpc, redis) and gateway mTLS do not apply; mTLS moves to the backend;
• an upstream with "scheme": "tls" is refused with a 503 and a diagnostic rather than silently double-wrapping — a second handshake would send the client's ClientHello to the upstream as payload;
• apisix_stream_metrics_zone counts nginx sessions and has no per-server switch, so a mixed port counts each client connection twice.

It is an explicit opt-in per listen port, never a default.

Changes

File	Change
apisix/cli/schema.lua	per-listen tls_passthrough
apisix/cli/ops.lua	groups stream listens by TLS mode (plain / passthrough / mixed) × proxy_protocol_to_upstream; rejects two listens on one address that would need different server blocks; names and length-checks the internal sockets
apisix/cli/ngx_tpl.lua	renders ssl_preread on, and for mixed listens the outer preread block plus the two internal servers and their upstreams
apisix/ssl.lua	server_name(clienthello, preread) — reads $ssl_preread_server_name when nothing was terminated locally, normalising "" to nil so fallback_sni still applies to a no-SNI ClientHello
apisix/init.lua	stream_preread_phase(tls_passthrough, behind_mixed_hop) skips verify_tls_client and restores the listen address; new stream_tls_route_phase() for the outer block
apisix/schema_def.lua	stream_route.tls_passthrough
apisix/stream/router/ip_port.lua	passes the mode through; no routing logic change
apisix/upstream.lua	scheme: tls under passthrough returns 503, ahead of the traffic-split short circuit so an inline upstream from that plugin is covered too
t/APISIX.pm	two test-framework fixes needed by the new .t, see below

Grouping listens by their TLS mode extends the existing proxy_protocol_to_upstream split rather than replacing it; t/cli/test_stream_proxy_protocol.sh (11 assertions on that grouping) stays green.

Test framework

Both fixes are required by the new .t and both were latent bugs:

• a bare --- stream_tls_verify section carries an empty value, which Perl treats as false, so the springboard called sslhandshake with verification disabled. With defined, a stream TLS test can actually verify the served chain. Checked both directions: pointing custom_trusted_cert at a certificate that does not sign the backend's now fails the case, pointing it at the backend's CA passes.
• a block that sets stream_server_config alongside stream_tls_request rendered a second, conflicting stream {} block, because on that path the stream block lives in the main config.

Tests

t/stream-node/tls-passthrough.t — 9 cases, 27 assertions:
t/cli/test_stream_tls_passthrough.sh — 8 assertions

Checklist

• I have explained the need for this PR and the problem it solves
• I have explained the changes or the new features added to this PR
• I have added tests corresponding to this change
• I have updated the documentation to reflect this change
• I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

[membphis]

Reviewed commit: 04fd78a199924450d1f97b5bb3fa6ec86d9052f0.

[P1] Fix the invalid upstream PROXY protocol header in mixed mode

With both tls: true and tls_passthrough: true, enabling proxy_protocol_to_upstream (directly or through the global default) causes the inner server to generate another PROXY protocol header after the Unix socket hop.

set_real_ip_from unix: restores the client address, but the destination remains the Unix socket. Assigning api_ctx.var.server_addr / server_port only updates the Lua variable cache; NGINX's PROXY protocol writer still reads c->local_sockaddr / c->local_socklen.

An isolated stream/realip reproducer built from the NGINX bundled in OpenResty 1.29.2.4, using the same Unix socket forwarding pattern, captured a header of this form:

PROXY TCP4 127.0.0.1 unix:/tmp/stls-.../inner.sock <client-port> 0

The destination is not a valid PROXY v1 TCP4/TCP6 address. A strict PROXY protocol backend rejects it before it can process the application data or TLS handshake. This affects both the terminating and passthrough branches of a mixed listener when upstream PROXY protocol is enabled. Dedicated passthrough/terminating listeners do not take this Unix socket path.

Please fix the upstream header so that it carries valid original source/destination addresses and ports, and add regression tests for both mixed-mode branches against a backend that actually parses PROXY protocol. The existing rendering check and real-client-IP test do not exercise this combination.

The reproduction validates the NGINX transport behavior; it was not a full APISIX/EE end-to-end run.

Code references: mixed listener template, Lua-only destination restoration.

[P2] Additional socket forwarding in mixed mode: author assessment requested

Please assess this point separately from the P1 correctness fix above.

For a fixed-IP upstream, without retries or additional plugin calls, the mixed path has two outbound connections per stream session: the outer proxy_pass $stream_tls_target connects to an internal Unix socket, then the inner proxy_pass apisix_backend connects to the business upstream. Compared with the direct listener path:

• Business-upstream connection: 1 -> 1.
• Internal Unix socket connection: 0 -> 1.
• Total outbound socket connections: 1 -> 2.

This is a local forwarding hop, not an extra remote-service lookup. It adds socket, buffering/copying, and scheduling work for both mixed-mode branches; dedicated passthrough/terminating listeners retain their direct path. The topology is verified from the implementation, but no latency or throughput regression has been measured here.

Could you confirm whether sharing one listener requires retaining this hop, evaluate connection-rate/throughput and resource usage against the direct modes, and state whether to keep this trade-off or use separate listener modes / an implementation without the extra hop? This item needs your explicit assessment; it is separate from the required PROXY protocol fix.

[membphis]

LGTM

[AlinsRan]

[P1] Fix the invalid upstream PROXY protocol header in mixed mode

Thanks — P1 reproduced exactly as you describe, and it is fixed in 2a2d39f.

[P2] Additional socket forwarding in mixed mode: author assessment requested

the hop is inherent to sharing one listener, not an implementation choice. ssl_preread on and listen ... ssl cannot coexist on one server.

So the 1 -> 2 outbound connections you counted is the price of putting both modes on one port. Dedicated tls and tls_passthrough listens keep the direct path — verified from the rendered config, they proxy_pass apisix_backend directly — and remain the documented recommendation whenever a whole port uses a single mode.

[membphis]

LGTM

[membphis]

Non-blocking observation for multi-worker deployments: mixed TLS mode makes the routing decision twice. The public listener first selects the terminating or passthrough internal server, and the internal Unix-socket server then matches the route again to select the upstream. During a control-plane or route/service configuration update, different workers may temporarily observe different snapshots, so a connection could combine the TLS mode from one snapshot with the upstream from another. This is more exposed with a large worker count. No action is required for this approval, but please keep the consistency window in mind and consider a multi-worker update regression or carrying the routing/version decision across the internal hop.