[client] NotLeaderOrFollowerException persists when a stale TabletServer endpoint is reused

[gyang94]

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

0.8.0

Please describe the bug 🐞

During a rolling upgrade, a TabletServer may restart with a new endpoint, while its old endpoint is reused by another TabletServer. A client may then keep a cached connection for the original server ID that actually connects to the other TabletServer.

A request such as listOffsets reaches the wrong server and receives NotLeaderOrFollowerException. The current handling completes the future exceptionally without disconnecting the cached connection or forcing a cluster metadata refresh. Subsequent caller-side retries reuse the same connection, making the failure persistent.

This is related to #2097 and #3389. Unlike the transient leader-change case described in #3389, the stale endpoint remains reachable because it now belongs to another TabletServer, so no NetworkException is raised to invalidate the connection.

Steps to reproduce

The following sequence illustrates how the problem occurs rather than requiring a deterministic IP-reuse setup:

1. The client caches server A -> endpoint X.
2. During a rolling upgrade, server A restarts at endpoint Y, and endpoint X is reused by server B.
3. Before its membership metadata is refreshed, the client creates a connection for server A using endpoint X. The connection is cached under server A's UID, but its actual peer is server B.
4. A later metadata refresh may update server A -> endpoint Y, but connection lookup still reuses the existing connection cached by server UID.
5. Requests for buckets led by server A continue to reach server B and receive NotLeaderOrFollowerException.
6. Because the exception does not invalidate or disconnect the cached connection, subsequent retries keep using the wrong connection.

Solution

On NotLeaderOrFollowerException, invalidate the affected routing metadata, disconnect the cached connection for the selected leader, refresh cluster metadata, and retry with a bounded retry count.

Also consider replacing an existing connection when the host or port changes for the same server UID.

Are you willing to submit a PR?

• I'm willing to submit a PR!

[BackendArchitectX]

Hi @gyang94, I’d like to work on this issue.

I’ll first reproduce it on current main, specifically the case where the same TabletServer UID is advertised with a new endpoint while the old endpoint remains reachable. I’ll then narrow the fix based on the reproducer and add regression coverage.

Could you please assign this issue to me if appropriate?

[BackendArchitectX]

I reproduced the issue on current main.

NettyClient caches connections by server UID only. When the same TabletServer UID is used with a different host/port, the client still reuses the old connection while that endpoint remains reachable.

I’ve added a deterministic regression test for this behavior and am now working on the smallest connection-layer fix.

[gyang94]

@BackendArchitectX Thanks for your work. Please go ahead. And I can help to review your fix.
btw I don't have permission to assign issue. We need committers help to assign it to you.

[BackendArchitectX]

Purpose

Linked issue: close #4256

During a rolling upgrade, a TabletServer may keep the same logical server UID while moving to a different host or port. If the previous endpoint remains reachable and is reused by another TabletServer, NettyClient currently continues using the connection cached only by server UID.

This means refreshed metadata cannot take effect at the transport layer. Even when a caller supplies an updated ServerNode, connection lookup can return the existing connection to the previous endpoint, causing requests to continue reaching the wrong TabletServer.

This change makes physical RPC connection identity endpoint-aware by keying connections with server UID, host and port.

Brief change log

• Key cached NettyClient connections by server UID, host and port instead of server UID alone.
• Allow connections to different physical endpoints for the same logical server UID to coexist.
• Keep disconnect(serverUid) as a logical-server operation by closing all endpoint-specific connections belonging to that UID.
• Preserve UID-level isReady() semantics when multiple endpoint connections exist.
• Make connection metric registration endpoint-specific so simultaneous connections sharing a UID do not overwrite each other's metric bookkeeping.
• Add deterministic regression coverage proving that an updated endpoint for the same TabletServer UID is actually used while the previous endpoint remains reachable.
• Add lifecycle coverage verifying that disconnect(serverUid) closes all connections belonging to that logical server.

Why not replace the existing connection?

Related PR #4216 handles a same-UID endpoint change by replacing and closing the connection currently associated with the UID.

This PR deliberately keeps physical endpoints independently keyed instead.

TabletServerGateway instances can capture a specific ServerNode. During metadata transitions, gateways referring to the previous and updated endpoints may therefore coexist temporarily. If connection ownership is still based only on UID, requests from those gateways can repeatedly replace each other's connection and close a connection that another caller is using.

Using (serverUid, host, port) as the physical connection identity avoids making the RPC layer decide which reachable endpoint is newer. The ServerNode supplied by the caller determines where that request is routed, while existing idle-connection handling eventually cleans up unused endpoints.

Why no NotLeaderOrFollowerException retry here?

NotLeaderOrFollowerException exposes the stale routing condition, but the persistent behavior in #4256 comes from the transport cache preventing an updated ServerNode from taking effect.

For FlussAdmin.listOffsets, metadata is refreshed before requests are constructed. On a subsequent caller retry, the refreshed TabletServer endpoint can therefore already be available. With UID-only connection caching, however, that updated endpoint is still masked by the old reachable connection.

This PR fixes that transport-layer condition directly and intentionally does not introduce another retry policy.

Tests

Passed:

• .\mvnw.cmd test -Dtest=NettyClientTest -pl fluss-rpc

  • 9 tests
  • 0 failures
  • 0 errors

• .\mvnw.cmd verify -pl fluss-rpc -Dtest="!ApiErrorTest"

  • 89 tests
  • 0 failures
  • 0 errors
  • 2 skipped

• Checkstyle: 0 violations

• git diff --check

The complete fluss-rpc verification on Windows additionally encounters the existing ApiErrorTest#testStringifyException line-ending assertion: the test expects LF while the Windows JVM stack trace uses CRLF. This test is unrelated to the changed RPC connection behavior.

The new regression intentionally isolates the transport condition rather than reproducing an entire rolling upgrade: two reachable TabletServers use the same UID with different endpoints, and the test verifies that a request supplied with the updated endpoint reaches that endpoint rather than the previously cached one.

API and Format

No public API, RPC protocol or storage-format changes.

Documentation

No documentation changes. This fixes existing RPC connection behavior when a server endpoint changes.

Generative AI disclosure

Yes — ChatGPT (GPT-5.6 Sol) assisted with root-cause analysis, implementation review and test planning.

[BackendArchitectX]

Thanks @gyang94. I've opened PR #4263 with the endpoint-aware connection fix and deterministic regression coverage.

I also documented the relationship with #4216 and the difference in connection handling.

Thanks for offering to take a look. Any feedback would be appreciated.

[BackendArchitectX]

@loserwang1024, @gyang94 mentioned that a committer is needed to assign #4256.

I?ve reproduced the stale-endpoint case and opened #4263 with the endpoint-aware NettyClient fix and deterministic regression coverage. I also coordinated with @litiliu on the overlap with #4216, and we agreed that #4263 should own the transport-level endpoint-change handling.

Could you assign #4256 to me when convenient? I?ll continue driving #4263 through review and resolution.

Thanks @gyang94 and @litiliu for the coordination.