Cut fixed per-request costs out of the reshard control plane - #816
Open
copybara-service[bot] wants to merge 1 commit into
Open
Cut fixed per-request costs out of the reshard control plane#816copybara-service[bot] wants to merge 1 commit into
copybara-service[bot] wants to merge 1 commit into
Conversation
copybara-service
Bot
force-pushed
the
test_973205820
branch
2 times, most recently
from
September 3, 2026 11:09
1325000 to
5c23960
Compare
Every Stage-3 coordination paid four avoidable fixed costs on its framed RPCs (coordinate, GET_METADATA, receiver arm): 1. The 4-byte length prefix and the body went out as two separate send() calls with Nagle enabled, on requests and responses alike, exposing every hop to the delayed-ACK stall (tens of ms on small RPCs). 2. The destination controller was asked for every registered unit's full pool manifest on every request, although work units register once per engine lifetime. 3. Client sockets carried no keepalive, so a black-holed peer was only detected at the full receive timeout. 4. The framed server never reaped its per-connection threads: one std::thread handle and stack per request, held until shutdown. Changes: - framed_rpc: single-buffer framing on both directions; TCP_NODELAY on client and accepted sockets; SO_KEEPALIVE plus TCP_USER_TIMEOUT bounded by the call's I/O timeout on client sockets; the accept loop joins finished connection threads. - reshard_coordinator: destination metadata is cached per controller address. Staleness (engine replacement) surfaces as a plan-build or receiver-arm failure. A failed attempt that used the cache drops the entry; while no receiver has acknowledged its arm the attempt is side-effect-free beyond the abandoned claim and is replayed once on fresh metadata. Once any receiver has acknowledged, the failure is returned as is and the next request re-queries. Validation: reshard package builds and reshard_service_test passes in the OSS bazel build (ml-build container, clang-18), including the new RemoteMetadataCachedAndRefreshedOnStaleFailure test (cache hit on the second request, exactly one refetch after a fingerprint-mismatch replay) and PartialReceiverArmFailureDropsCacheWithoutReplay test (one of two receivers refuses its arm: no replay, the cache entry is dropped, and the next request re-queries). GitHub: #808 PiperOrigin-RevId: 973205820
copybara-service
Bot
force-pushed
the
test_973205820
branch
from
September 3, 2026 12:47
5c23960 to
24a7fe6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cut fixed per-request costs out of the reshard control plane
Every Stage-3 coordination paid four avoidable fixed costs on its framed
RPCs (coordinate, GET_METADATA, receiver arm):
calls with Nagle enabled, on requests and responses alike, exposing
every hop to the delayed-ACK stall (tens of ms on small RPCs).
pool manifest on every request, although work units register once per
engine lifetime.
detected at the full receive timeout.
std::thread handle and stack per request, held until shutdown.
Changes:
client and accepted sockets; SO_KEEPALIVE plus TCP_USER_TIMEOUT bounded
by the call's I/O timeout on client sockets; the accept loop joins
finished connection threads.
address. Staleness (engine replacement) surfaces as a plan-build or
receiver-arm failure. A failed attempt that used the cache drops the
entry; while no receiver has acknowledged its arm the attempt is
side-effect-free beyond the abandoned claim and is replayed once on
fresh metadata. Once any receiver has acknowledged, the failure is
returned as is and the next request re-queries.
Validation: reshard package builds and reshard_service_test passes in the
OSS bazel build (ml-build container, clang-18), including the new
RemoteMetadataCachedAndRefreshedOnStaleFailure test (cache hit on the
second request, exactly one refetch after a fingerprint-mismatch replay)
and PartialReceiverArmFailureDropsCacheWithoutReplay test (one of two
receivers refuses its arm: no replay, the cache entry is dropped, and the
next request re-queries).
GitHub: #808