Skip to content

Cut fixed per-request costs out of the reshard control plane - #816

Open
copybara-service[bot] wants to merge 1 commit into
mainfrom
test_973205820
Open

Cut fixed per-request costs out of the reshard control plane#816
copybara-service[bot] wants to merge 1 commit into
mainfrom
test_973205820

Conversation

@copybara-service

@copybara-service copybara-service Bot commented Aug 29, 2026

Copy link
Copy Markdown

Cut fixed per-request costs out of the reshard control plane

Every Stage-3 coordination paid four avoidable fixed costs on its framed
RPCs (coordinate, GET_METADATA, receiver arm):

  1. The 4-byte length prefix and the body went out as two separate send()
    calls with Nagle enabled, on requests and responses alike, exposing
    every hop to the delayed-ACK stall (tens of ms on small RPCs).
  2. The destination controller was asked for every registered unit's full
    pool manifest on every request, although work units register once per
    engine lifetime.
  3. Client sockets carried no keepalive, so a black-holed peer was only
    detected at the full receive timeout.
  4. The framed server never reaped its per-connection threads: one
    std::thread handle and stack per request, held until shutdown.

Changes:

  • framed_rpc: single-buffer framing on both directions; TCP_NODELAY on
    client and accepted sockets; SO_KEEPALIVE plus TCP_USER_TIMEOUT bounded
    by the call's I/O timeout on client sockets; the accept loop joins
    finished connection threads.
  • reshard_coordinator: destination metadata is cached per controller
    address. Staleness (engine replacement) surfaces as a plan-build or
    receiver-arm failure. A failed attempt that used the cache drops the
    entry; while no receiver has acknowledged its arm the attempt is
    side-effect-free beyond the abandoned claim and is replayed once on
    fresh metadata. Once any receiver has acknowledged, the failure is
    returned as is and the next request re-queries.

Validation: reshard package builds and reshard_service_test passes in the
OSS bazel build (ml-build container, clang-18), including the new
RemoteMetadataCachedAndRefreshedOnStaleFailure test (cache hit on the
second request, exactly one refetch after a fingerprint-mismatch replay)
and PartialReceiverArmFailureDropsCacheWithoutReplay test (one of two
receivers refuses its arm: no replay, the cache entry is dropped, and the
next request re-queries).

GitHub: #808

@copybara-service
copybara-service Bot force-pushed the test_973205820 branch 2 times, most recently from 1325000 to 5c23960 Compare September 3, 2026 11:09
Every Stage-3 coordination paid four avoidable fixed costs on its framed
RPCs (coordinate, GET_METADATA, receiver arm):

1. The 4-byte length prefix and the body went out as two separate send()
   calls with Nagle enabled, on requests and responses alike, exposing
   every hop to the delayed-ACK stall (tens of ms on small RPCs).
2. The destination controller was asked for every registered unit's full
   pool manifest on every request, although work units register once per
   engine lifetime.
3. Client sockets carried no keepalive, so a black-holed peer was only
   detected at the full receive timeout.
4. The framed server never reaped its per-connection threads: one
   std::thread handle and stack per request, held until shutdown.

Changes:

- framed_rpc: single-buffer framing on both directions; TCP_NODELAY on
  client and accepted sockets; SO_KEEPALIVE plus TCP_USER_TIMEOUT bounded
  by the call's I/O timeout on client sockets; the accept loop joins
  finished connection threads.
- reshard_coordinator: destination metadata is cached per controller
  address. Staleness (engine replacement) surfaces as a plan-build or
  receiver-arm failure. A failed attempt that used the cache drops the
  entry; while no receiver has acknowledged its arm the attempt is
  side-effect-free beyond the abandoned claim and is replayed once on
  fresh metadata. Once any receiver has acknowledged, the failure is
  returned as is and the next request re-queries.

Validation: reshard package builds and reshard_service_test passes in the
OSS bazel build (ml-build container, clang-18), including the new
RemoteMetadataCachedAndRefreshedOnStaleFailure test (cache hit on the
second request, exactly one refetch after a fingerprint-mismatch replay)
and PartialReceiverArmFailureDropsCacheWithoutReplay test (one of two
receivers refuses its arm: no replay, the cache entry is dropped, and the
next request re-queries).

GitHub: #808
PiperOrigin-RevId: 973205820
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant