feat: add the dispatch and routing extension points a push transport needs#31
Open
mdashti wants to merge 3 commits into
Open
feat: add the dispatch and routing extension points a push transport needs#31mdashti wants to merge 3 commits into
mdashti wants to merge 3 commits into
Conversation
A transport that returns worker metrics out-of-band, rather than over the coordinator_channel return stream, needs its driver to decode the frames and file them into the executed plan's store before the per-task EXPLAIN rewrite reads it: metrics_store() with a public insert, a no-gRPC decode_task_metrics, and the frame builders collect_plan_metrics_protos and set_received_time. The metrics codec moves out from behind grpc with them, and the gRPC client drops its private copy of the decode. The test pins the frame-to-store path. Co-authored-by: Stu Hood <stuhood@gmail.com>
An embedder whose plan nodes the coordinator's codec cannot represent, or whose serialization needs embedder-side handling the codec extension point cannot express, serializes the dispatch bytes itself. The coordinator hands it the TaskKey and the ready-to-run per-task plan it would otherwise encode; returning None falls back to the coordinator's own encode. The tests pin the contract from all three sides: consultation with nested stages already Remote, source-provided bytes running the query, and a source error failing the dispatch. Co-authored-by: Stu Hood <stuhood@gmail.com>
A pull-based transport never places a produced partition: the consumer computes its own slice inside the boundary's execute. A push-based transport places every partition before any consumer asks, so it reads the boundary's consumer layout through route_partition instead of re-deriving it from node properties and drifting when the layout changes. NetworkCoalesceExec overrides it with an error: its consumers read whole per-producer-task groups, not slices. Co-authored-by: Stu Hood <stuhood@gmail.com>
stuhood
force-pushed
the
moe/protocol-grpc-shm
branch
from
0dfe291 to
36af89b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
This PR adds the two dispatch-side extension points a push-based transport needs:
DispatchPlanSourceandNetworkBoundary::route_partition.Why
Split out of the transport-completion PR (#27) at review, since neither is exercised by the in-process transport itself. Both exist for a transport that places produced partitions before any consumer asks and serializes plans with its own codec, like our shared-memory mesh (#28).
How
DispatchPlanSource: the coordinator hands over theTaskKeyand the ready-to-run per-task plan, and the embedder serializes it with its own codec; returningNonefalls back to the coordinator's encode.NetworkBoundary::route_partition+PartitionRoute: the consumer-slice layout, readable from the produce side, instead of re-deriving it from node properties and drifting when the layout changes.NetworkCoalesceExecrefuses with an error, since its consumers read per-producer-task groups, not slices.Tests
The dispatch source's contract is pinned from all three sides: consultation once per dispatched task with nested stages already
Remote, source-provided bytes running the query, and a source error failing the dispatch. The routing test pins the sliced formula on planner-built boundaries and the coalesce refusal. The shared-memory transport (#28) exercises both end to end.