feat: add an in-process worker transport (no gRPC)#27
Open
mdashti wants to merge 3 commits into
Open
Conversation
This was referenced Jun 29, 2026
mdashti
force-pushed
the
moe/protocol-grpc-shm
branch
from
deb340e to
02ff23b
Compare
stuhood
force-pushed
the
gabrielmusat/abstract-grpc
branch
from
fa78bca to
c814a9c
Compare
stuhood
force-pushed
the
moe/protocol-grpc-shm
branch
2 times, most recently
from
3d4c5d8 to
c46b2e3
Compare
mdashti
force-pushed
the
moe/protocol-grpc-shm
branch
from
c46b2e3 to
f07eb09
Compare
mdashti
changed the base branch from
gabrielmusat/abstract-grpc
to
upstream-main-e05109b
mdashti
force-pushed
the
moe/protocol-grpc-shm
branch
2 times, most recently
from
903cd33 to
9270c0f
Compare
mdashti
commented
Jul 9, 2026
The prost message types carry no tonic dependency, so a transport that is not gRPC can speak the same wire shape without pulling in the gRPC stack. Only the tonic client and server stay gated; the generator emits those gates so a regeneration cannot drop them. tonic-prost feeds only the generated client and server, so it moves behind the feature too. Co-authored-by: Stu Hood <stuhood@gmail.com>
The benchmarks crate's dev-dependency on the lib re-unified grpc into every test build, so a genuine no-gRPC test run was impossible; the dataset suites move into the benchmarks crate and the gRPC-coupled test utilities gate behind grpc. A unit-test-no-grpc job then runs the whole lib suite with the feature off. Co-authored-by: Stu Hood <stuhood@gmail.com>
InProcessChannelResolver routes the three protocol methods straight to a co-located Worker, with no gRPC, no IPC, and no serialization round-trip: the reference implementation of the protocol for a co-located worker, and the first transport that exercises the abstraction with grpc off. Its end-to-end test (a distributed GROUP BY across tasks) runs under the no-gRPC CI job. Co-authored-by: Stu Hood <stuhood@gmail.com>
mdashti
force-pushed
the
moe/protocol-grpc-shm
branch
from
9270c0f to
9940349
Compare
Collaborator
Author
|
This is now on-par with the upstream PR: datafusion-contrib#548 |
mdashti
force-pushed
the
moe/protocol-grpc-shm
branch
2 times, most recently
from
be44b9c to
0dfe291
Compare
stuhood
force-pushed
the
moe/protocol-grpc-shm
branch
from
0dfe291 to
36af89b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ticket(s) Closed
N/A
What
This PR makes the
WorkerChannelabstraction usable by a transport that is not gRPC, and adds the in-process reference transport that proves it. Mirrored upstream as datafusion-contrib#548.Why
datafusion-contrib#512 abstracted the worker protocol, but gRPC stayed the only implementation and the proto sat behind the
grpcfeature, so nothing built or ran without it.How
One commit per concern, each with its own test:
grpc. Only the tonic client and server stay gated, and the generator emits those gates.benchmarksdev-dependency (it re-unifiedgrpcinto every test build) and added aunit-test-no-grpcCI job that runs the whole lib suite with the feature off.InProcessChannelResolver: routes the three protocol methods to a co-locatedWorker. No gRPC, no IPC, no serialization round trip.metrics_store()with a publicinsert, and a no-gRPCdecode_task_metrics: a driver whose transport returns metrics out-of-band files the decoded frames into the store the EXPLAIN rewrite reads.Tests
The in-process transport runs a distributed
GROUP BYend to end withgrpcoff in CI, and a unit test pins the metrics frame-to-store path.Stack
The dispatch and routing extension points a push transport needs are on top of this (#31); the shared-memory transport is on top of that (#28), and its diff is only
src/shm/*.