[RFC]: Add openengine gRPC server support to trtllm-serve

[tanmayv25]

[RFC]: Add openengine gRPC server support to trtllm-serve

Goal: Add openengine as a selectable gRPC protocol for trtllm-serve. Ship the protocol bindings as an opt-in pip extra so the default install gains no new dependency.

Motivation

OpenEngine is a vendor-neutral gRPC contract for inference engines being developed by the Dynamo team. Adding OpenEngine support to trtllm-serve is intended to allow it to serve as a drop-in inference engine for clients implementing the OpenEngine contract (for example, with the Dynamo sidecar architecture).

Dependency Impact

OpenEngine’s bindings are generated by buf and published on the Buf Schema Registry. Consume them as pip packages from the https://buf.build/gen/python index, pinned to OpenEngine v0.1.0 at BSR module commit 768a93c7b44e40f28c692ad0b471a8f2. Ship them as an optional extra: pip install --extra-index-url https://buf.build/gen/python "tensorrt_llm[openengine]". The extra is listed in requirements-openengine.txt and wired into setup.py extras_require["openengine"]. The default install gets no new dependency. No buf CLI, no vendored .proto, no committed generated protobuf bindings, no build-time codegen.

The extra pins:

• openengine-openengine-grpc-python — BSR-generated gRPC stubs
• openengine-openengine-protocolbuffers-python and -pyi — BSR-generated messages and type stubs
• grpcio>=1.67.1,<2
• protobuf>=6.31.1,<7

Proposed Change

• Add selector --grpc-protocol {smg,openengine}, default smg. --grpc is unchanged. Backward compatible.
• Add subpackage tensorrt_llm/grpc/openengine/
• Add test tests/unittest/grpc/openengine
• The default HTTP server is unchanged; the sibling is disabled unless the port is given.
• First integration is a protocol stub: every RPC returns UNIMPLEMENTED
• Will keep adding implementing new features sequentially..

Feedback Period

~1 week.

CC

@NVIDIA/trt-llm-runtime-devs — owns tensorrt_llm/commands/serve.py and tensorrt_llm/grpc.
@NVIDIA/trt-llm-oss-compliance — owns requirements.txt; dependency and license review.
Assignee: laikhtewari.

Any Other Things

OpenEngine is experimental and pre-adoption (v0.1.0). Breaking changes are possible until OpenEngine reaches a stable state. This lands behind an opt-in flag and an opt-in extra, so the blast radius is small. The OpenEngine community will be responsible for maintaining these changes within the TensorRT-LLM OpenEngine gRPC server.

[tanmayv25]

Step 1 of this RFC is up as a draft PR for early review: #17019.

[QiJune]

From the TRT-LLM runtime maintainer perspective, we are open to reviewing OpenEngine as an optional, community-contributed gRPC integration. However, I would like to clarify TRT-LLM’s neutral position and integration boundaries before we move forward.

1. TRT-LLM should remain protocol-neutral

I do not think this RFC should designate OpenEngine as the replacement for SMG or as TRT-LLM’s preferred long-term target. In particular, I suggest rewording these statements:

“one contract can replace engine-specific protos like SMG’s”

“That makes it a better long-term target”

From TRT-LLM’s perspective, SMG and OpenEngine are both external model-gateway protocol integrations. We are open to accepting both under the same contribution policy. Their long-term adoption should be determined by technical maturity, ecosystem adoption, contributor activity, and maintenance quality—not predetermined by this RFC.

Therefore, for the convergence question, my preference is to make no replacement or convergence decision now. The two integrations should remain independent.

2. Keep protocol implementations isolated

The directory structure should reflect that neutrality:

  tensorrt_llm/grpc/
  ├── smg/
  └── openengine/

  tests/unittest/grpc/
  ├── smg/
  └── openengine/

Protocol bindings, request conversion, servicers, lifecycle code, and protocol-specific tests should remain inside their respective directories. The root grpc package should be protocol-neutral.

3. Protocol-specific dependencies should remain isolated

OpenEngine should not change the default TRT-LLM installation, dependency resolution, import path, or release process.

The existing unconditional SMG dependency is a separate historical issue and should not be treated as justification for adding another mandatory dependency. We can evaluate whether to make the SMG dependency optional separately.

4. Clarify ownership and compatibility expectations

These protocol integrations are expected to be contributed and maintained by their respective communities.

TRT-LLM maintainers will review contributions for repository quality, integration safety, and impact on the core runtime. The contributing communities are expected to own the adapter implementation, tests, protocol version updates, and related bug fixes.

Adapter code may use TRT-LLM internal APIs and data structures when necessary. However, doing so does not create a compatibility guarantee for those internal interfaces. TRT-LLM’s scheduler, executor, KV cache manager, internal queues, private attributes, and Python object structures will continue to evolve with the core runtime.

If an internal change breaks a protocol adapter, the corresponding adapter maintainers are expected to update it. Compatibility with protocol-specific adapters should not block normal TRT-LLM runtime development or releases.

With these boundaries documented in the RFC, we can continue the technical review of the proposal. The dependency choices and individual implementation PRs should still be reviewed separately.

[connorcarpenter15]

Thanks for laying out these boundaries. I updated the RFC and PR to align with them:

1. The RFC no longer presents OpenEngine as a replacement for SMG or as TRT-LLM’s preferred long-term protocol. It makes no replacement or convergence decision; the integrations remain independent.

2. PR [#17016][feat] Add selectable OpenEngine gRPC server #17084 keeps all new OpenEngine code under tensorrt_llm/grpc/openengine/ and its tests under tests/unittest/grpc/openengine/. It does not add OpenEngine-specific behavior to the root tensorrt_llm.grpc package. I intentionally left the existing SMG layout unchanged to avoid mixing an unrelated SMG reorganization into this PR; that can be handled separately if desired.

3. The generated bindings remain isolated behind tensorrt_llm[openengine] and are imported only when --grpc-protocol is set to openengine. The default installation, dependency set, import path, and ordinary trtllm-serve behavior remain unchanged.

4. The RFC assigns ownership of the adapter, tests, protocol updates, and integration bugs to the OpenEngine contributor community. The integration documentation also states that using TRT-LLM internal APIs creates no compatibility guarantee and that adapter maintenance must not block normal TRT-LLM development or releases.

PR #17084 remains limited to the transport and schema stub: every RPC returns UNIMPLEMENTED, and no request reaches the engine. Dependency approval and later RPC implementation can continue as separate reviews.