feat(rdma): add dmabuf-first MR registration with MORI_DISABLE_DMABUF_REG#355
Open
jhchouuu wants to merge 3 commits into
Open
feat(rdma): add dmabuf-first MR registration with MORI_DISABLE_DMABUF_REG#355jhchouuu wants to merge 3 commits into
jhchouuu wants to merge 3 commits into
Conversation
…_REG - Add RdmaDeviceContext::RegisterRdmaMemoryRegionAuto which tries ibv_reg_dmabuf_mr first (export dmabuf fd via hipMemGetHandleForAddressRange) and falls back to classic ibv_reg_mr on any failure. Avoids RLIMIT_MEMLOCK on large GPU heaps and the EINVAL we hit on hosts where the peermem path is unhappy. - Snapshot MORI_DISABLE_DMABUF_REG into a RdmaDeviceContext member at construction (matching the Context::sdmaEnabled / p2pDisabled cache pattern) so all MRs on the same context pick the same path. - Route the symmetric_memory non-VMM RDMA registration through the new Auto wrapper. The VMM path already has its own dmabuf fd and keeps calling RegisterRdmaMemoryRegionDmabuf directly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduce
RdmaDeviceContext::RegisterRdmaMemoryRegionAuto, a dmabuf-first MR registration path that falls back to classicibv_reg_mrwhen the HIP driver can't export a dmabuf fd oribv_reg_dmabuf_mrfails. The non-VMM symmetric memory allocator switches to this wrapper.Why
ibv_reg_mron large GPU heaps occasionally hitsEINVALon hosts where the peermem kernel module is missing / unhappy or whereRLIMIT_MEMLOCKis too low. The dmabuf path bypasses both: the kernel pins the dmabuf object, not user pages.transport_ib_common.cpptriesibv_reg_dmabuf_mrfirst, falls back silently). rocSHMEM still uses classicibv_reg_mronly, so adopting dmabuf moves mori ahead on the AMD side.Behavior
ibv_reg_mr(pinned)ibv_reg_mrRegisterRdmaChunksMORI_DISABLE_DMABUF_REG=1/true/on/yesibv_reg_mrDesign notes
RdmaDeviceContextmember at construction (mirrors theContext::sdmaEnabled/p2pDisabledcache). All MRs on the same context pick the same path, avoiding hard-to-debug mixed-mode states from late env mutations.ibv_reg_dmabuf_mrfailure we log aMORI_APP_WARNonce per call and silently fall back toibv_reg_mr— same policy as NVSHMEM.Test plan
MORI_DISABLE_DMABUF_REG=1forces classic path and still passesibv_reg_mrwithout aborting