[HOLD] ibv: remove several 16-bit node number limitations#8
Draft
PHHargrove wants to merge 1 commit into
Draft
Conversation
This commit removes the following assumptions that `GASNET_MAXNODES` is no larger than 16 bits: + XRC table uses 32-bit fields + XRC sort helper doesn't assume (node <= INT_MAX) + `gasnetc_epid_t` is now a 64-bit type (32 bits for node number) + Dynamic connect logic uses a 28-bit field for node number Of those changes, only the widening of `gasnetc_epid_t` has any potential impact on performance-critical paths on LP64 platforms (having made two struct types larger by up to four bytes each). The other change are relevant only during initialization and/or (non-default) dynamic connection establishment. Additionally a check for 16-bit overflow at initialization has been removed. However, that was already impossible since GEX fixed `gex_Rank_t` at 32 bits (to allow definition in `gasnet_fwd.h`) where G-1 had allowed ibv-conduit to narrow it to just 16. Together the changes in this commit support up to 28-bit node numbers in the initialization and connection logic. In the absence of dynamic connection support (not currently a configure-time option), there are no known barriers to job startup with 32-bit node numbers. However, this commit does NOT change `GASNET_MAXNODES` because the AM header is still limited to a 16-bit field for the source rank.
PHHargrove
force-pushed
the
ibv-pre-maxnodes
branch
from
92d105e to
6386ce6
Compare
Collaborator
Author
|
See https://bitbucket.org/berkeleylab/gasnet/pull-requests/464 for prior history of this PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is "DRAFT" status, pending completion of DC work to motivate a larger node number space.
ibv: remove several 16-bit node number limitations
This commit removes the following assumptions that
GASNET_MAXNODESis no larger than 16 bits:gasnetc_epid_tis now a 64-bit type (32 bits for node number)Of those changes, only the widening of
gasnetc_epid_thas any potential impact on performance-critical paths on LP64 platforms (having made two struct types larger by up to four bytes each). The other change are relevant only during initialization and/or (non-default) dynamic connection establishment.Additionally a check for 16-bit overflow at initialization has been removed. However, that was already impossible since GEX fixed
gex_Rank_tat 32 bits (to allow definition ingasnet_fwd.h) where G-1 had allowed ibv-conduit to narrow it to just 16.Together the changes in this commit support up to 28-bit node numbers in the initialization and connection logic. In the absence of dynamic connection support (not currently a configure-time option), there are no known barriers to job startup with 32-bit node numbers.
However, this commit does NOT change
GASNET_MAXNODESbecause the AM header is still limited to a 16-bit field for the source rank.