nvshmem

Efficient Distributed GPU Programming for Exascale, an SC/ISC Tutorial

Artifact for OSDI'23: MGG: Accelerating Graph Neural Networks with Fine-grained intra-kernel Communication-Computation Pipelining on Multi-GPU Platforms.

N-Ways to Multi-GPU Programming

My notes on various HPC papers.

Source code for the CPU-Free model - a fully autonomous execution model for multi-GPU applications that completely excludes the involvement of the CPU beyond the initial kernel launch.

GPU-accelerated linear solvers based on the conjugate gradient (CG) method, supporting NVIDIA and AMD GPUs with GPU-aware MPI, NCCL, RCCL or NVSHMEM

GPU 计算与通信融合入门：stream overlap / epilogue peer write / NVSHMEM warp 分工 / copy engine，四套可运行示例

Repository for ICS'25: MG-𝛼GCD: Accelerating Graph Community Detection on Multi-GPU Platforms

2-D Jacobi Laplace solver in four parallel models — MPI+OpenMP, HDF5 parallel I/O, OpenACC, NVSHMEM — scaling to 1120 cores and 40 A100s on Leonardo

NVIDIA NVSHMEM is a parallel programming interface for NVIDIA GPUs based on OpenSHMEM. NVSHMEM can significantly reduce multi-process communication and coordination overheads by allowing programmers to perform one-sided communication from within CUDA kernels and on CUDA streams.

2× L4 GPUs (Modal `L4:2` ) in comparision to megatronlm

Comparing GPU-communication frameworks. World-first multi-GPU maximum cardinality bipartite matching.

Stencil-aware multi-GPU Conjugate Gradient solver (CUDA + MPI, NCCL, NVSHMEM), measured on 8× A100-SXM4-80GB. SpMV 2.08× faster than cuSPARSE CSR, CG 1.44× faster than NVIDIA AmgX at equal algorithm, 93.5% strong scaling efficiency (2D). Profiled with Nsight Systems and Nsight Compute.

An MLIR dialect and toolchain for analyzing, verifying, and optimizing SPMD parallel programs across MPI, OpenSHMEM, NVSHMEM, NCCL, and OpenMP: detecting data races, collective-communication mismatches, and thread-safety errors, and improving performance. Mirror of the RWTH Aachen GitLab repository; pull requests are welcome and merged there.

Where RDMA faults survive or vanish from the NIC to the application (CPU verbs, NCCL, NCCL GIN, NVSHMEM): pre-registered measurements, five propagation rules, library-internal transparent recovery, and an NVSHMEM bug fixed upstream (NVIDIA/nvshmem#117)

An unofficial mirror of NVSHMEM source code releases by NVIDIA

CuTe MoE Communication and Overlap Kernels