?
Towards OpenUCX and GPUDirect Technology Support for the Angara Interconnect
Modern supercomputers consist of thousands of GPUs interconnected by high-speed HPC fabric. Angara interconnect supports Remote Direct Memory Access (RDMA) semantic, hence making it promising fabric solution for GPU-based supercomputers with a large amount of high-bandwidth RAM. Within this work we extend Angara interconnect low-level API allowing one transparently register (pin) GPU and user-space memory, hence enabling zero-copy approach and GPUDirect RDMA support. We present our implementation of Angara interconnect support within the state-of-the-art OpenUCX and OpenMPI communication stack. Memcpy function optimized for AMD GPU achieves up to 10×10× bandwidth speed up for medium to large MPI messages. With the help of Rendezvous protocol and our memory registration API the MPI bandwidth is increased by up to 68×