Overlapping communications in gyrokinetic codes on accelerator‐based platforms

Communication and computation overlapping techniques have been introduced in the five-dimensional gyrokinetic codes GYSELA and GKV. In order to anticipate some of the exa-scale requirements, these codes were ported to the modern accelerators, Xeon Phi KNL and Tesla P 100 GPU. On accelerators, a serial version of GYSELA on KNL and GKV on GPU are respectively 1.3x and 7.4x faster than those on a single Skylake processor (a single socket). For the scalability, we have measured GYSELA performance on Xeon Phi KNL from 16 to 512 KNLs (1024 to 32k cores) and GKV performance on Tesla P 100 GPU from 32 to 256 GPUs. In their parallel versions, transpose communication in semi-Lagrangian solver in GYSELA or Convolution kernel in GKV turned out to be a main bottleneck. This indicates that in the exa-scale, the network constraints would be critical. In order to mitigate the communication costs, the pipeline and task-based overlapping techniques have been implemented in these codes. The GYSELA 2D advection solver has achieved a 33% to 92% speed up, and the GKV 2D convolution kernel has achieved a factor of 2 speed up with pipelining. The task-based approach gives 11% to 82% performance gain in the derivative computation of the electrostatic potential in GYSELA. We have shown that the pipeline-based approach is applicable with the presence of symmetry, while the task-based approach can be applicable to more general situations.