Efficient All-to-All Collective Communication Schedules for Direct-Connect Topologies
arxiv(2023)
摘要
The all-to-all collective communications primitive is widely used in machine
learning (ML) and high performance computing (HPC) workloads, and optimizing
its performance is of interest to both ML and HPC communities. All-to-all is a
particularly challenging workload that can severely strain the underlying
interconnect bandwidth at scale. This paper takes a holistic approach to
optimize the performance of all-to-all collective communications on
supercomputer-scale direct-connect interconnects. We address several
algorithmic and practical challenges in developing efficient and
bandwidth-optimal all-to-all schedules for any topology and lowering the
schedules to various runtimes and interconnect technologies. We also propose a
novel topology that delivers near-optimal all-to-all performance.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要