单一通信器，三家供应商：将 MoA 预测理论扩展至真实的 NVIDIA+AMD+CPU MPI 集成
One Communicator, Three Vendors: Extending the MoA Predictive Theory to a Real NVIDIA+AMD+CPU MPI Ensemble
Lenore Mullin · Gaetan Hains
中文摘要
该研究将 MPI 扩展验证至多供应商通信器，在 Slurm 异构作业中首次将 A100、MI100 和 CPU 共九个 rank 置于同一通信器下。理论提出基于 $1/t_i$ 的权重分配，但实测发现 MI100 耗时与行数呈非线性关系（6 行耗时 597 ms，线性预测仅 1.6 ms）。启动几何模型预测准确，优化 MI100 启动设置后耗时从 600 ms 降至 14.6 ms，能效提升 32 倍，但 8192 行时仍比 A100 慢约 95 倍。重跑实验表明，先前 8.1 倍加速及 22 ms Gatherv 结果是四 rank 共享单 GPU 的传输伪影。基于形状推导消息大小的 key/value 分区测试中，8.5 MB 环形 Allreduce 耗时 11.3-16.0 ms，接近预测值；计算时间复现误差约 1%，通信时间波动达 4 倍。
关键要点
- 01首次实现 A100、MI100 和 CPU 九 rank 在单一 MPI 通信器下的异构集成。
- 02MI100 耗时与行数呈非线性，优化启动设置后耗时降至 14.6 ms，能效提升 32 倍。
- 03发现先前 8.1 倍加速及 22 ms Gatherv 为四 rank 共享单 GPU 导致的传输伪影。
- 048.5 MB 环形 Allreduce 实测 11.3-16.0 ms，接近预测值，但通信时间波动达 4 倍。
解读
尚无解读。
原始英文摘要
arXiv:2610.22769v1 Announce Type: cross Abstract: The companion MPI-extension paper validated the first communication-bearing machine-shape ensemble: four NVIDIA A100 ranks and four CPU ranks on one NCSA Delta node under one Open MPI job, reassembled by MPI_Gatherv. This paper extends it to a multi-vendor communicator. A Slurm heterogeneous job spanning Delta's A100 and AMD MI100 partitions places A100, MI100 and CPU ranks (nine in all) under one communicator for the first time. We give the theory (a w_i proportional to 1/t_i weight split for three concurrent realizations; a het-job specification as an admissible directive) and report real results, including what the run exposed: the 1/t split assumes time linear in rows, and this fails for the MI100 rank (6 rows took 597 ms against a linear prediction of 1.6 ms). A launch-geometry model, with predictions recorded before measuring, succeeds for the A100 (time flat from 128 to 3456 rows, stepping at exactly 3520 and 6976) and places the MI100 floor in the key loop; changing its launch settings cut about 600 ms to 14.6 ms and the energy per call at 6 rows by 32x, though the MI100 remains about 95x slower than an A100 at 8192 rows. Controlled reruns show that the companion paper's 8.1x A100 result and 22 ms Gatherv were artifacts of four ranks sharing one GPU under Open MPI's default transport. A key/value-split partition whose message sizes are derived from shapes alone is then run: the ring Allreduce of the 8.5 MB message took 11.3, 16.0 and 15.3 ms against a predicted 11-12 ms, the small Allreduce missed its prediction in every run, and compute times reproduce to about 1% while communication times vary up to 4x between runs. Each idea is introduced with a figure and closed with worked examples.