Methodology and Observation of Congestion Control Impact on MoE Training Job Completion Time

Author: Alex Bort, Keysight Technologies Collection: 2025 OCP Global Summit - Expo Hall Sessions

Summary

Keysight Technologies presents research demonstrating that Mixture of Experts (MoE) AI training workloads create a unique congestion control challenge not present in dense 3D model-parallel training. The core finding is that MoE workloads alternate between AlltoAll (forward pass, many QPs with low per-QP bandwidth) and AllReduce (backward pass, few QPs with high per-QP bandwidth) operations on the same queue pairs (QPs). When DCQCN congestion control throttles down QP rates during congested AlltoAll phases, it cannot recover quickly enough before the high-bandwidth AllReduce phase begins, causing AllReduce to start at throttled rates and degrading overall job completion time.

Experimental results on a 400G two-tier leaf-spine topology using Keysight's Aries One traffic generators and Kite Data Center Builder software confirm: AllReduce alone achieves 95% of theoretical maximum bandwidth with no tail latency; AlltoAll alone achieves approximately 30% of ideal due to congestion but maintains regular patterns without tail latency; however, alternating AlltoAll and AllReduce (as MoE does) introduces significant tail latency in both collectives, with distorted timeline patterns showing different GPUs transmitting at different speeds.

Three solution categories are identified: (1) remove congestion through network design, (2) isolate AllReduce from AlltoAll on separate QP groups (complex, resource-intensive), or (3) improve network adaptiveness. Preliminary experiments show dynamic ECN on switches provides much higher bandwidth than static ECN without manual tuning, and fine-tuning DCQCN responsiveness (not sensitivity) on NICs provides near-flat performance independent of data size. Neither finding is conclusive but both are promising for addressing MoE congestion.