Analysis of Network Traffic Patterns for Collective Communication

Author: Arvin Sasen, Anju, Matt, Ashwin, Meta Collection: 2023 Global Summit - Networking

Summary

Meta engineers present a detailed empirical analysis of how all-to-all collective communication creates unexpected and problematic network traffic patterns in AI training clusters. The work is motivated by a prior Meta paper (arxiv 2104.05158) showing that communication accounts for roughly 40% of training completion time as clusters scale to 128+ GPUs, with all-to-all and all-reduce being the dominant bottlenecks. Using a testbed of 4 AI nodes (Zion EX, an OCP hardware platform) with 8 accelerators each (32 total, using 200G NIC endpoints), the team performed hardware packet capture and mirroring to analyze the fine-grained traffic characteristics of all-to-all operations.

The key discovery is that although all-to-all should theoretically produce uniform traffic where each sender transmits at 1/24th of line rate (in a 24-sender configuration), the actual RDMA NIC behavior produces bursty, spiky transmission patterns. Individual flows burst to 50-80 Gbps in microsecond intervals rather than transmitting at the expected steady ~8.3 Gbps. When these bursts from multiple senders align at a receiver, they stack and exceed the 200G line rate, creating incast conditions that reach 250-350 Gbps with burst widths of approximately 25 microseconds. This causes switch buffer accumulation (modeled at 200+ KB for a single port, growing larger in practice), which is multiplied across the 16 or more ports in a rack experiencing simultaneous incast. The shared ASIC buffer then triggers ECN marking and PFC, degrading end-to-end training performance. The team argues this problem cannot be solved by the network fabric alone and requires co-design across NICs, network fabric, and collective communication libraries, referencing the Chakra project (ML Commons) for standardized execution trace analysis.