The Challenges and Practices of Network Stability in Alibaba's Large-Scale Computing Clusters

Author: Shao Shi, Alibaba; Surendra Anubolu, Distinguished Engineer, Broadcom Collection: 2024 OCP Global Summit - Networking

Summary

This joint Alibaba-Broadcom presentation addresses two fundamental challenges in maintaining network stability for distributed training at Alibaba's scale: (1) rapidly detecting, localizing, and recovering from communication faults, and (2) observing real network communication behavior to identify performance optimization opportunities. Alibaba's clusters deploy 15K GPUs per pod with network components at 10x the GPU count (switches, optical modules, fiber optics), resulting in a monthly link failure rate of 0.057% and switch critical error rate of 0.051% -- meaning communication issues occur daily.

The core contribution is Alternative Marking DSCP (A.M.D.), an end-to-end packet loss detection mechanism inspired by RFC 8321 and extending Alibaba's existing Intra-Aggregate-Drop (intra-job) counter-based detection. Unlike the per-hop intra-job approach, A.M.D. uses a DSCP field marked by the NIC (not the switch chip) to carry alternate marking information across the entire network path, enabling port-level accounting that covers all network components including the critical last hop (NIC-to-ToR), which accounts for nearly half of all network components and has the highest failure ratio due to multi-vendor integration. A.M.D. has been deployed in Alibaba's production environment for three years and provides real-time fault localization within seconds.

Broadcom's Surendra Anubolu complements the Alibaba portion by discussing the importance of hardware-based streaming telemetry at microsecond granularity (approximately 100 microseconds) for port-level queue and buffer statistics. This is essential because AI training traffic is bursty, synchronous, and cyclic, making coarse-grained counter polling (hundreds of milliseconds to seconds) insufficient to detect load-balancing issues that only manifest during the communication phase of training steps.