From Vision to Execution: The Story of Alibaba's 51.2T Switch

Author: J. Ya, Principal Engineer & Senior Director, Alibaba Cloud Collection: 2023 Global Summit - Networking

Summary

Alibaba Cloud presents Tigatron, its home-built 51.2 terabit fabric switch designed as the cornerstone of its HPN 7.0 (High Performance Network) architecture for AI/ML training clusters. Tigatron uses a 128x400GE QSFP112 configuration with a PHY-less VSR design running SONiC. The system began passing traffic in December 2022 and entered mass production in July 2023. The switch features a modular Universal Network Platform (UMP) design with three standardized components: an Open Control Module (OCM) using an Aspeed D CPU, a standardized BMC module using SODIMM 260-pin connectors, and a Removable Switch Module (RSM) that can be swapped from the chassis to support different ASICs (Broadcom Tomahawk 5, NVIDIA Spectrum-4, Cisco G200, Marvell Teralynx 10) and different optic form factors (QSFP112, OSFP, co-packaged optics).

The HPN 7.0 architecture employs a Clos topology optimized for AI workloads with large radix scale-out, multi-rail communication, and adaptive routing readiness. A single segment under one top-of-rack switch supports roughly 1,000 GPUs, two tiers support over 10,000 GPU servers, and three tiers can scale to hundreds of thousands of servers. Alibaba has also developed a cold plate liquid cooling variant of the Tigatron switch that cools both the ~800W switch ASIC and all 128 QSFP connectors, with this system operational as of approximately August 2023. The presentation includes significant discussion of Linear Pluggable Optics (LPO), where Alibaba has tested nearly 10 vendors' SR4 and DR4 modules achieving BER better than 1e-6 (some reaching 1e-9), with DR4 showing more consistent and better BER than SR4, particularly silicon photonic DR4.