Optimal Path Utilization for Multi-Plane Fabric Design

Author: Wenda Ni, ByteDance; Jai Kumar, Broadcom; ChenChen Qi, ByteDance Collection: 2024 OCP Global Summit - Networking

Summary

This joint ByteDance-Broadcom presentation addresses the limitations of Adaptive Routing and Switching (ARS) in multi-tier AI fabric topologies, where ARS's reliance on local link state leaves it blind to remote downstream congestion and failures. ByteDance operates GPU clusters ranging from 1K to 128K GPUs using rail-optimized Clos topologies with two to four tiers, running RoCEv2 over lossless Ethernet with PFC. While deploying ARS on their custom 51.2T switch platform (64x800G OSFP with linear-drive pluggable optics and PHY-less design) delivered significant improvements -- 18.9% higher GPU bus bandwidth for AllReduce and 27.7% for AllToAll workloads -- three gap scenarios remained: remote downlink congestion, remote link failure, and asymmetric multi-link topologies.

The proposed solution extends ARS with remote link quality awareness through a two-layer hierarchical ECMP architecture. Rather than having multiple leaf routes share a single ARS-enabled next-hop group, each downstream path gets its own dedicated ECMP group. A Layer 1 next-hop group carries remote link quality information, which propagates to the Layer 2 ARS-enabled group for forwarding decisions. Remote quality updates are delivered via a proprietary microsecond-level notification protocol (deliberately kept outside BGP to preserve routing stability). The implementation introduces a new SAI object called the ARS Path Profile, with publisher and subscriber roles for downstream quality monitoring. ByteDance and Broadcom are seeking community participation in refining the SONiC HLD and SAI specification enhancements.