Fabric Resiliency at Scale

Author: NVIDIA (Speaker unidentified on slides) Collection: 2024 OCP Global Summit - Networking

Summary

This presentation demonstrates why global adaptive routing (Global AR) is essential for maintaining AI training performance under fabric link failures. As AI model sizes grow, training jobs run longer, scale larger, and demand higher bisectional bandwidth. Any fabric failure -- link or switch -- can cause disproportionate performance degradation if the network only reacts based on local conditions (queue occupancy, port load). The speaker shows through real test results from NVIDIA's AI clusters in Israel that without global knowledge of fabric state, losing even a fraction of uplinks causes nonlinear performance collapse far worse than the proportional bandwidth loss.

The core argument is illustrated through a leaf-spine topology with 96 GPUs across 3 leaf switches connected via 4 spines with 8 parallel links each (baseline bisection bandwidth of 382 GB/sec). Under an all-to-all communication pattern, the system with only local adaptive routing drops to half its bandwidth when just 8 of 32 spine links fail, and continues degrading disproportionally. With Global AR enabled via NVIDIA Spectrum-X, degradation tracks linearly with actual bandwidth lost. At 50% uplinks down, Spectrum-X retains ~90% performance versus ~70% for traditional Ethernet; at 88% uplinks down, the gap widens further.

The presentation concludes with a call to action for AI infrastructure teams to adopt Spectrum-X and NVIDIA AIR simulation environments, emphasizing that clusters of 100,000+ GPUs already exist in production, making link failures statistically inevitable and global fabric awareness a necessity.