Evolution of Ethernet-Based Switch Platforms and Fabrics to Meet Meta's AI Training Clusters

Author: Perves, Meta (FBOS software); Lingjun (Andun), Meta (hardware team) Collection: 2025-OCP-Global-Summit-Networking

Summary

This talk describes Meta's Non-Scheduled Fabric (NSF) architecture for AI training networks and introduces the Minipack3N platform — the first Meta switch using Nvidia's Spectrum 4 ASIC alongside the existing Minipack3 (Broadcom Tomahawk 5) and Kodiak3 (Cisco G200) platforms. Meta's AI network evolution progressed through three phases: (1) Deterministic Path Routing (DPR) — static prefix pinning to uplinks; (2) Enhanced ECMP (eCMP) — deeper packet field hashing; (3) First-Hop Traffic Engineering (FHTE) — controller-based least-congested path programming. The current generation runs two parallel architectures: DSF (Disaggregated Scheduled Fabric, VOQ-based, up to 18.5K GPUs) and NSF (Non-Scheduled Fabric, shallow-buffer Ethernet, scalable to 20K+ GPUs).

NSF uses the same 51.2T merchant silicon Ethernet switches at all three layers (Rack Training Switch RTSW, Fabric Training Switch FTSW, Spine Training Switch STSW), making deployment and configuration uniform. Each AI rack contains 72 GPUs (GB300) connected to three RTSWs (24 GPUs per RTSW). One L1 zone = 216 GPUs (3 RTSWs × 72 GPUs). At 96 L1 zones per fabric, NSF supports 20,736 GPUs. Switches are shallow-buffer (~160 MB), using ECN+PFC for lossless transport and Adaptive Routing (AR) for optimal link utilization — flowlet mode or per-packet mode. FBOS (Facebook Open Switch system) is open-source and uses OCP's SAI interface; it supports any merchant silicon with SAI-compliant SDK. The Minipack3N introduces Nvidia Spectrum 4 as a third ASIC option, with the same 4RU form factor, pluggable SCM module, and similar power/buffer profile as the other platforms.