RDMA Fabric

Overview

An RDMA (Remote Direct Memory Access) fabric is the high-speed network interconnect backbone for large-scale AI training clusters, enabling GPUs to communicate directly without CPU involvement at ultra-low latency. RDMA fabrics are acutely sensitive to optical link instability (link flaps) because each L1 event triggers a cascade: PCS reset, L2/MAC re-initialization, IGP reconvergence, IB-12 RDMA timeout failures, DLB next-hop table churn, and DCQCN congestion management disruption. This sensitivity scales geometrically: what is manageable at 2,000 nodes becomes operationally critical at 20,000. For AI training workloads, each checkpoint failure puts compute hours equal to the checkpoint interval multiplied by the GPU count at risk.

OCI's operational approach favors reducing capacity (aggressively grooming suspect links) over tolerating instability, based on evidence that a stable lower-capacity fabric outperforms an unstable full-capacity one for training workloads.