Novel Approaches to Network Telemetry: Essential for AI Performance

Author: Roop Mukherjee, NVIDIA Networking (with Alon Gal and Omar Shai) Collection: 2024 OCP Global Summit - Networking

Summary

This presentation argues that traditional network telemetry, designed for workloads with diverse traffic patterns and fat-tailed distributions, is fundamentally unsuitable for AI training clusters. AI training traffic is highly regular, bimodal (either full line rate or zero), and synchronized across ranks within a job. Because AI training performance is dictated by its slowest link, even a single degraded NIC out of 100,000 can cost an entire day of training time. The key insight is that telemetry for AI networks should exploit the inherent symmetry and synchronicity of training traffic rather than looking for statistical outliers in the traditional sense.

The authors demonstrate a symmetry-based telemetry approach deployed on NVIDIA clusters with up to 100K GPUs connected by approximately 2,500 switches and 500,000 ports. By aggregating symmetric traffic across scalable units (SUs) and compressing periodic data into histograms rendered as heatmaps, operators can rapidly identify the "needle in the haystack" -- a single misbehaving link or NIC that causes the entire training job to slow down. This methodology is being incorporated into NVIDIA's Spectrum-X platform stack and is already running at several large customer sites.