AI Cluster Networking

Overview

AI cluster networking refers to the specialized interconnect architecture required to support large-scale GPU clusters for AI training and inference. Unlike traditional data center networking — which primarily optimizes for throughput and uptime across diverse workloads — AI cluster networking must simultaneously satisfy five demanding requirements:

1. Ultra-high bandwidth: Multi-terabits per rack are now required. NVIDIA's NVL72 rack outputs 72 Tbps across 180 connections and 2,880 fiber connections; port speeds are progressing from 200G to 400G to 800G to 2x800G to 1.6T per port in rapid succession.
2. Sub-microsecond latency: Broadcom has reduced switch ASIC latency from 600–750 ns to approximately 250 ns specifically for AI workloads. Every nanosecond of switch latency contributes to overall training job duration at scale.
3. Real-time telemetry: Millisecond-level monitoring of network state is required for rapid fault detection and workload remediation in AI training jobs.
4. Zero-trust security: AI workloads require verification of every connection rather than implicit trust; security cannot be traded for latency improvements.
5. Scalable topology: Either Clos (leaf-spine, with ECMP load balancing) for current deployments, or Dragonfly (circular multi-connected) for large-scale future clusters starting ~2027.

The exponential growth of AI model scale — DeepSeek v1 at 67B parameters versus DeepSeek v3 at 671B, a 10x increase in roughly one generation — is driving corresponding growth in backend network bandwidth requirements.