Telemetry Based Load Balancing of AI/ML Workloads

Author: Basar Cini, Product Manager & Surendra Anubolu, Distinguished Engineer, Broadcom Collection: 2023 Global Summit - Networking

Summary

Broadcom presents a production deployment case study from Tencent's hyperscale AI network demonstrating how inband telemetry (IFA -- In-band Flow Analyzer) and telemetry-based load balancing deliver up to 40% bandwidth utilization improvement over baseline DCQCN (Data Center Quantized Congestion Notification). The presentation covers two core technologies: IFA for real-time per-packet telemetry collection (capturing arrival time, departure time, ingress/egress ports, and queue depth at each hop without control-plane involvement) and stateful packet drop monitoring for identifying and diagnosing silent drops at scale.

Tencent's HPC network architecture uses a rail-optimized topology supporting 64K to 256K GPUs, organized as 256-GPU blocks aggregated into 16-64 block pods and 16-pod clusters, running RDMA over Converged Ethernet. The key challenge is that computational power does not scale linearly with GPU count due to network-induced losses: 0.1% RDMA packet loss causes 50% computational power loss, and network latency can account for up to 50% of job completion time in worst cases. NCCL bandwidth benchmarks show a 35% performance drop at 24,576 GPUs even with the improved NCCL 2.4 tree algorithm compared to theoretical maximums.

The telemetry-based solution deploys an IFA agent on each endpoint node alongside TCCL (Tencent Collective Communication Library, analogous to NCCL). The IFA agent monitors per-QP path quality using HPCC (High Precision Congestion Control) messages embedded in data packets, maintains a table mapping QPs to flow tuples and IFA metrics, and can autonomously switch traffic paths when degradation is detected. A centralized controller (NetQ) also receives telemetry for self-healing actions including scheduling and isolation. This dual-loop (endpoint-local and controller-based) approach achieves the 40% improvement over DCQCN, which itself requires tuning 15 configuration knobs.