Fungible Hardware Tables as an Ordered Chain for Traffic Engineering and AI/ML Applications

Author: Memory Mundran (Meta), Jaykumar (Broadcom), Shri Krishna Kare (Meta) Collection: 2023 Global Summit - Networking

Summary

Meta and Broadcom present a collaborative solution to traffic engineering challenges in Meta's backend AI/ML training network. HPC training jobs running on 32 to 256 GPUs produce low-entropy, bursty elephant flows (RoCE v2 RDMA traffic on fixed UDP destination ports) that cause congestion when using standard ECMP. The initial approach of deterministic static path pinning (pinning odd-numbered host traffic to one set of 8 uplinks, even to another) works when jobs use full racks but fails with partial rack scheduling and link drains. Meta's solution uses a centralized controller that receives job placement information from the scheduler, combines it with real-time topology from Open/R, runs a Constrained Shortest Path First (CSPF) algorithm, and programs optimal per-flow paths into switches using exact-match hardware tables.

The key technical challenge was that SAI's existing pipeline lacked support for exact-match tables, which offer much higher scale than TCAM for source-destination prefix pair forwarding. Broadcom introduced a new SAI specification enhancement called "chain groups" -- a software abstraction representing fungible hardware tables as ordered chains. Chain groups allow multiple table groups (TCAM-based and exact-match) to be combined within a single ACL stage without adding new pipeline stages, maintaining backward compatibility. The implementation in FBOSS (Facebook Open Switching System) demonstrates how chain groups and exact-match tables can be deployed incrementally over warm boot (vboot) without disrupting traffic. Benchmark results show that adaptive traffic engineering maintains consistent job completion times even with imperfect scheduling and reduced network capacity (14-18 uplinks), while deterministic path routing degrades significantly.