Dual ToR Evolution: Active-Active ToR Deep Dive

Author: Ying, Engineer Lead, SONiC Team, Microsoft Collection: 2023 Global Summit - Networking

Summary

Microsoft presents the evolution of dual Top-of-Rack (ToR) technology in SONiC, progressing from single ToR to active-standby dual ToR and now to active-active dual ToR. The active-standby design uses a smart Y-cable with a MUX chip that duplicates northbound server traffic to both ToRs while allowing only one ToR's southbound I/O through. The active-active design replaces the MUX/MCU-based Y-cable with an FPGA and SoC on the server NIC side, using DAC (Direct Attach Copper) cables with gRPC control protocol running in-band on the data path instead of out-of-band I2C.

The active-active architecture doubles the effective server-to-ToR bandwidth by dispatching traffic per 5-tuple stream across both uplinks simultaneously. Both ToRs advertise the same set of VIP addresses upstream to T1 switches, enabling ECMP from T1 to T0. The FPGA on the server side implements flow-level ECMP dispatching for northbound traffic and handles ICMP-based link health monitoring. A SoC hosts a gRPC server that both ToRs connect to for controlling the FPGA's forwarding state. When a link fails, the standby-side ToR creates a tunnel route to forward traffic to the active ToR for delivery to the server. Microsoft deliberately chose this approach over MC-LAG and EVPN multi-homing due to scale concerns (10,000+ switches) and the desire to avoid overlay complexity in the underlay network.