Leveraging Open Technologies to Monitor Packet Drops in AI Cluster Fabrics

Author: Aldrin Isaac, eBay Collection: 2024 OCP Global Summit - Networking

Summary

Aldrin Isaac from eBay presents a critique of traditional network telemetry and proposes sFlow Drop Notification as a practical, vendor-neutral solution for monitoring packet drops in AI cluster fabrics. The core argument is that classical telemetry based on SNMP or gNMI polling of low-fidelity counters (ifInDiscards, ifOutDiscards) tells operators only that a loss occurred at a certain point in the network, but reveals nothing about who was impacted, why the packet was dropped, or which specific flow was affected. This approach, while already inadequate for traditional networks (by Isaac's 30-year experience), is completely unworkable for AI training where synchronized traffic bursts overwhelm shallow-buffer switches and packet drops directly impact job completion time.

The presentation traces the history of In-band Network Telemetry (INT), which emerged in 2015 with multiple open approaches (INT, IOAM RFC9197/9322/9326, IFA, TCP-INT) but failed to become mainstream after eight years due to over-emphasis on esoteric use cases, analysis paralysis, and fragmented vendor implementations. Isaac argues that what operators actually need is much simpler: knowing where packets are dropped, why they are dropped, and who was affected -- delivered in a vendor-neutral format that overlays with existing flow visibility.

sFlow Drop Notification is proposed as that solution. It leverages existing mirror-on-drop capabilities in current-generation DC switch ASICs, maps proprietary vendor drop reason codes to standardized vendor-independent codes, and uses Linux's native drop_monitor infrastructure so that both ASIC drops and kernel drops are handled equivalently. A reference implementation architecture for SONiC is presented, with implementation details from Edgecore and InMon. The solution is rate-limited (not sampled) and CPU-processed, which Isaac defends with the analogy: "in a storm you don't need to catch every raindrop to characterize the nature of the storm."