Accurate Cable Length Measurement for Optimized Data Center Networking

Author: Mahek Mahajan, Broadcom; Jasim Baga, Meta (FBOS team) Collection: 2025-OCP-Global-Summit-Networking

Summary

This presentation argues that cable length measurement has become a first-class operational concern in AI data center networking, where it was historically ignored. The root cause is that AI workloads produce bursty traffic that puts extreme demands on buffer provisioning: under-provisioning causes packet drops and retransmission stalls to AI collectives; over-provisioning increases latency and reduces network efficiency. Buffer provisioning must be matched precisely to actual propagation delay, which is a direct function of cable length. Minor discrepancies — even tens of meters off from plan — can cause performance degradation or unexplained packet loss.

Meta's experience with DSF (Disaggregated Scheduled Fabric) is the motivating deployment story. DSF is a VOQ (Virtual Output Queueing)-based end-to-end scheduling fabric that supports up to 18.5K GPUs in a two-stage configuration and powers the Prometheus (gigawatt-plus) data center build. DSF has strict end-to-end cable length requirements, made tighter when error correction was added, and further tightened for dual-stage configurations. Meta detected an out-of-spec condition early through cable-length monitoring, triggered a painful but manageable recabling effort, and avoided discovering the issue through unexplained packet loss in production.

The proposed solution is a vendor-agnostic, platform-neutral cable length detection feature implemented in the SAI (Switch Abstraction Interface) and SDK stack. On link-up, the SAI triggers the SDK to send a PTP-format PDelay_Req packet to the peer; the peer timestamps it and returns a PDelay_Resp with embedded timestamps; the requester calculates propagation delay as D = (t2-t1 + t4-t3)/2. Hardware timestamping (not clock synchronization) is required. The result is cached in the SDK and queryable via SAI. Meta achieved less than 10 meters of measurement error. The authors are taking this to the SAI community for standardization.