Accelerating AI Hardware NPI Cluster-Less Validation of GPUs and Networking

Author: Matt Bergeron, Meta; Arvind Srinivasan, Meta; Ashutosh Kumar, Keysight Collection: 2025-OCP-Global-Summit-Networking

Summary

This talk presents "one-arm" or "cluster-less" validation methodology for AI accelerators and NICs that avoids the need for a full cluster test bed during New Product Introduction (NPI). The core insight is that during a collective communication operation (e.g., all-to-all), every participant knows exactly what every other participant will send and receive. This allows a hardware collectives emulator — here Keysight's KAI Data Center Builder — to mock all remote endpoints while a single accelerator plus NIC runs real collective communications without modification. The SUT (system under test) is reduced from an entire cluster to a single node, and further to just the DUT (device under test) by introducing virtual ranks that emulate arbitrary numbers of remote peers from a single physical mock.

The collaboration between Meta and Keysight produced measured data on collective completion time (CCT) at 8–24 physical ranks and extended to 256 virtual ranks. At 24 ranks (23 physical mocks at 400 Gbps each), NIC Rx achieved approximately 97% of line rate (17.0 Gbps vs. 17.4 Gbps roofline). CDF of flow completion time showed 70% of flows taking longer than ideal, with FCT varying from 7.65 to 7.9 ms, indicating measurable variability even without congestion-causing packet loss. An unexplained NIC Tx outlier — where some flows completed far faster than expected — was reproducible with virtual ranks and is still under investigation, pointing to possible NIC scheduler unfairness.