In-Network Collective Acceleration for AI Fabrics

Author: Surendra Anubolu, Broadcom; Nikhil Shetty, Oracle Cloud Infrastructure (OCI) Collection: 2025-OCP-Global-Summit-Networking

Summary

Collective operations (AllReduce, AllToAll, AllGather, ReduceScatter) account for more than 90% of AI fabric bandwidth. As GPU model sizes grow and inference latency requirements tighten, the cost of exposing GPUs to idle-wait during collective completion grows proportionally. This presentation from Broadcom and Oracle Cloud Infrastructure argues that the network fabric — not the GPU — is the natural locus for collective acceleration because switches carry 50 Tbps or more of bandwidth while requiring only ~3 TFlops of BF16 compute to perform reduction, a ratio that makes in-network computation (INC) extremely cost-effective relative to adding the same capability to GPU silicon.

Broadcom's Tomahawk Ultra is presented as a production silicon implementation of INC, supporting both a pull-based (read) protocol optimized for latency and a push-based (write) protocol optimized for performance. Both operate over standard RoCE — no proprietary headers — enabling any NIC with RDMA support to interoperate. Lab measurements with a 64-node testbed demonstrate 2x reduction in AllReduce completion time and significantly more predictable tail latency compared to ring-based collective protocols.

Oracle Cloud Infrastructure (OCI) validates this direction from an operator perspective, citing collective-aware fabric management already deployed in their scale-out network and outlining requirements for open programmable interfaces, secure multi-tenant partitioning, and SAI-based telemetry integration for INC hardware configuration and monitoring.