AI Model Scaling

Overview

AI model scaling refers to the exponential growth in both model parameter counts and the GPU cluster sizes required to train them. Language and recommender model parameters have grown at approximately 14x per year from 2018 to 2025, reaching Llama 4 Behemoth at 2 trillion parameters and ChatGPT-4 at 1.8 trillion parameters. Concurrently, GPU cluster sizes have grown from 20K to 100K to emerging million-GPU scale, each transition requiring fundamentally new networking and infrastructure architectures.

This scaling trend is the primary driver behind the three-tier AI networking model (scale-up, scale-out, scale-across) and motivates investments in open networking standards such as Ultra Ethernet, P4-programmable NICs, and multi-plane fabric architectures. Communication overhead grows with scale: at million-GPU training runs, network communication stall averages 30% of total training time and exceeds 50% for some model architectures.