Distributed Training Parallelism

Overview

Distributed training parallelism refers to the set of techniques used to shard large language models and their training data across multiple GPUs or nodes, enabling training of models too large to fit in a single accelerator's memory. Three primary axes are combined in practice:

• Tensor parallelism — splits individual weight tensors across devices. A tensor parallelism degree of 2 or 4 significantly reduces per-node memory footprint for large models, allowing training of very large context windows on commodity cluster nodes.
• Pipeline parallelism — distributes sequential model layers across machines, enabling deeper models by trading memory for inter-node communication latency.
• Data parallelism — replicates the model across worker groups, each processing a different data shard, with gradients aggregated across replicas.

There is no universally optimal parallelism configuration: the correct combination depends on accelerator type, interconnect bandwidth, and model architecture. As demonstrated by Foxconn's FoxBrain team, GPU FLOP utilization varies substantially between well-tuned and poorly-tuned configurations. Meta-Llama 3.1 models (8B, 70B, 405B parameters) were used as reference examples to illustrate how memory footprint scales with both model size and context length, motivating the need for all three parallelism axes when training at the 70B+ scale.