Data vs Tensor vs Pipeline Parallelism
Comparing the three ways to split neural networks across multiple GPUs.
Verdict: Use Data Parallelism if the model fits entirely on one GPU; if it doesn't fit, use Pipeline Parallelism for simplicity, or Tensor Parallelism if you need maximum inter-layer throughput.
The Short Answer
When scaling AI, you must distribute work across multiple GPUs. Data Parallelism copies the entire model onto every GPU and splits the input data (GPU 1 trains on images 1-10, GPU 2 trains on 11-20). Pipeline Parallelism puts Layer 1 on GPU 1 and Layer 2 on GPU 2, acting like a factory assembly line. Tensor Parallelism mathematically slices a single massive matrix layer in half, putting the left side on GPU 1 and the right side on GPU 2, calculating them simultaneously.
Where They Differ
| Feature | Data Parallelism (DP) | Pipeline Parallelism (PP) | Tensor Parallelism (TP) |
|---|---|---|---|
| What gets split? | The batch of data | The sequential layers | Individual weight matrices |
| Network Communication | Only at the end of a batch | Between layers (moderate) | Within every layer (massive) |
| Model Size Limit | Must fit on 1 GPU | Can span many GPUs | Can span many GPUs |
Choose Data Parallelism When
- Your model easily fits on a single GPU: If you are training a 2B parameter model, it fits cleanly into 16GB of VRAM. You just copy it to 8 GPUs and feed each GPU different data batches to train 8x faster. This requires almost no custom engineering (native in PyTorch DDP).
Choose Pipeline Parallelism When
- Your model is too big for one GPU: A 70B model requires 140GB of VRAM just for the weights; it physically cannot load onto a standard 80GB GPU. Pipeline parallelism puts layers 1-40 on GPU A and layers 41-80 on GPU B.
- You are crossing network boundaries: Because GPUs only communicate at layer boundaries, the network overhead is relatively low, making it suitable for linking GPUs across different physical servers.
Choose Tensor Parallelism When
- You need the absolute lowest latency: Pipeline parallelism introduces "bubbles" (GPU B sits idle waiting for GPU A to finish layer 1). Tensor parallelism splits the math of a single layer across multiple GPUs simultaneously. Because the GPUs must sync their math during the layer calculation, it requires massive interconnect bandwidth (NVLink) and can only be done between GPUs on the exact same motherboard.
What People Get Wrong
People assume they just pick one. In reality, frontier models (like GPT-4 or Llama-3) are trained using 3D Parallelism—they use all three simultaneously. They use Tensor Parallelism to split layers across 8 GPUs inside a single server, Pipeline Parallelism to link 10 servers together, and Data Parallelism to replicate that 80-GPU pod hundreds of times to churn through the data.