Pipeline Parallelism
A distributed training technique that splits a massive neural network's layers sequentially across multiple GPUs to fit models larger than a single memory bank.
Think of It Like This
Like a bucket brigade where one person handles the first half of the water transfer and immediately passes the bucket to the next person.
Unlike data parallelism where every GPU holds a full model copy, pipeline parallelism places layers 1-10 on GPU A, and layers 11-20 on GPU B. The intermediate activations are passed between devices. To prevent devices from idling (pipeline bubbles), micro-batching techniques like GPipe or PipeDream are heavily utilized.