FSDP
A memory optimization technique that splits model parameters, gradients, and optimizer states across multiple GPUs to enable the training of massive networks.
Think of It Like This
Like a group of friends splitting the cost of a massive dinner bill so no single person has to carry the entire financial burden on their own credit card.
Fully Sharded Data Parallel (FSDP) eliminates the memory redundancies found in standard Data Parallel training, where every GPU holds a full copy of the model. By sharding the state, FSDP allows developers to train models that far exceed the VRAM of a single chip. Parameters are temporarily un-sharded just-in-time for the forward and backward passes during execution.