Pre-training
The initial, computationally massive phase where a model learns fundamental patterns, features, and knowledge from a vast dataset without task-specific labels.
Think of It Like This
Like a child spending years learning how to walk, talk, and understand gravity before they eventually specialize in playing a sport.
During pre-training, an LLM learns grammar and facts via next-token prediction, while a vision model learns edges and textures via contrastive learning. This phase requires thousands of GPUs and terabytes of data. The resulting 'foundation model' can then be cheaply adapted to hundreds of downstream tasks via fine-tuning.