Next-Token Prediction
The fundamental pre-training objective of autoregressive language models, where the network learns to guess the subsequent word in a given text sequence.
Think of It Like This
Like a game of autocomplete where you have to correctly guess the next word someone is going to say based on their sentence so far.
By consuming trillions of tokens of text and constantly predicting the next token, LLMs internalize the grammar, facts, and reasoning patterns of human language. The model outputs a probability distribution over its entire vocabulary for what token should follow. This simple objective is responsible for the massive emergent capabilities of modern AI.