Phi-1
The 2023 Microsoft paper that proved massive parameter counts aren't necessary if the training data is of extremely high, 'textbook' quality.
Paper: Textbooks Are All You Need
Authors: Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Yin Tat Lee, Yuanzhi Li, Yongqiang Chen · 2023
Read the paper
The Problem
The prevailing wisdom in AI (reinforced by the Chinchilla laws) was that you needed massive datasets (trillions of tokens) scraped from the internet to train a capable model. However, internet data is mostly "garbage": repetitive, toxic, or poorly formatted. The models needed billions of parameters just to memorize and filter through this noise to extract actual knowledge.
The Idea
Microsoft researchers asked a simple question: What if we only fed the model perfect data? They created "Phi-1," a tiny model with only 1.3 Billion parameters. Instead of training it on the whole internet, they trained it on just 7 Billion tokens of "textbook quality" data. Half of this data was heavily filtered, high-quality code from the web. The other half was entirely synthetic: they used GPT-3.5 to generate artificial Python textbooks, exercises, and solutions. By ensuring every single token had high educational value, they believed a tiny model could learn to code.
How It Works
The training of Phi-1 focused entirely on data curation:
- Filtering: They used a classifier to score snippets of web data for "educational value," keeping only the absolute best.
- Synthetic Textbooks: They prompted GPT-3.5 to write textbooks on various coding topics, ensuring a high density of clear explanations and logic.
- Training: The 1.3B model was trained on this tiny 7B token dataset for just 8 passes (epochs). Because the dataset was so small, the training compute was negligible compared to frontier models.
Despite its tiny size, Phi-1 scored 50.6% on the HumanEval coding benchmark—beating models 10x to 50x its size (like LLaMA 1 65B).
Why It Mattered
Phi-1 proved that "Data Quality > Data Quantity." It sparked a massive resurgence of interest in Small Language Models (SLMs). It proved that for specific domains (like coding or reasoning), a heavily curated, synthetically generated dataset could allow a model that runs on a smartphone to outperform a massive datacenter model trained on noisy web data.
What Came After
Microsoft scaled this philosophy into the Phi model family (Phi-1.5, Phi-2, Phi-3), maintaining their status as the most capable small models in the world. The paper also validated the use of synthetic data (using large models to generate training data for smaller models) as the primary path forward once the industry runs out of high-quality human data on the internet.