Core ML

Tabular Foundation Models

What if you could train a Transformer on millions of completely random Excel spreadsheets, so that when you give it a brand new spreadsheet, it can predict the target column in 1 second without any training?

Instead of training a new model via gradient descent, TabPFN takes the entire training dataset as the 'context' and outputs predictions for the test set in a single forward pass.

Why Does This Exist?

If you want to classify a text document today, you don't collect 10,000 labeled examples and train a model from scratch via gradient descent. You just send the text to a massive Foundation Model (like GPT-4), give it 5 examples in the prompt, and ask for the answer. This is called In-Context Learning.

But for tabular data, we are still stuck in 2014. If you have a CSV file with 1,000 rows, you have to split it into train/test, do hyperparameter tuning on XGBoost, train the model, evaluate it, and deploy it. It takes hours.

Tabular Foundation Models exist to bring the magic of LLM-style In-Context Learning to tabular data. They aim to create a single, massive model that can look at any small tabular dataset and instantly predict the missing values without requiring a single step of gradient descent.

Think of It Like This

Imagine a master detective. Standard ML (XGBoost) is like a student who has never solved a crime before. You have to give them 10,000 solved cases (training data) so they can slowly learn how to find patterns. A Tabular Foundation Model is the master detective. They have already solved millions of wildly different cases. If you hand them just 50 solved cases from a brand new city (the context), they instantly understand the pattern and can solve the 51st case immediately, zero-shot.

How It Actually Works

The breakthrough model in this space is TabPFN (Prior-Data Fitted Network), developed by researchers at the University of Freiburg (2022).

1. Generating the Training Data

You cannot train a tabular foundation model on "the internet" because public tabular datasets are rare, messy, and all have different numbers of columns. Instead, TabPFN is trained entirely on synthetic data. The researchers created a mathematical machine (a Bayesian prior) that randomly generates millions of synthetic datasets. Some datasets are linear, some are highly non-linear, some have 2 classes, some have 10.

2. Meta-Learning the Transformer

The researchers feed these synthetic datasets into a Transformer. The input to the Transformer is a set of training rows (e.g., $x_1, y_1, x_2, y_2 \dots$ ). The target for the Transformer is to predict the label for a test row ( $y_{test}$ ) given only its features ( $x_{test}$ ). The Transformer is trained via gradient descent to be the world's best algorithm at looking at a small table and guessing the pattern.

3. Zero-Shot Inference

Once trained, the weights of TabPFN are frozen forever. When you give it your real, real-world Excel file (e.g., predicting customer churn), TabPFN does not "train". It simply takes your training rows as the context window (just like a prompt in ChatGPT), and uses standard Transformer self-attention to output the predictions for your test rows in a single forward pass. It takes 1 second, requires zero hyperparameter tuning, and on datasets with fewer than 1,000 rows, it routinely beats finely-tuned XGBoost models.

Show Me the Code

Using TabPFN is almost identical to using scikit-learn, but notice that fit doesn't actually run gradient descent—it just loads your data into memory to be used as context.

1import pandas as pd2from sklearn.model_selection import train_test_split3from sklearn.metrics import accuracy_score4from tabpfn import TabPFNClassifier5
6# X: A standard pandas DataFrame (Must have < 100 features)7# y: Target variable (Classification)8X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)9
10# 1. Initialize the pre-trained Foundation Model11# We do not need a GPU for inference, and there are NO hyperparameters to tune!12classifier = TabPFNClassifier(device='cpu', N_ensemble_configurations=4)13
14# 2. "Fit" the model15# TabPFN doesn't train. It just stores X_train and y_train in memory16# to use them as the "Prompt" during the forward pass.17classifier.fit(X_train, y_train)18
19# 3. Predict the test set20# This runs a single forward pass of the Transformer.21# It attends to X_test, looks back at X_train/y_train, and outputs the answer.22predictions = classifier.predict(X_test)23
24print(f"TabPFN Accuracy (0 seconds of training): {accuracy_score(y_test, predictions):.3f}")

Watch Out For

Scale Limits. Because TabPFN uses Transformer self-attention across the rows of the dataset, it suffers from the quadratic context window limit. You cannot feed it a table with 1 million rows; it will run out of memory. As of today, Tabular Foundation Models only work on "small" tabular data (e.g., fewer than 10,000 rows and 100 columns). For massive enterprise datasets, XGBoost remains undefeated.

The Quick Version

Training tabular models from scratch is slow and requires hyperparameter tuning.
Tabular Foundation Models (like TabPFN) are pre-trained Transformers that can do In-Context Learning on tables.
They are trained on millions of randomly generated synthetic datasets.
During inference, you feed the model your training data as a "prompt," and it outputs predictions for your test data in a single, instant forward pass without any gradient descent.

Related concepts

Zero-Shot Prompting