Core ML

Time Series Foundation Models

Massive, pre-trained neural networks that have already 'read' millions of time series. You can use them to forecast your own data without training them from scratch (Zero-Shot forecasting).

Foundation models are pre-trained on diverse global data. They can forecast a completely unseen dataset (Zero-Shot) by relying on universal patterns they learned during pre-training.

Why Does This Exist?

If you want to build a sentiment analysis classifier today, you don't train a neural network from scratch on your own data. You take a Foundation Model (like GPT-4 or Llama) that has already been pre-trained on the entire internet, and you use it out-of-the-box. This is called Zero-Shot Learning.

Until recently, this did not exist for Time Series. Because time series data doesn't share a common "vocabulary" like the English language (sales volume is fundamentally different from server CPU usage), researchers assumed a universal time series model was impossible.

In 2023 and 2024, researchers from Amazon (Chronos) and Google (TimeSFM) proved it is possible. They built Time Series Foundation Models that can forecast your data perfectly without ever being trained on it.

Think of It Like This

Imagine a world-class financial analyst.

If you give them a chart of a company's stock they have never seen before, they can still make a highly educated guess about where it will go. Even though they don't know the specific company, they understand universal concepts like "trends," "momentum," "support/resistance," and "mean reversion" because they have looked at thousands of other stocks in their lifetime.

A Time Series Foundation model is that analyst. It has looked at millions of public datasets (weather, traffic, finance) during pre-training, so it already understands the universal laws of time.

How It Actually Works

The biggest hurdle in building a time series foundation model is the lack of a shared vocabulary. An LLM reads English words from a fixed dictionary. A time series is just an infinite spectrum of floating-point numbers.

The Tokenization Trick (Chronos by Amazon)

To solve this, Amazon's Chronos treats time series data exactly like a language problem.

Scaling: It takes the raw time series and normalizes it so the values sit in a standard range.
Quantization: It divides the Y-axis into 4096 discrete "bins" (like slots). If a number falls into Bin 42, it is assigned the token <BIN_42>.
LLM Pre-training: The time series is now literally a sentence of tokens: "<BIN_42> <BIN_45> <BIN_40>". They train a standard language model (based on the T5 architecture) to predict the next token in the sentence.

The Patching Trick (TimeSFM by Google)

Instead of converting numbers to words, Google's TimeSFM treats the time series like an image. It chops the continuous time series into small chunks called "Patches" (e.g., 32 data points per patch). It feeds these patches into a Transformer network, allowing the model to look at the macroscopic shape of the data rather than obsessing over individual data points.

Show Me the Code

Using a foundation model is completely different from classical ML. Notice that there is no model.fit() or training step! We just load the pre-trained weights and immediately ask for a forecast.

1# Using Amazon's Chronos library2import pandas as pd3import torch4from chronos import ChronosPipeline5
6# Load the pre-trained 'small' model from HuggingFace7pipeline = ChronosPipeline.from_pretrained(8    "amazon/chronos-t5-small",9    device_map="cuda",10    torch_dtype=torch.bfloat16,11)12
13# Load a completely unseen dataset14# e.g., Monthly sunspot observations15df = pd.read_csv("sunspots.csv")16context = torch.tensor(df["sunspots"].values)17
18# Zero-Shot Forecast: Predict the next 24 months19# We don't train it. We just pass the context and ask for 24 steps.20forecast = pipeline.predict(21    context,22    prediction_length=24,23    num_samples=20 # For probabilistic bounds24)25
26print("Forecast Complete!")

Watch Out For

Covariates (External Features) are usually not supported. The biggest drawback of first-generation Foundation Models is that they are strictly univariate. You can give them past sales, but you cannot easily pass in "Price", "Marketing Spend", or "Holidays". If your forecast heavily depends on external features, you are still better off training an XGBoost or Temporal Fusion Transformer model from scratch.

The Quick Version

Time Series Foundation Models are massive neural networks pre-trained on millions of diverse, public datasets.
They allow for Zero-Shot Forecasting: You can use them to forecast your own data without training them.
Amazon's Chronos works by converting continuous numbers into discrete "tokens" and treating forecasting as a language translation problem.
Google's TimeSFM works by chopping the data into "patches" and processing them visually.
They are incredibly powerful baselines, but currently struggle to incorporate external variables like price or weather.

Related concepts

Probabilistic Forecasting