Core ML

Deep Ensembles

A single model will confidently lie to you if it doesn't know the answer. But if you ask five different models, they will all lie in completely different ways, exposing the fact that they are guessing.

Why Does This Exist?

In aleatoric-vs-epistemic-uncertainty, we learned that Epistemic uncertainty occurs when a model is given data it has never seen before (Out-of-Distribution, or OOD data).

The most dangerous flaw of modern neural networks is that they do not know when they are looking at OOD data. If you train a network exclusively on cats and dogs, and then show it a picture of an airplane, the network does not output "I don't know." Instead, it aggressively maps the airplane's geometry into its limited knowledge base and outputs something like "Dog (99.9% Confidence).".

Techniques like temperature-scaling cannot fix this. Temperature scaling only softens the probabilities; it would just change it to "Dog (85%)".

Deep Ensembles solve this problem entirely. By training multiple models and measuring how much they disagree with each other, we can accurately detect when the AI is completely guessing.

Think of It Like This

Imagine you are a detective interrogating a suspect. You ask the suspect: "Where were you on the night of the 12th?"

If you only ask the suspect once, they will confidently lie: "I was at the movies!" You have no way to know they are lying because they sound very confident.

Now, imagine the suspect has five clones. You put all five clones in separate interrogation rooms and ask them the same question. Because they are making up a lie on the spot, they will all guess differently. Clone 1 says "The movies." Clone 2 says "The park." Clone 3 says "Asleep." Their individual confidence doesn't matter anymore. The fact that they violently disagree with each other proves they have no idea what the true answer is.

How It Actually Works

A Deep Ensemble is shockingly simple to implement.

1. Training Multiple Models

You take your dataset and train 5 separate neural networks. The architecture and the data are exactly the same. The only difference is the Random Initialization. When a neural network starts training, its weights are randomized. Because the starting weights are different, each of the 5 models will find a slightly different mathematical path to solve the problem.

2. In-Distribution (Known Data)

When you show a picture of a Dog to the 5 models, they will all easily recognize it.

M1: Dog (95%)
M2: Dog (92%)
M3: Dog (98%)

Because the Variance across the predictions is practically zero, you can trust the prediction. Epistemic uncertainty is low.

3. Out-of-Distribution (Unknown Data)

When you show a picture of an Airplane to the 5 models, their math breaks down. Because each model learned a slightly different mathematical representation of the world, they will all hallucinate in completely different directions.

M1: Dog (99%)
M2: Bird (85%)
M3: Frog (96%)

When you calculate the Variance across these predictions, it will be massive. The system flags this high variance as high Epistemic Uncertainty, rejects the prediction, and alerts a human operator that the data is Out-of-Distribution.

Show Me the Code

In code, creating an ensemble just means running a for loop over a list of models and calculating the variance of their outputs.

1import numpy as np2
3# Assume we have trained 5 separate neural networks4ensemble = [model_1, model_2, model_3, model_4, model_5]5
6def predict_with_uncertainty(input_data):7    predictions = []8    9    # 1. Ask all 5 models for their prediction10    for model in ensemble:11        # Assuming the model outputs probability of class 112        prob = model.predict(input_data)13        predictions.append(prob)14        15    predictions = np.array(predictions)16    17    # 2. Calculate the Final Output (The Mean)18    mean_prediction = np.mean(predictions)19    20    # 3. Calculate the Epistemic Uncertainty (The Variance)21    epistemic_uncertainty = np.var(predictions)22    23    return mean_prediction, epistemic_uncertainty24
25# Example Output for Known Data:26# Mean: 0.94, Variance: 0.001 (Very safe to trust)27
28# Example Output for Unknown Data:29# Mean: 0.50, Variance: 0.45 (DO NOT TRUST. Models are disagreeing violently)

Watch Out For

The Compute Cost

The reason Deep Ensembles are not used everywhere is cost. If you have a massive LLM like Llama-3 that takes 10,000 GPUs to train, you cannot afford to train it 5 separate times just to get an ensemble. Similarly, running 5 models during inference takes 5x the memory and compute. For massive models, researchers use cheaper approximations of ensembles, like monte-carlo-dropout or LoRA ensembles.

The Quick Version

Single neural networks are dangerously overconfident on data they have never seen before (Out-of-Distribution data).
Deep Ensembles solve this by training 5 (or more) identical models with different random starting weights.
On known data, all 5 models will agree.
On unknown data, all 5 models will hallucinate in completely different ways.
By measuring the variance (disagreement) between the models, you can accurately quantify Epistemic (ignorance) uncertainty.
It is the gold standard for AI safety, but it comes at a massive 5x compute cost.