Core ML

Temperature Scaling

Deep neural networks are naturally overconfident. Temperature scaling mathematically softens their probabilities, making a '99% confident' model actually right 99% of the time.

Why Does This Exist?

In the real world, probability means something specific. If a weatherman says there is a 90% chance of rain, and you track him for 100 days where he made that exact prediction, it should actually rain on exactly 90 of those days. If it only rains on 60 of those days, the weatherman is dangerously overconfident. We say he is poorly calibrated.

Modern deep neural networks (like ResNet or Transformers) are incredibly accurate, but they are notoriously poor at calibration. If a modern neural network says it is 99% confident that a picture is a dog, it is usually only right about 80% of the time.

This happens because networks are trained using Cross-Entropy Loss, which actively encourages them to push their probabilities as close to 100% as possible. Temperature Scaling is the simplest, most effective post-processing technique to fix this overconfidence, bringing the model's output probabilities back in line with reality.

Think of It Like This

Think of a neural network like an overly arrogant teenager taking a multiple-choice test.

Even if the teenager is only slightly leaning toward answer A over answer B, their arrogance causes them to yell, "I am 100% certain it is A!"

Temperature Scaling is like a humility filter. You apply the filter to the teenager's voice. The filter doesn't change the fact that they picked answer A, but it forces them to say, "I am leaning toward A, maybe 60% sure." The filter correctly aligns their spoken confidence with their actual knowledge.

How It Actually Works

To understand Temperature Scaling, you must look at the math immediately before a model outputs a probability.

1. Logits and Softmax

The final layer of a neural network outputs raw numbers called logits. Let's say a network is deciding between Dog and Cat, and outputs logits: [5.0, 0.0].

To turn these raw numbers into probabilities, they are passed through the Softmax function. Softmax exaggerates differences. Even though 5.0 and 0.0 aren't astronomically far apart, Softmax turns them into [99.3%, 0.7%]. The model becomes instantly overconfident.

2. The Temperature Parameter (T)

Temperature Scaling introduces a single scalar value, $T$ (Temperature). Before passing the logits into the Softmax function, you divide all the logits by $T$ .

If $T = 1$ , nothing changes.
If $T = 2$ , you divide the logits by 2. The new logits are [2.5, 0.0].

3. Softening the Output

When you pass [2.5, 0.0] through the Softmax function, the output is [92.4%, 7.6%].

The model still predicts "Dog" (the top choice didn't change), but the probability has been softened. The model is now acting appropriately humble.

4. Finding the Perfect Temperature

How do you know if $T$ should be 1.5, 2.0, or 3.7? You find the perfect temperature by using a holdout validation set. You run an optimizer (like L-BFGS) to find the exact value of $T$ that minimizes the Expected Calibration Error (ECE) on that holdout set.

Show Me the Code

Because Temperature Scaling only requires learning a single number ( $T$ ), it takes almost zero compute and can be applied to any pre-trained network in seconds.

1import torch2from torch import nn, optim3
4class ModelWithTemperature(nn.Module):5    def __init__(self, model):6        super().__init__()7        self.model = model8        # Initialize T (temperature) to 1.5 as a starting point9        self.temperature = nn.Parameter(torch.ones(1) * 1.5)10
11    def forward(self, input):12        # 1. Get the raw logits from the base model13        logits = self.model(input)14        15        # 2. Divide logits by T16        return logits / self.temperature17
18def calibrate_temperature(model_with_temp, val_loader):19    """20    Optimizes the temperature parameter T on a validation set.21    """22    # Standard Cross-Entropy Loss evaluates calibration well23    nll_criterion = nn.CrossEntropyLoss()24    25    # We ONLY train the temperature parameter, not the model!26    optimizer = optim.LBFGS([model_with_temp.temperature], lr=0.01)27
28    def eval():29        optimizer.zero_grad()30        loss = 031        # Calculate loss over the validation set32        for inputs, labels in val_loader:33            logits = model_with_temp(inputs)34            loss += nll_criterion(logits, labels)35        loss.backward()36        return loss37
38    # Run the optimizer to find the perfect T39    optimizer.step(eval)40    print(f"Optimal Temperature (T): {model_with_temp.temperature.item():.2f}")

Watch Out For

It Does Not Fix Epistemic Ignorance

Temperature scaling is a global fix. If you set $T=2.0$ , it softens every prediction the model makes. It cannot tell the difference between aleatoric-vs-epistemic-uncertainty. If you show the model a picture of a car (which it has never seen), it will still confidently output "Dog (92%)". Temperature scaling only fixes baseline overconfidence on data the model actually knows; it is useless for Out-of-Distribution (OOD) detection.

The Quick Version

Deep learning models are inherently overconfident; a 99% probability often means the model is only 80% accurate.
Temperature Scaling is a post-processing technique that fixes this without retraining the model.
It works by taking the raw output numbers (logits) and dividing them by a Temperature value ( $T$ ) before converting them to probabilities.
This mathematically softens the probabilities, aligning the model's confidence with its actual accuracy.
While it perfectly calibrates the model on known data, it does not help the model recognize unknown (Out-of-Distribution) data.