Core ML

Integrated Gradients

Basic saliency maps fail when a network is extremely confident (gradient saturation). Integrated Gradients fixes this by calculating gradients along a linear path from a blank image to the actual image.

Integrated Gradients accumulates gradients across a linear path from a baseline (blank) image to the actual input, bypassing saturated zones.

Why Does This Exist?

In the pursuit of explaining neural networks, saliency-maps are the most intuitive approach: just calculate the gradient of the prediction with respect to the input pixels.

However, basic saliency maps suffer from a mathematical flaw called gradient saturation. If a network is incredibly confident in its prediction, it stops learning. If a picture of a fire truck is so obviously a fire truck that the model outputs a 99.999% probability, tweaking a red pixel slightly will not change that probability. Because the probability doesn't change, the mathematical gradient is exactly zero.

When you ask the basic saliency map, "Which pixels drove this prediction?", it will look at the zero gradients and incorrectly highlight nothing. The model is essentially screaming, "I am so confident this is a fire truck that I don't care about the pixels anymore!"

Integrated Gradients (IG) was invented to solve this exact problem. It is mathematically proven to satisfy several key axioms of fair attribution, making it one of the most rigorous explainability tools in deep learning.

Think of It Like This

Think of gradient saturation like trying to measure the importance of water to a houseplant.

If you have a perfectly healthy, fully watered houseplant (the saturated model), adding one extra drop of water (the gradient) will not make the plant any healthier. If you only look at that single drop, you might incorrectly conclude that water is useless to plants.

Integrated Gradients solves this by starting with a completely dead, unwatered plant (the baseline). It slowly adds water, drop by drop, tracking the plant's health improvement at every single step, until the plant is fully healthy. By summing up the improvements across the entire journey, it perfectly measures the true importance of the water.

How It Actually Works

Integrated Gradients works by calculating gradients not just on the final image, but on a series of intermediate images.

1. Define the Baseline

You must choose a baseline input that represents a "neutral" state with zero information. For computer vision, this is typically a solid black image (all pixel values = 0).

2. Interpolate the Path

You create a sequence of images (usually between 20 and 50) that slowly transition from the baseline (black) to the actual input image.

Image 1 is 100% black.
Image 10 is 50% the original image, 50% black (it looks like a heavily darkened photo).
Image 20 is the full, original image.

3. Calculate and Accumulate Gradients

You pass all 20 of these images through the neural network and calculate the basic pixel gradients for each one.

In the heavily darkened images, the network is not confident at all. Therefore, the gradients are massive and highly informative. As the image gets closer to the original, the network becomes confident, and the gradients drop to zero (saturation).

By mathematically integrating (summing up) the gradients across all 20 steps, you capture the rich, informative gradients from the early steps and bypass the saturated zero-gradients of the final step.

4. The Completeness Axiom

Because of the math involved, Integrated Gradients guarantees a property called Completeness. The sum of the Integrated Gradients across all pixels will exactly equal the difference between the model's prediction on the original image and the model's prediction on the baseline image. This makes it mathematically rigorous in the exact same way that shap is rigorous.

Show Me the Code

Integrated Gradients requires looping over the interpolated images and accumulating the gradients. Libraries like Captum (built by Meta) handle this natively in PyTorch.

1import torch2from captum.attr import IntegratedGradients3
4def generate_integrated_gradients(5    model: torch.nn.Module, 6    input_image: torch.Tensor, 7    target_class_idx: int8) -> torch.Tensor:9    """10    Uses Captum to generate an Integrated Gradients heatmap.11    """12    model.eval()13    14    # 1. Initialize the IG algorithm with the model15    ig = IntegratedGradients(model)16    17    # 2. Define the baseline (a completely black tensor of the same shape)18    baseline = torch.zeros_like(input_image)19    20    # 3. Calculate the attributions21    # n_steps=50 means it will generate 50 intermediate images 22    # between the baseline and the input.23    attributions, delta = ig.attribute(24        inputs=input_image,25        baselines=baseline,26        target=target_class_idx,27        n_steps=50,28        return_convergence_delta=True29    )30    31    # 4. Collapse RGB channels to create a 2D heatmap32    heatmap = torch.sum(torch.abs(attributions), dim=1).squeeze()33    34    return heatmap35
36# The resulting heatmap is vastly superior and less noisy 37# than a basic saliency map.

Watch Out For

Choosing the Wrong Baseline

Integrated Gradients is completely dependent on the baseline you choose. A black image is standard for computer vision, but what if the object you are trying to detect is a black cat at night? Interpolating from black to black will yield zero gradients. In specific domains (like text or medical imaging), choosing the mathematically "neutral" baseline is a complex research problem.

The Quick Version

Basic saliency maps fail when a model is highly confident, because the gradients drop to zero (gradient saturation).
Integrated Gradients solves this by calculating gradients along a stepped path from a neutral baseline (e.g., a black image) to the final image.
Because the model is not confident on the darkened, intermediate images, the gradients are rich and informative.
By summing up the gradients across the entire path, IG bypasses saturation and provides a highly accurate, less noisy heatmap.
It is mathematically rigorous and satisfies the Completeness axiom, meaning it accounts for 100% of the prediction's deviation from the baseline.