Vision Explainability Overview

Explanations answer which pixels the model used. Different methods answer at different resolutions, and the honest ones state their resolution upfront.

On the 2 by 2 toy image the coarse Grad-CAM map keeps 0.50 of its mass inside the box while pixel saliency keeps 0.33, so resolution decides the method

Why Does This Exist?

A classifier that says dog with 0.97 confidence could be reading the dog, the grass, or the watermark in the corner. Operators, auditors, and debuggers all need to know which, but no single heatmap serves all three: a coarse region map guides a human glance, while a pixel map debugs a spurious texture. Picking a method without naming its resolution produces confident-looking answers to different questions.

This page maps the vision attribution family and states what each rung assumes. Grad-CAM weights late feature maps for coarse regions. Saliency maps take input gradients for pixel detail. Integrated gradients average along a baseline path to fix saturation. Attention rollout aggregates transformer layers. General background on what explanations can and cannot promise lives in model explainability, with the surrogate-sampling rung in LIME and the game-theoretic rung in SHAP.

Think of It Like This

A stadium usher describing a seat

Ask where your friend sits and the usher can point at the whole stand (coarse, always right), walk you to the row (medium effort, usually right), or read the exact seat number off a blurry ticket (precise, sometimes wrong). Each answer has a resolution and a failure mode, and you would never compare the three answers without saying which kind each one is.

Attribution methods are those three ushers. Grad-CAM points at the stand. Saliency reads the ticket. Integrated gradients re-read it under better light. Where it stops: ushers rarely lie, while heatmaps can look plausible and still misreport the model, so every map below must pass a sanity check before it counts as an explanation.

How It Actually Works

The ladder and its prices, briefly. Grad-CAM pools class gradients over late channels and upsamples, so maps are coarse (a 7 by 7 grid stretched over the image) but stable. Saliency differentiates the score with respect to pixels, so maps are full resolution but noisy, and they saturate when confidence is extreme. Integrated gradients integrate along a baseline path with 20 to 50 steps, costing that many backward passes. Rollout multiplies layer attentions with residual mixing. Use the coarse rung for human triage and the pathed rungs for debugging saturation, and always state which rung a figure comes from.

Worked example

Take the $2 \times 2$ toy image $x = [[3, 1], [0, 2]]$ with a linear dog score $s = 2x_{00} + x_{01} + 0x_{10} + 3x_{11} = 6 + 1 + 0 + 6 = 13$ . Saliency is the gradient magnitude $[[2, 1], [0, 3]]$ with total mass $6$ . Draw the box around the left column: saliency mass inside is $2 + 0 = 2$ , a fraction of $2/6 = 0.33$ . Grad-CAM with a single channel and pooled weight $w = (2 + 1 + 0 + 3)/4 = 1.5$ gives $1.5 \times x = [[4.5, 1.5], [0, 3]]$ with total mass $9.0$ ; box mass is $4.5 + 0 = 4.5$ , a fraction of $4.5/9.0 = 0.50$ . Same image, same box: the coarse map concentrates 0.50 inside while the pixel map spreads to 0.33, which is why the method choice, not the image, decides what the figure emphasizes.

Code

Stdlib Python replays the toy arithmetic: the score, the saliency box fraction, and the Grad-CAM box fraction.

1x = [[3, 1], [0, 2]]2w = [[2, 1], [0, 3]]3
4s = sum(w[r][c] * x[r][c] for r in range(2) for c in range(2))5print(s)6# -> 137print(round(sum(w[r][0] for r in range(2)) / sum(sum(row) for row in w), 2))8# -> 0.339
10cam = [[round(1.5 * x[r][c], 1) for c in range(2)] for r in range(2)]11print(cam)12# -> [[4.5, 1.5], [0.0, 3.0]]13print(round(sum(cam[r][0] for r in range(2)) / sum(sum(row) for row in cam), 2))14# -> 0.5

Score 13, saliency fraction 0.33, Grad-CAM map [[4.5, 1.5], [0.0, 3.0]] with fraction 0.5, exactly the prose numbers. On image classification outputs the same ladder applies: triage with the coarse map, debug with the pathed ones.

Watch Out For

Presenting a plausible map as a faithful one

A smooth heatmap over the dog looks right even when the model used the background. The symptom is a beautiful figure with no sanity check attached. Randomize the late weights and re-run: if the map barely moves, it explains the image prior, not your model.

Declaring one method better from a single image

One picture where Grad-CAM looks tighter than saliency proves nothing about the next picture. The symptom is a method choice argued from vibes instead of tests. Fix: compare methods over a labelled set with a fixed metric, such as box overlap at matched resolutions, and rerun after randomizing weights.

The Quick Version

Explanations state which pixels mattered and at what resolution.
Grad-CAM is coarse and stable; saliency is sharp and noisy; integrated gradients cost 20 to 50 backward passes.
Rollout aggregates transformer layers instead of differentiating pixels.
On the toy image the box fractions read 0.50 for Grad-CAM and 0.33 for saliency.
No sanity check means no explanation claim.