Masked Autoencoder Vision

Hide three quarters of an image and force the net to paint the missing pixels. Guessing so much from so little teaches shape and context.

MAE hides three quarters of patches and rebuilds pixels from the rest

Why Does This Exist?

Language models learn by filling blanks, but vision blanks are easy to copy from neighbors. MAE raises the rate to $75$ percent masked, so copying fails and real structure must be learned. The encoder sees only $25$ percent, which also trains about three times faster. Later object detection and semantic segmentation fine-tunes start from that encoder.

This page covers pixel rebuilding only. A patch is a $16 \times 16$ square turned into one token, so mask ratio $r = 0.75$ hides $147$ of $196$ tokens for a $224$ photo, and the loss is mean squared error on masked pixels only.

Think of It Like This

A restorer with most of a fresco missing

A restorer gets a wall with three quarters chipped away. Small holes could be dabbed over, but huge gaps force study of poses, folds, and perspective across the whole wall. The rebuilt wall proves the study.

Visible patches are the chips that remain, the decoder is the restorer, and pixel error is the inspection. Where it stops: fresco gaps stay fixed, while MAE draws a fresh random mask each pass.

How It Actually Works

Split $x$ into $N = 196$ patches. Sample mask $M$ with $|M| = 0.75N$ . Encode only visible $x_{vis}$ with ViT $f$ . Pad with learned mask tokens and decode with light transformer $g$ to $\hat{x}$ . Pay $L = \sum_{i \in M} \lVert \hat{x}_i - x_i \rVert_2^2 / |M|$ on normalized pixels. In plain words: $f$ is the big reader, $g$ is the small painter, and $M$ is the hidden set scored each step.

Worked example

Take one masked patch with true normalized pixels $[0.5, -0.3, 0.8]$ and prediction $[0.4, -0.1, 0.6]$ . Errors are $0.5 - 0.4 = 0.1$ , $-0.3 - (-0.1) = -0.2$ , and $0.8 - 0.6 = 0.2$ . Squares are $0.01 + 0.04 + 0.04 = 0.09$ , and the mean is $0.09 / 3 = 0.03$ . Over $147$ masked patches the mean stays near $0.03$ early and falls below $0.01$ when context clicks. Only global shape lowers every patch at once.

Code

The point of this snippet is the masked loss: mean squared error over the fixture patch reproduces the hand-computed $0.03$ with only the standard library.

1true = [0.5, -0.3, 0.8]2pred = [0.4, -0.1, 0.6]3
4sq = [round((t - p) ** 2, 4) for t, p in zip(true, pred)]5mse = round(sum(sq) / len(sq), 4)6print(sq, mse)  # -> [0.01, 0.04, 0.04] 0.03

The output $0.03$ matches the hand computation, confirming the loss scores masked pixels only.

Watch Out For

Linear probing MAE and calling it weak

MAE features are built for fine-tuning, not frozen probes. Linear tests trail contrastive runs by many points. The symptom is poor frozen accuracy with top fine-tune scores. Fine-tune the backbone for pretrained models use instead of judging frozen numbers.

Masking too little and learning to copy

At $25$ percent masked the network interpolates neighbors instead of learning shape, and reconstructions look great while transfers lag. The symptom is low pixel loss with weak detection gains. Fix: keep the rate near $0.75$ so local copying cannot solve the task.

The Quick Version

MAE masks 75 percent of patches and encodes the visible 25 percent.
A light decoder rebuilds normalized pixels with mean squared error.
The fixture patch scores $0.01 + 0.04 + 0.04 = 0.09$ total, mean $0.03$ .
High masking stops copying from neighbors.
Features shine after fine-tuning, not frozen probing.