Autoencoder (Bottleneck Compression)

What you are looking at

Sixteen 7×7 images. Each is a soft blob sitting somewhere on a ring, so 49 pixels per image but really only one thing varies between them: the angle.

An autoencoder squeezes each image through a narrow layer and tries to rebuild it. The encoder turns 49 numbers into a few; the decoder turns those few back into 49. Nothing else about the network matters here, because the bottleneck is the only place information can be lost.

What the bottleneck actually is

The useful way to think about the middle layer is not "a compressed copy of the image". It is a set of mixing weights.

The decoder holds a small collection of template images, shared across the whole dataset — every input gets rebuilt from the same templates. What the code says is only how much of each. A bottleneck of 4 means every one of the sixteen images has to be expressible as four numbers weighting four fixed templates.

That is why the code panel is a bar chart of signed values rather than a picture. A negative weight means subtract that template, which is as meaningful as adding it, and a magnitude-only view would throw that away.

Why this lab computes PCA

There is no gradient descent here, and that is deliberate rather than a shortcut.

A linear autoencoder trained to convergence on squared error provably spans the same subspace as principal component analysis — this is a theorem, from Baldi and Hornik in 1989, not an approximation. So the lab computes the optimum directly. Every number you see is the best a linear autoencoder could do, which means any loss on screen is a property of the bottleneck rather than of an undertrained network.

The trade is that you cannot watch it learn. What you get instead is that nothing here is random at all: no seed, no sampling, no initialisation. Every figure below is exact and will be identical on your machine.

The exchange rate

Drag Bottleneck size at the What Was Lost stage and watch Variance kept:

The returns fall off sharply, and they do so by construction: PCA sorts its components by how much variance they carry, so each one you add explains less than the last. A steep early curve like this is exactly the condition under which compression is worth doing — five numbers standing in for forty-nine, at 95% fidelity.

Two other things are worth noticing in that panel. The error map is per-pixel, and at a small bottleneck it shows a bright spot where the blob should be and a bright ring where the reconstruction put light that was never there — the decoder is not merely missing the blob, it is hedging across the whole ring. And the code's values arrive in pairs of similar size, as do the eigenvalues (0.62, 0.62, then 0.23, 0.23). That is the ring showing up in the maths: it takes two numbers, a sine and a cosine, to say where on a circle you are, so the templates come two at a time.

Break it, twice

Squeeze too hard. At 2 units the code keeps 67.7% and the rebuild stops being the image you picked — it smears along the ring, because two numbers cannot say where on the ring the blob sat. Nothing is broken and no number is invalid. This is what "lossy" means when you look at it directly.

Or give it capacity it cannot use. Push the bottleneck past 15 and Wasted units starts counting. Sixteen images, once centred, span at most fifteen directions — so there is no sixteenth direction of variance to project onto. The encoder writes exact zeros into those slots, the reconstruction stops changing entirely, and the code panel hatches the dead bars so they are distinguishable from ones that merely happen to be small.

That second one is worth sitting with. Capacity beyond the data's effective rank is not slightly wasteful, it is provably unusable — and the marker on the slider is computed from the dataset's own spectrum rather than written down.