Generative Adversarial Networks (GANs)

The asymmetry that makes a GAN strange

Two networks, and only one of them ever sees the data.

The discriminator gets both batches — 40 real samples and 40 fakes — and learns to tell them apart. Ordinary supervised learning.

The generator gets nothing. It is handed 40 Gaussian latent codes, turns them into 40 points, and is never told where the real clusters are, how many there are, or whether it is close. Everything it will ever learn about the data arrives as one vector per sample: $\partial L / \partial x$ at its own output, backpropagated through the critic.

That vector is the whole channel, and it is what this lab draws.

The critic's field

At the Discriminator stage the shaded background is a real forward pass at 400 grid positions — the critic's score for how real each point in the plane looks. Pale shading is a near coin flip; strong shading is confidence.

Press Step and watch it sharpen. It is doing the easy job: the fakes start in a blob well below both clusters, and separating them takes a couple of rounds.

The signal

At the Signal stage every second generated point sprouts an arrow. That arrow is the direction the critic's score rises fastest at that point — the direction the generator will move it, and the only instruction it receives.

Signal to G is the mean length of those arrows. Two things about it are worth watching:

It is not constant. Early on the arrows all point the same way, up towards the clusters, because the critic has learned little more than "real things are up there". As training proceeds they fan out and shorten, because the critic starts caring about which cluster and how far in.

It can be exactly zero. Not small — zero. The critic's four hidden units are ReLUs, and a ReLU that is off passes nothing back at all. When all four are inactive at a generated point, the chain of products contains a literal $0$, and that sample receives no instruction whatsoever. Its arrow does not shrink; it disappears.

What a tie looks like

At the Adversarial Loop stage both networks train, one round at a time, and the second panel shows both losses.

They do not go to zero. They converge on 0.693, which is $\ln 2$, which is $-\log(0.5)$ — the loss of a coin flip. That is the point of the whole construction: the objective

$$\min_G \max_D \; \mathbb{E}_{\text{real}}[\log D(x)] + \mathbb{E}_z[\log(1 - D(G(z)))]$$

is a saddle point, not a minimum. Neither player is trying to win. A discriminator loss near zero is not success, it is one player having run away with the game, and the quiz asks you about exactly that.

Two readouts grade the generator, and it takes both:

Modes covered is how many of the two clusters actually have samples on them — a diversity measure. Realism is the mean distance from each fake to the nearest real point. Neither works alone: a generator piled entirely onto the left cluster scores excellent realism and covers one mode, and an untrained one also covers one or zero modes but sits nowhere near the data. One number could not tell those two apart.

Expect a detour on the way. The generator usually overshoots past the clusters before settling — far enough that some samples leave the frame, which the scene shows by pulling them onto the edge and dimming them rather than letting them vanish.

Break it: starve the generator

Algorithm 1 of the original GAN paper runs $k$ discriminator steps for every one generator step, and sets $k = 1$. That looks needlessly timid. The critic is the easier network to train, and letting it get good first sounds like it ought to help.

The Discriminator steps per round slider is that $k$. Take it up.

At 12 steps per round the generator never populates the second cluster — not on any of the eight seeds available through Reseed. Signal to G falls to 0.0000 and the arrows go out. The generator is not losing the game; it has been disconnected from it. A critic that is already certain about every fake has no gradient left to hand back, and a teacher who is always right teaches nothing.

The boundary is sharper than it looks. At 1 and 2 steps every seed ends up covering both clusters. At 3, one of the eight never does — which is worse than a clean failure, because it looks like luck. That is the honest reason the paper picked 1.