Multi-Layer Perceptron

What you are looking at

On the left, 150 points in two classes, and behind them the network's answer at every position in the plane — shaded toward whichever class it predicts, and paler where it is less sure. That shading is not a drawn curve. It is 900 separate forward passes, one per cell, so the boundary you see is wherever the answers happen to change. Nobody drew it.

On the right, the network itself: two inputs, the hidden layers you have asked for, and one sigmoid output. Edge thickness is the size of a weight and a dashed edge is a negative one, so you can watch the weights change as it trains. The caption counts the parameters, which is what you are spending when you turn up Hidden Layers or Neurons / Layer.

None of the three datasets can be split by a straight line. That is the only thing they have in common, and it is the point.

Why one layer is enough to bend the boundary

A layer does two things, and only the second one matters here:

$z = Wx + b, \qquad a = f(z)$

The first is a matrix multiply — a rotation, a stretch, a shear. The second is $f$, applied to each element on its own. Every interesting thing a network does comes from $f$, and here is why: matrix multiplies compose into matrix multiplies. Stack three of them and

$W_3(W_2(W_1x)) = (W_3W_2W_1)x$

which is one matrix. Put a non-linearity between them and no such collapse is available. There is nothing to multiply out.

Folding paper

A linear layer can slide the paper, turn it, and stretch it. Do that three times and you have still only slid, turned and stretched — you could have got there in one move.

A non-linearity folds it. Now the two halves are on top of each other and a single straight cut goes through both, in two places. Fold again and one cut becomes four. Depth is worth having because each fold multiplies what the next cut can do — but you have to actually fold, and a stack of slides and stretches never does.

Watch it happen

Press Play on the third stage. Each epoch is one full-batch gradient step: every point contributes its gradient, the average is taken, and every weight moves against it by the Learning Rate. At the default 0.5 the rings are separated within about forty epochs and you can watch the boundary bend as the loss falls.

Accuracy starts at 50%, which is the score for guessing on a balanced dataset — and it is worth noticing that the readout says 50% rather than 0%. An untrained network is not wrong about everything; it is right about half of it, by luck.

The moons are the slowest of the three, because they are nearly separable by a straight line — a linear model already gets 87.3% — so the last thirteen percent is the network learning a small awkward curve at the tips. Turn the learning rate up if you do not want to wait.

The setting that makes it all go away

The fourth stage unlocks Activation and adds one readout: Boundary Curvature. It measures how far the network's output is from any straight-line boundary, as a fraction of its own range — so 0 means exactly a line, and anything above 0 means the space has genuinely been warped.

Set Activation to Linear. On the rings:

Not approximately equal. The same, to the third decimal place, for five times the parameters. And 1e-15 is not a small curvature — it is the floating-point floor, the rounding error left over from multiplying matrices together. The boundary is not nearly straight. It is straight.

With the output sigmoid on the end, what you have built is exactly logistic regression on the original two features:

$\hat{y} = \sigma(W_{\text{eff}}x + b_{\text{eff}}), \qquad W_{\text{eff}} = W_L \cdots W_1$

Switch Activation back to ReLU and the same architecture reaches 100%.

Curvature 0 does not always mean a straight line

Set Activation to ReLU, Hidden Layers to 3, Neurons to 8, and push Learning Rate to 3. Accuracy drops to exactly 50% and Boundary Curvature reads 0.0000 — the same number the linear network reads. It is the opposite situation.

The steps have been large enough to push every ReLU unit's input below zero, and a ReLU below zero outputs zero and has a derivative of zero, so nothing flows forward and nothing flows back. The network now emits one constant value everywhere. Its boundary is not straight; there is no boundary at all, and it cannot recover, because the gradient that would fix it is also zero. This is the dying ReLU problem.

A single curvature number cannot tell "perfectly straight" from "nothing there", which is why the readout is flagged in the error colour when the second one happens, and why the reading behind it reports a shape — line, curve, or none — rather than only a magnitude.

The initialisation is not a detail

The initial weights are drawn from a Gaussian whose width depends on the activation: $\sqrt{2/n}$ for ReLU, $\sqrt{1/n}$ for tanh and linear, where $n$ is the number of inputs to the layer.

That factor of two is not decoration. He initialisation derives it specifically for ReLU, to compensate for the half of the signal ReLU sets to zero — so the variance coming out of a layer matches the variance going in. Apply the same $2/n$ to a unit that zeroes nothing and you double the variance at every layer instead of preserving it. Three layers means eight times the signal, gradients that compound to match, and training that diverges.

This lab is a good place to say so, because that exact mistake made this page's central demonstration impossible: with $2/n$ everywhere, a three-layer linear network blew up to NaN within nine epochs, which is not what "the boundary stays a straight line" is supposed to look like.

How little is enough

Three neurons in one hidden layer reach 100% on the rings — thirteen parameters. Two get to somewhere between 66% and 93% depending on the activation, and one to about 70%, and no amount of training moves either.

Then try those same three neurons as three layers of one. ReLU manages 50%; tanh manages 73%. Every signal in that network has to pass through a single number at each layer, and no later layer can recover what that projection threw away — the width of the narrowest layer caps what the whole network can express, and depth cannot buy it back.

Notice that the parameter count points the other way. Three layers of one has nine parameters; one layer of three has thirteen. The cheaper network is the much worse one, which is the useful part: parameter count is not capacity, and neither is depth.

One last thing worth knowing, because it is the sort of gap that catches people out: two hidden units are provably enough to represent XOR. Gradient descent from this initialisation does not find that solution. What a network can express and what your optimiser will actually discover are two different questions, and only the first one has a tidy theorem.