Multi-Layer Perceptron

Why Does This Exist?

Two pressure mats in a warehouse doorway, one inside and one outside, and the alert should fire when exactly one is pressed. A single perceptron can't draw that boundary. The alert cases sit on opposite corners of a square with the quiet cases on the other diagonal, and no straight line splits that pattern. That's the wall the last page ended on.

Clearing it takes two extra units.

Give the first the weights $(1, -1)$ and the second $(-1, 1)$, then clip both at zero. Unit one is nonzero only when the inside mat is pressed and the outside one isn't; unit two only in the reverse case. Each detects a pattern — an asymmetry between the mats — rather than reading an input.

Now describe the four situations using only what those two units report. Both quiet gives $(0,0)$. Both pressed also gives $(0,0)$. Inside only gives $(1,0)$, outside only $(0,1)$. The alert cases now sit at two separate corners while both quiet cases have collapsed onto the origin, so a straight line splits them: alert when the two numbers sum above 0.5.

Nothing about the doorway changed. The coordinates did. That reframe is the page — a hidden layer is a learned change of representation, and the output layer is an ordinary linear model working in the space that change produced.

Think of It Like This

How It Actually Works

One layer, three operations

Multiply, shift, bend:

$h^{(\ell)}$ is the vector leaving layer $\ell$, $h^{(\ell-1)}$ is what arrived from below, $W^{(\ell)}$ is that layer's weight matrix, $b^{(\ell)}$ its bias vector, and $\phi$ (phi) is the nonlinearity, applied to each entry on its own. Chain them; the last output is the prediction.

Shapes are where people lose an afternoon, so here's one network in full. Tile 64 mats across the warehouse floor and sort each moment into one of four things: empty, a person walking, a pallet jack, a stack of boxes. 128 moments per batch.

• The input batch is 128 rows by 64 columns.
• $W^{(1)}$ is 64 by 32 with 32 biases, so layer one passes on 128 by 32.
• $W^{(2)}$ is 32 by 16, so layer two passes on 128 by 16.
• $W^{(3)}$ is 16 by 4, so the output is 128 by 4 — one score per class per moment.

The batch dimension rides through untouched, and every weight matrix reads the width below it and writes the width above it. That's the entire shape rule. Total parameters: 2,676.

The nonlinearity is load-bearing

Remove $\phi$ and stack two layers. You get $W^{(2)}(W^{(1)}x)$, and matrix multiplication is associative, so that's $(W^{(2)}W^{(1)})x$ — one matrix, one linear layer. Depth with nothing between the layers buys literally nothing; a hundred of them still draws one flat boundary. In the doorway network the product of those two weight matrices is the zero matrix, so the version without clipping answers "no alert" to all four cases forever. Which $\phi$ to pick is its own page.

What depth actually buys

One hidden layer, wide enough, gets within any error you name of any continuous function on a bounded closed region. Cybenko proved it for sigmoid-shaped units in 1989, Hornik generalised it in 1991, and it gets quoted as though it settles something.

It settles one thing: such a network exists. It says nothing about whether gradient descent finds it, how many units "wide enough" turns out to be — sometimes exponentially many — or how much data pins those parameters down.

Depth wins anyway, for a statable reason. Targets built in stages need exponentially fewer units in layers than flattened into one, which Telgarsky made precise in 2016. On the floor grid: layer one finds pressure edges, layer two assembles outlines, layer three names them. A single wide layer can't share that work, so it memorises every outline at every position and the unit count explodes.

Show Me the Code

Hand-set weights, so the only variable is the clipping.

1import numpy as np2
3X = np.array([[0.0, 0], [0, 1], [1, 0], [1, 1]])  # inside mat, outside mat4W1 = np.array([[1.0, -1.0], [-1.0, 1.0]])  # unit 1: inside only.  unit 2: outside only5W2 = np.array([[1.0], [1.0]])  # either asymmetry means somebody is standing there6
7
8def alerts(X: np.ndarray, nonlinear: bool) -> np.ndarray:9    H = X @ W110    if nonlinear:11        H = np.maximum(H, 0.0)  # drop this line and the two matrices merge into one12    return (H @ W2 - 0.5).ravel() > 013
14
15print("with ReLU   ", alerts(X, True).astype(int))16print("without ReLU", alerts(X, False).astype(int))17print("collapsed to", (W1 @ W2).ravel())18# -> with ReLU    [0 1 1 0]19# -> without ReLU [0 0 0 0]20# -> collapsed to [0. 0.]

Same parameters, same architecture. One line of clipping separates the pattern you wanted from a model that says nothing to anything.

Watch Out For

The Quick Version

• A layer is a matrix multiply, a bias, and a nonlinearity applied entry by entry. An MLP is several in sequence.
• The hidden layer builds coordinates where the answer is linearly separable, and the output layer is a linear model in that new space.
• Exclusive-or needs two hidden units, each detecting a pattern rather than reading an input.
• Without the nonlinearity, any stack of linear layers collapses into one matrix.
• Batch rides through untouched; each matrix reads the width below and writes the width above.
• Universal approximation promises existence. Not that the optimiser finds it, not that the width is affordable.

What to Read Next

• Backpropagation gives all 2,676 numbers a gradient in one pass.
• Activation Functions is the choice of $\phi$, as a chain of fixes.
• Gradient Descent moves the weights once gradients exist.
• Loss Functions sets what those four scores are measured against.
• Dropout is the first lever when a wide layer starts memorising.
• Definitions nearby: Matrix, Tensor, and Linear Combination.