Neural Networks
Perceptron & Activation Functions
Push a value through a single neuron and watch each stage light up. Then drag along the activation curve until its derivative collapses to nothing.
A neuron is a weighted sum followed by one squashing function — and the slope of that function, not the function itself, is what decides whether learning can happen at all.
Stage 1 of 5: In and out
Weighted sum 3.00, Output 0.9526
- Input value
- Activation output
- Slope of the activation
- Saturated — no gradient flows
The first value entering the neuron.
The second value entering the neuron.
Two inputs, one output. Nothing between them yet. Drag the input values and watch the number arrive.
A neural network is, at its core, built of very simple mathematical operations. Before we can talk about deep layers, backpropagation, or emergent behaviour, we have to look closely at the fundamental building block: the artificial neuron, or perceptron.
In this lab, we strip everything back to a single unit. You control two incoming values, the weight applied to each, and a bias term. The neuron simply multiplies each input by its weight, adds them together with the bias, and pushes the result through an activation function.
Why an activation function?
Without an activation function, a neural network would just be a series of linear transformations. No matter how many layers you stack, a composition of linear functions is always just another linear function. The network would only ever be able to draw straight lines, rendering depth entirely useless.
The activation function introduces non-linearity. It allows the network to bend and warp the space, giving it the capacity to learn complex, curving boundaries and intricate representations.
But this non-linearity comes with a hidden cost, and it's a cost that stalled the progress of deep learning for decades.
The vanishing gradient
During training, the network learns by adjusting its weights based on the error of its predictions. To know how to adjust a weight, the network uses calculus—specifically, the chain rule—to trace the error backward through the layers. This requires taking the derivative (slope) of the activation function.
If you push the weighted sum (z) too high or too low into a sigmoid or tanh function, the output flattens out. The slope approaches zero. When the slope is zero, the gradient becomes zero, and the error signal cannot travel back through that neuron. The network stops learning. This is the vanishing gradient problem.
Play with the controls above. Switch between Sigmoid, Tanh, and ReLU. Watch what happens to the slope as you push the inputs to their extremes. When the slope flatlines, the neuron is effectively dead to the learning algorithm.
Reference
- Weighted sum
- z = w₁x₁ + w₂x₂ + b
- Sigmoid
- σ(z) = 1 / (1 + e⁻ᶻ), σ′(z) = σ(z)·(1 − σ(z)) — peaks at 0.25
- Tanh
- tanh(z), derivative 1 − tanh²(z) — peaks at 1, and zero-centred
- ReLU
- max(0, z), derivative 1 for z > 0 and 0 for z < 0
- Saturation
- σ′ → 0 as |z| grows, so the gradient vanishes and learning halts
- Dead ReLU
- a unit stuck at z < 0 outputs 0 with derivative 0 — permanently
- Why non-linear
- composing linear maps gives a linear map, so depth would buy nothing
Break it on purpose
Drive the weighted sum far from zero with a sigmoid and the derivative falls to nearly zero: the neuron is saturated, and no gradient can flow back through it. Switch to ReLU and push the sum negative instead — the output and the derivative are both exactly zero, and the unit is dead for good.