Visual explainer

Activation Functions

Follow sigmoid, ReLU and GELU from local slopes to a checked XOR construction, then separate activation effects from full-network gradients.

Affine layers alone cannot separate the four XOR points.

Nonlinear activations let hidden layers bend a network's decision boundary. Without them, stacked affine maps, each a weighted sum plus bias, collapse into one affine map. The four illustrative XOR inputs need outputs 0, 1, 1, 0; their crossing class diagonals prevent a single straight separator.

Sigmoid Bends the Line

Sigmoid has maximum local slope 0.25 at zero.

Sigmoid maps finite inputs strictly between 0 and 1: about 0.119 at −2, 0.5 at zero and 0.881 at 2. Its local slope is output × (1 − output), peaking at 0.25. Values near the tails change slowly; the function isn't flat everywhere outside zero.

ReLU Gates the Input

ReLU has local slope zero below zero and one above zero.

ReLU takes the larger of zero and its input. At −1 it returns zero with local derivative zero; at 2 it returns 2 with derivative one. These are activation derivatives, not guarantees about the full gradient. At the kink, the classical derivative is undefined.

GELU Rounds the Corner

Exact GELU is smooth and can output negative values.

Exact GELU multiplies the input by the standard normal cumulative probability at that input. It gives about −0.159 at −1, zero at zero and 0.841 at 1. Smoothness doesn't guarantee a nonzero derivative: its slope reaches zero near −0.752.

Bends Crack XOR

Two ReLU hidden units represent XOR with explicit weights.

Let s be the sum of the two inputs. Compute h₁ = ReLU(s), h₂ = ReLU(s − 1), then score = h₁ − 2h₂. Sums 0, 1, 2 produce scores 0, 1, 0. Predict one above 0.5. These chosen weights demonstrate representation, not a training result.

Change One Activation

Switching activations changes scores even with fixed weights.

Replace both ReLUs with exact GELU without retraining. The same sums now produce about 0.317, 0.841, 0.272. All four labels still match at threshold 0.5, but the scores differ. Activation choice changes the function; compare training behavior and held-out results instead of assuming an automatic repair.

Where It Breaks

Small local factors can shrink a backward signal.

In a scalar chain with unit interlayer weights, ten sigmoid slopes multiply to at most 0.25¹⁰, about 0.000000954. A negative ReLU blocks its local route exactly. Full gradient propagation also includes weights, other paths and the loss derivative; neither observation proves the entire network freezes.

The Quick Version

Affine layers alone cannot separate the four XOR points.
Sigmoid's maximum local slope is 0.25.
ReLU's local slope is zero below zero, one above.
Smooth GELU can output negatives and have zero slope.
Two constructed ReLU units represent all four XOR labels.
Switching activations changes scores even when weights stay fixed.
Full gradients depend on weights and paths, too.