Skip to content
AI360Xpert
Core ML
Visual explainer

Activation Functions

Why neural networks need non-linearity, and how Sigmoid, ReLU, and GELU each solve the previous function's critical failure.

Without a non-linear activation, a neuron is just a linear model, and the network collapses.
Without a non-linear activation, a neuron is just a linear model, and the network collapses.

Without an activation function bending the signal, stacking layers achieves nothing. Mathematically, a deep network composed purely of linear operations collapses into a single linear model.

Sigmoid and Vanishing Gradients

Sigmoid is smooth but saturates at both ends, causing gradients to vanish.
Sigmoid is smooth but saturates at both ends, causing gradients to vanish.

Sigmoid squashes outputs nicely between 0 and 1, but its slope flattens out at the edges. When multiplying gradients through many deep layers, these flat slopes cause the learning signal to vanish completely.

The ReLU Fix

ReLU is simple and fast, but negative inputs produce zero gradient, risking dead neurons.
ReLU is simple and fast, but negative inputs produce zero gradient, risking dead neurons.

ReLU fixes the vanishing gradient by maintaining a slope of exactly 1 for all positive inputs. However, anything below zero gets a slope of 0, meaning the neuron stops learning entirely—a dead neuron.

Smooth Gating with GELU

GELU and SwiGLU provide a smooth gating mechanism, preserving gradients near zero.
GELU and SwiGLU provide a smooth gating mechanism, preserving gradients near zero.

GELU and SwiGLU smooth out the sharp kink of ReLU. By providing a probabilistic or learned gating mechanism, they preserve gradients even for slightly negative inputs, making them the standard for large transformers.

Where It Breaks

Choosing the wrong activation function for the depth and task can stall training completely.
Choosing the wrong activation function for the depth and task can stall training completely.

If you use Sigmoid in a 50-layer network, or pick a learning rate that drives all ReLUs negative, training halts. The loss curves plateau early and stay there, completely stalled by dead neurons or vanishing gradients.

The Quick Version

  • Linear operations alone cannot build complex features; an activation function is required.
  • Sigmoid causes gradients to vanish in deep networks because it saturates at the extremes.
  • ReLU maintains a constant gradient for positive inputs but can create dead neurons.
  • GELU and SwiGLU smooth the transition, preserving gradients and improving training in transformers.
  • The wrong choice of activation can completely stall training.

What to Read Next