Skip to content
AI360Xpert
Core ML
Visual explainer

Multi-Layer Perceptron

Stacking layers of neurons lets a network build a hierarchy of features. See how depth, width, and non-linearity combine to solve complex problems.

Stacking layers allows each successive layer to extract higher-level features.
Stacking layers allows each successive layer to extract higher-level features.

Adding hidden layers between the input and output transforms a flat classifier into a deep network. Each successive layer builds upon the features extracted by the previous one.

The Forward Pass

During the forward pass, activations flow layer by layer from input to output.
During the forward pass, activations flow layer by layer from input to output.

Data flows strictly in one direction. Each layer calculates its weighted sums, applies its non-linear activation functions, and passes the resulting activations to the next layer.

The Need for Non-Linearity

Without non-linear activations, a deep network collapses mathematically into a single layer.
Without non-linear activations, a deep network collapses mathematically into a single layer.

If you stack linear layers without non-linear activations between them, the math collapses. The entire deep network is mathematically equivalent to a single linear transformation, meaning depth buys you nothing.

Depth vs Width

Depth buys a hierarchy of features, while width only buys raw capacity.
Depth buys a hierarchy of features, while width only buys raw capacity.

A wide, shallow network has the capacity to memorize patterns, but a deep network learns a hierarchy. Depth allows the network to compose simple features into complex abstractions.

Where It Breaks

Without proper initialization, activations either shrink to zero or explode, preventing deep MLPs from training.
Without proper initialization, activations either shrink to zero or explode, preventing deep MLPs from training.

As networks get deeper, the signals passing through them tend to either shrink away to nothing or explode into infinity. Without careful initialization, deep MLPs become mathematically untrainable.

The Quick Version

  • Stacking layers builds a hierarchy of abstract features.
  • Activations flow from input to output in a forward pass.
  • Non-linearity is required to stop the math from collapsing into a single layer.
  • Depth enables feature composition; width merely increases capacity.
  • Poor initialization breaks deep networks due to exploding or vanishing signals.

What to Read Next