1x1 Convolution

A 1x1 convolution mixes channels at each pixel without touching neighbours, so it squeezes wide layers down, expands them back up, and adds depth cheaply.

A 1x1 squeeze from 256 to 64 channels cuts the block from 589,824 weights to 69,632 while adding nonlinearity.

Why Does This Exist?

Wide layers are expensive. A 3x3 convolution from 256 channels to 256 channels holds $3 \times 3 \times 256 \times 256 = 589{,}824$ weights, and stacking several of those burns compute fast. But much of that width is redundant: the layer mostly needs to recombine channel information, not re-read spatial patterns at full width.

The 1x1 convolution fixes this by separating channel mixing from spatial mixing. With kernel size 1 it reads no neighbours at all; at each pixel it applies one learned linear mix across channels, followed by a nonlinearity. Squeeze 256 channels to 64 with a 1x1, run the costly 3x3 at width 64, then expand back.

This page covers that bottleneck pattern: the per-pixel rule and the parameter arithmetic. If convolutions themselves are new, start with convolutional layers, which build the sliding-window picture this page assumes. The elementwise arithmetic underneath is in basic CNN operations. What comes after is the residual bottleneck that made this pattern standard, in residual networks.

Think of It Like This

A mixing desk before the effects pedal

A band runs 256 microphone lines into a mixing desk that blends them down to 64 stems, runs those stems through one expensive effects pedal, then expands back to a full mix. The pedal hears every ingredient but processes far fewer lines.

The 1x1 layer is the desk. The 3x3 that follows is the pedal. Where the desk stops blending is where the analogy stops: the 1x1 adds its own nonlinearity, so it computes rather than just routing.

How It Actually Works

The per-pixel rule

Let $X$ be the input feature map with $C_{in}$ channels over a grid of positions, $W$ the learned weights of shape $C_{out} \times C_{in}$ , and $b$ a bias with one value per output channel. At each position $(i, j)$ , output channel $c$ is:

Y[c, i, j] = b[c] + \sum_{k} W[c, k] \cdot X[k, i, j]

Here $k$ runs over the $C_{in}$ input channels, so each output pixel is a weighted mix of the input channels at that same pixel. No neighbours enter. Spatial size is unchanged, and the cost of the layer is $C_{in} \times C_{out}$ weights plus $C_{out}$ biases.

Worked example: the 8.5 times bottleneck

Take a block that processes 256 channels. Done directly, one 3x3 layer at full width costs $3 \times 3 \times 256 \times 256 = 589{,}824$ weights.

With a 64-channel bottleneck instead:

The squeeze 1x1 costs $256 \times 64 = 16{,}384$ .
The middle 3x3 at width 64 costs $3 \times 3 \times 64 \times 64 = 36{,}864$ .
The expand 1x1 costs $64 \times 256 = 16{,}384$ .

Total $16{,}384 + 36{,}864 + 16{,}384 = 69{,}632$ , about 8.5 times fewer than the direct $589{,}824$ , with two extra nonlinearities included. The diagram above draws exactly these five stages and these four numbers.

Code

This snippet recomputes the bottleneck arithmetic so you can check the claim on your own widths:

1def block_cost(c_in: int, c_mid: int, c_out: int) -> int:2    squeeze = c_in * c_mid  # 1x1 down-projection weights3    middle = 9 * c_mid * c_mid  # 3x3 at bottleneck width4    expand = c_mid * c_out  # 1x1 up-projection weights5    return squeeze + middle + expand6
7
8direct = 9 * 256 * 2569print(direct)10# -> 58982411
12print(block_cost(256, 64, 256))13# -> 6963214
15print(round(direct / block_cost(256, 64, 256), 1))16# -> 8.5

Watch Out For

Expecting a 1x1 to add spatial context

A 1x1 kernel reads exactly one pixel, so stacking only 1x1 layers never grows the receptive field. The symptom is a network that classifies textures but cannot use shape or layout. Pair every bottleneck with a spatial kernel that actually looks at neighbours.

Squeezing channels until information chokes

Projecting 512 channels down to 8 destroys distinctions no later layer can recover. The symptom is training loss that stalls early while a wider bottleneck trains fine. Keep the squeeze ratio near 4 to 1 as in Inception blocks, and widen before assuming the idea failed.

The Quick Version

A 1x1 convolution mixes channels at each pixel and reads no neighbours, leaving spatial size unchanged.
Squeezing 256 channels to 64 before a 3x3 cuts that block from 589,824 weights to 69,632 with extra nonlinearities.
Bottlenecks power Inception modules, ResNet blocks and pointwise steps of separable convolutions.
A 1x1 never grows the receptive field, so it always pairs with a spatial kernel.
Squeeze near 4 to 1; narrower projections choke information instead of compressing it.