SwiGLU

Why Does This Exist?

A feed-forward network inside a transformer block is small on purpose: expand a token's representation, run it through a nonlinearity, project it back down. For years that nonlinearity was ReLU, then GELU — a single activation function applied elementwise to one projection, no exceptions. It worked, but "one projection, one squashing function" is a narrow way to transform information. The network gets to decide how much to squash a value, but it never gets to decide, per dimension, whether that value should pass at all based on some other signal computed from the same input.

Gating fixes that narrowness. Instead of one projection running through an activation, you compute two projections from the same input and multiply them together, elementwise, so one acts as a learned, per-dimension valve on the other. That idea predates transformers by years — LSTMs are built entirely out of gates — but applying it inside a feed-forward block was a smaller, later step. SwiGLU is that step: keep the expand-then-contract shape of a normal FFN, but make the expansion gated instead of plain. Empirically, it beats plain ReLU and GELU FFNs at the same parameter count by a small but consistent margin, and "small but consistent margin, at no real extra cost" is exactly the kind of change that spreads through the field fast. Most large language models built after roughly 2022 use it.

Think of It Like This

How It Actually Works

The two branches

SwiGLU projects the same input $x$ through three separate learned weight matrices, $W_1$, $W_2$, $W_3$, and combines them as:

$xW_1$ and $xW_3$ are both linear projections of the identical input $x$, into the same expanded dimension — the diagram's two parallel boxes. $\text{Swish}(xW_1)$ applies the Swish activation, $\text{swish}(z) = z \cdot \sigma(z)$, to the first branch only; the second branch, $xW_3$, stays a plain linear projection with no nonlinearity of its own.

The gate

$\odot$ is elementwise multiplication. Each dimension of $\text{Swish}(xW_1)$ multiplies the matching dimension of $xW_3$ — the "gate", output by the Swish branch, scales the "value" carried by the linear branch, dimension by dimension. A dimension where the gate is near zero effectively shuts that value off; a dimension where the gate is near one lets it through mostly unchanged. Crucially, the gate itself is learned and input-dependent — it isn't a fixed function like plain ReLU, it's a whole separate projection of $x$ that the network trains to decide what matters.

Back down to model width

The gated product still lives in the expanded dimension, so $W_2$ projects it back down to the model's working width — exactly the contraction step an ordinary FFN ends with. Nothing about the surrounding transformer architecture changes: SwiGLU drops into the same slot a ReLU or GELU FFN occupied, same input shape in, same output shape out. Only the internal computation between those two shapes gets richer.

Why Swish, specifically

Swish is smooth and non-monotonic near zero — unlike ReLU, which is flat at exactly zero for any negative input, Swish dips slightly negative before rising, which keeps a small, non-zero gradient flowing even for inputs a plain ReLU would kill outright. That smoothness is why the gate uses Swish rather than a harder function: a gate that can vary continuously, rather than snapping fully open or fully shut, gives training a smoother surface to optimize over.

Show Me the Code

A minimal SwiGLU feed-forward block: two parallel projections, one gated, then a contraction back to the input width.

1import numpy as np2
3
4def swish(x: np.ndarray) -> np.ndarray:5    return x / (1.0 + np.exp(-x))6
7
8def swiglu_ffn(9    x: np.ndarray, w1: np.ndarray, w2: np.ndarray, w3: np.ndarray10) -> np.ndarray:11    gate = swish(x @ w1)      # (n, d_ff) — the Swish-activated branch12    value = x @ w3             # (n, d_ff) — the plain linear branch13    return (gate * value) @ w2  # elementwise gate, then project back to d_model14
15
16rng = np.random.default_rng(0)17x = rng.normal(size=(2, 8))                        # 2 tokens, model width 818w1 = rng.normal(scale=0.3, size=(8, 16))            # expand to d_ff = 1619w3 = rng.normal(scale=0.3, size=(8, 16))20w2 = rng.normal(scale=0.3, size=(16, 8))            # contract back to d_model = 821out = swiglu_ffn(x, w1, w2, w3)22print(out.shape)              # -> (2, 8) — same shape as the input, as any FFN preserves23print(round(float(out[0, 0]), 4))  # -> -0.1005

Two independent projections go in at width 16, multiply elementwise, then collapse back to width 8 — the shape never leaves the ordinary FFN contract.

Watch Out For

The Quick Version

• SwiGLU replaces a plain FFN activation with a gated one: two projections of the same input, one run through Swish, multiplied elementwise.
• The formula is $(\text{Swish}(xW_1) \odot xW_3)W_2$ — expand twice, gate, contract once.
• The gate is learned and input-dependent, unlike a fixed activation like ReLU or GELU.
• SwiGLU needs three weight matrices instead of a plain FFN's two, which models usually offset with a smaller expansion width.
• It drops into the same input-shape-in, output-shape-out slot as any other FFN variant, so nothing else in the block changes.

What to Read Next

• Feed-Forward Networks is the plain expand-activate-contract block SwiGLU replaces the middle step of.
• Activation Functions covers Swish and the other nonlinearities SwiGLU's gate branch builds on.
• Transformer Architecture is where this FFN sub-layer sits inside the larger block.
• Encoder-Decoder Architectures covers the model variants that stack blocks using FFNs like this one.