Core ML

ALiBi Attention Bias

ALiBi skips positional embeddings entirely and instead subtracts a penalty proportional to distance straight from the attention scores, before softmax runs.

A penalty that grows linearly with the distance between query position i and key position j is subtracted from the raw attention score before softmax, with no penalty on the diagonal where i equals j

Why Does This Exist?

Positional encoding and rotary embeddings both give a model position by changing what a token's query or key vector is, before scoring. That fix works well within trained range, but both schemes still hit the same wall past it: positions the model never saw during training produce angles or encodings it never learned to use well, and quality drops off past the trained sequence length even though nothing technically breaks.

ALiBi asks a different question. What if position never touched the query or key vectors at all, and instead the model just directly penalized attention scores between distant tokens, by an amount that scales with how far apart they are? A query shouldn't need to decode distance out of a rotated or added vector if the model can be told the distance directly, applied as a flat penalty right where the score is computed.

That reframing turns out to matter for a very practical reason: a linear penalty added straight to the score generalizes past its trained length far better than either additive or rotary encoding does, because the penalty is the same simple linear function of distance whether that distance is 50 or 50,000. Nothing about it depends on having seen a particular position during training — only the distance, and the formula for a novel distance is exactly as well-defined as for a familiar one.

Think of It Like This

A conversation with a quiet, distance-based penalty

Picture a group conversation where everyone can hear everyone, but there's a house rule: the further away two people are sitting, the more you have to raise your voice for them to actually register what you said, on a strict, fixed scale — no exceptions, no learning curve. Nobody adjusts their voice based on who is speaking or what is being discussed; distance alone sets the penalty, subtracted evenly from how loud everyone effectively sounds to everyone else.

That flat, distance-only penalty is ALiBi. Every query-key pair gets its raw attention score reduced by an amount proportional to how many positions apart they are, applied uniformly, with no learned parameters deciding the rule. Because the rule never changes shape at any distance, it keeps working exactly the same way at distances the model never encountered during training.

How It Actually Works

Penalizing the score, not the input

ALiBi computes the ordinary attention score $q_i \cdot k_j$ for query position $i$ and key position $j$ , exactly as unmodified self-attention would, and then subtracts a linear penalty before the softmax:

\text{score}_{ij} = q_i \cdot k_j - m \cdot |i - j|

$m$ is a fixed, head-specific slope — not learned, chosen from a geometric sequence set before training — and $|i - j|$ is simply the distance between the two positions. The diagram above shows exactly this: the penalty is zero on the diagonal, where $i = j$ , and grows linearly moving away from it in either direction, the same slope in both directions.

Different heads, different slopes

Rather than using one slope everywhere, ALiBi assigns each attention head its own $m$ , spanning a range from very small to comparatively large across the heads in a layer. A head with a large $m$ effectively only attends to nearby tokens, since distant scores get penalized into irrelevance well before softmax. A head with a small $m$ still favors nearby tokens somewhat, but stays meaningfully open to distant ones. Stacking heads with different slopes gives the layer both short-range and long-range attention patterns simultaneously, without training a single one of those slopes.

Why length generalization improves

Because the penalty is a fixed linear function evaluated at $|i-j|$ , and never at $i$ or $j$ individually, running the model on a sequence longer than anything seen in training doesn't require extrapolating anything the model actually learned — the same slope $m$ just gets evaluated at a larger distance, producing a larger penalty, exactly the trend it always followed. Nothing about the mechanism changes shape past the trained length, which is the specific property additive and rotary encodings don't share as cleanly: both of those depend on the model having learned to use specific frequency patterns, and patterns tied to positions never seen in training are the ones most likely to behave unpredictably.

Show Me the Code

Computing an ALiBi-biased score matrix for a toy 5-token sequence and confirming the penalty is zero on the diagonal and grows linearly away from it.

1import numpy as np2
3
4def alibi_bias(seq_len: int, slope: float) -> np.ndarray:5    positions = np.arange(seq_len)6    distance = np.abs(positions[:, None] - positions[None, :])  # |i - j| matrix7    return -slope * distance                                     # penalty, always <= 08
9
10bias = alibi_bias(seq_len=5, slope=0.5)11print(bias[2, 2])            # -> -0.0 -- no penalty when i equals j12print(bias[0, 4])            # -> -2.0 -- distance 4, slope 0.5: -0.5 * 413print(bias[1, 3])            # -> -1.0 -- distance 2, slope 0.5: -0.5 * 2

bias is added directly to the raw $QK^\top$ scores before softmax; every diagonal entry stays at zero regardless of sequence length, and every off-diagonal entry is exactly -slope * distance, matching the arithmetic above.

Watch Out For

Treating ALiBi's slope as a hyperparameter to tune per model

The per-head slopes in ALiBi come from a fixed geometric sequence decided before training, not from a search — the original design intentionally removes a hyperparameter rather than adding one. Retuning slopes per dataset defeats the point of a scheme designed to require no positional learning at all, and there's no guarantee an arbitrarily chosen slope preserves the length-generalization property the fixed sequence was chosen to have.

Expecting ALiBi and rotary embeddings to compose without a decision

Both ALiBi and RoPE solve the relative-position problem, but they do it through entirely different mechanisms — one biases scores after the fact, the other rotates vectors before the dot product. Architectures generally pick one, not both, since stacking two distinct relative-position schemes on the same score computation isn't something either was designed or validated for. Check which one a given model actually uses before assuming a technique tied to the other applies.

The Quick Version

ALiBi adds no positional information to the input at all; queries and keys are computed exactly as in ordinary self-attention.
Instead, it subtracts a penalty proportional to $|i-j|$ directly from the raw attention score, before softmax.
Each attention head gets its own fixed, unlearned slope, giving some heads short-range focus and others longer range.
Because the penalty depends only on distance, not absolute position, it evaluates the same way at distances never seen in training.
ALiBi and rotary embeddings both fix relative position, through different mechanisms — models pick one, not both.