ALiBi Attention Bias
A positional scheme that adds no vector to the input at all, instead subtracting a distance-proportional penalty from attention scores before softmax runs.
Short for Attention with Linear Biases, ALiBi computes the ordinary score and then subtracts from it, where is a fixed, unlearned slope specific to each attention head and is the distance between the query and key positions. No positional embedding touches the query, key, or value vectors at any point.
Because the penalty is the same linear function of distance at any sequence length, ALiBi generalizes to sequences longer than its training length noticeably better than additive or rotary schemes, without needing a separate rescaling step.