Attention

What you are looking at

The core of a Transformer, in five steps. Along the top is a sequence of tokens. Below it is the attention matrix, where the row is the token doing the reading and the column is the token being read from. At the last stage a second panel appears: the focused row of that matrix, multiplied by the value matrix, giving one output vector.

Language models do not process words left to right. They process them all at once, and to work out what a word means in context a token has to look at the other words around it. The matrix decides exactly how much information flows between any two of them.

Scores can be negative, and the sign means something

The Scores stage shows Q·Kᵀ / √d before any normalisation. A large positive score means the query and the key point the same way. A negative score means they point apart — the two tokens are actively unalike, which is as real a piece of information as a strong match. In this sentence the scores run from −1.0 to 2.8, and the two signs are drawn as two different hues so the difference does not vanish into a pale cell.

Read the diagonal while you are here. She, sat, by and river all score highest against themselves, but the and bank do not: both score highest against river.

The budget

In the Weights stage, every row of the matrix sums to exactly 1.0.

This is the structural constraint of attention: it is a fixed budget. A token cannot pay full attention to every other token. If it needs to spend more on river, it is arithmetically forced to spend less somewhere else.

Effective tokens puts a number on how the budget got split. It is exp(H), where H is the row's entropy, and it runs from 1 — the whole budget on a single token — to 6, an even split across the sequence. At the default temperature the bank row reads 3.13 tokens effectively. It is the companion to the row sum: the sum tells you the budget is always 1, this tells you how many ways it went.

Asymmetry: reading versus being read

Attention is directional. If A pays a lot of attention to B, it does not follow that B pays a lot to A: the cell at (row B, col A) can be very different from (row A, col B).

That is because a token is not one representation. It is split into a Query — what am I looking for — and a Key — what do I contain. bank's query strongly matches river's key. river's query is looking for something else. Focus each in turn and watch the arcs swing round.

What the weights are for

Everything up to here decides how much. The blend stage is where it is spent.

Each token also carries a Value vector: what it contributes when something reads it. The focused token's output is the weighted average of all six value vectors, with the attention weights as the weights — A·V, the line that comes after the famous softmax. Multiply down each column of the two aligned grids, sum across, and the answer is the column on the right.

The value vectors in this lab are two-dimensional so the result is small enough to read. One axis is a river sense, the other a money sense. river sits far out on the first and nowhere on the second. bank sits between them, because on its own it is ambiguous — the whole reason for this sentence.

So the output says what the token has come to mean here. At the default temperature bank spends 0.556 of its budget on river and 0.306 on itself, and its output lands at (0.69, 0.23): mostly the water sense, some of its own still intact. Sense resolved measures that displacement — how far the blend moved from the average of all the values, as a fraction of the furthest a single token could have moved it.

Because the weights are non-negative and sum to one, the output is a convex combination: it can move anywhere inside the spread of the value vectors and never outside it. Attention cannot invent a meaning that none of the tokens carried.

The temperature dial, and both ways it breaks

The Temperature slider divides the raw scores before they are normalised. It is one parameter and it has two failure ends.

Push it up and the differences between scores shrink toward nothing. Every row goes uniform, and at the top of the range all six tokens' outputs land within 0.049 of each other — against 0.503 at the default. Every token has been handed the same answer, which is the mechanism having stopped: there is no longer any such thing as this word's context.

Pull it down toward 0.1 and it collapses the other way. One token takes nearly the whole budget, and the output becomes a copy of that token's value vector. bank at temperature 0.20 reads river with 0.951 and itself with 0.049 — its sense is pinned to the water reading, and it has thrown away every other word in the sentence to get there, including itself. Sharp is not the same as good.

What to take away

"Paying attention" in a neural network is taking a weighted sum of vectors. The matrix supplies the weights, softmax forces them to sum to one, and the temperature decides whether they say anything. The interesting setting is neither extreme: enough concentration to resolve something, enough spread to still be reading a sentence rather than a single word.