Gen AI

Self-Attention

A way for each word to look around the sentence and pull in exactly the context it needs to make sense.

One word forms a query that is compared against every word's key to decide how much of each value to pull in

Why Does This Exist?

Take the sentence "The trophy didn't fit in the suitcase because it was too big." What does "it" refer to, the trophy or the suitcase? You know instantly from context. A model needs some way to let the word "it" look around the sentence and gather the clues that pin down its meaning.

Self-attention is that mechanism. It lets every word dynamically decide which other words are relevant to it and blend their information in. It's the core operation inside transformers, and the reason they handle context so well.

Think of It Like This

Skimming a textbook with a question in mind

Say you're searching a textbook for one specific thing. You hold a question in your head (a query). You skim the section headings (keys) looking for matches, and where a heading fits, you read that section's content (value). Self-attention does exactly this for every word at once: each word poses a query, matches it against every word's key, and pulls in the values that fit best.

How It Actually Works

Each token is turned into three vectors:

Query (Q) is what this token is looking for.
Key (K) is what each token offers to others.
Value (V) is the actual content each token carries.

The mechanism then runs four steps:

Score. Take the dot product of a query with every key. A large dot product means "these two are relevant to each other."
Scale. Divide the scores by the square root of the vector size so the numbers don't blow up as vectors get bigger.
Softmax. Turn the scores into weights that add up to 1, a focus map saying how much attention to pay to each token.
Blend. Take a weighted sum of the values. Each token walks away with a context-aware version of itself.

"Self" attention just means Q, K, and V all come from the same sequence, so the sentence attends to itself.

How attention weights form

Step 1 of 3

Score

Take the dot product of the query with every key to get a raw relevance score.

Show all steps

Score: Take the dot product of the query with every key to get a raw relevance score.
Normalize: Softmax turns those scores into positive weights that sum to one.
Blend: Combine the value vectors in proportion to the weights — that weighted sum is the output.

Show Me the Code

1import numpy as np2
3
4def softmax(z: np.ndarray) -> np.ndarray:5    e = np.exp(z - z.max(axis=-1, keepdims=True))  # subtract max for stability6    return e / e.sum(axis=-1, keepdims=True)7
8
9def attention(q: np.ndarray, k: np.ndarray, v: np.ndarray) -> np.ndarray:10    d = q.shape[-1]  # the key/query dimension11    scores = q @ k.T / np.sqrt(d)  # similarity, scaled to tame large d12    weights = softmax(scores)  # turn scores into a 0-to-1 focus map13    return weights @ v  # blend values by how much to focus on each14
15
16q = k = v = np.eye(3)  # three tokens, toy identical vectors17print(attention(q, k, v).round(2))  # each token mixes in the others

The whole mechanism is just a similarity score, a softmax, and a weighted average. No recurrence, no convolution.

Watch Out For

Unscaled scores break the softmax

As vectors get larger, raw dot products grow with them, pushing softmax toward a near one-hot spike where gradients vanish. Dividing by the square root of the dimension keeps it well-behaved. That's the "scaled" in scaled dot-product attention.

It has no built-in sense of order

Self-attention treats its input as a set: shuffle the words and the math barely changes. Transformers add positional encoding precisely because attention alone can't tell first from last.

The Quick Version

Self-attention lets each token gather context from every other token.
It builds a Query, Key, and Value vector for each token.
Scores come from query-key dot products, scaled, then softmaxed into weights.
The output is a weighted sum of values, a context-aware version of each token.
It has no sense of order on its own, so transformers add positional information.