Core ML

Rotary Position Embeddings

Instead of adding a position vector to every token, rotary embeddings rotate a token's query and key vectors by an angle tied to its position in the sequence.

Rotating two vectors at different positions by an angle proportional to position — the angle between them stays the same whenever their distance apart is the same, regardless of where they sit in the sequence

Why Does This Exist?

Positional encoding fixes attention's order-blindness by adding a sine-and-cosine vector to each token's embedding before attention ever runs. That works, but it bakes in something attention doesn't actually need: an absolute position, added once and then carried around unchanged through every layer. What a query at position 50 actually wants to know, when it scores a key at position 47, is that the key is 3 steps behind — not that the two of them happen to sit at 50 and 47 specifically. Absolute position is the wrong quantity, and recovering "3 steps behind" out of two independently-added absolute vectors takes the network extra work it has to learn on its own.

The wall shows up hardest at the edges. A sinusoidal encoding is defined at any position mathematically, but the model only ever trained on encodings up to its training length. Push it further, and quality degrades — not because the formula breaks, but because the network's attention patterns were never shaped around angles that large. Extending a model past its trained length turns into its own hard problem, one this page's cousin, long-context extension, exists specifically to address.

Rotary position embeddings, RoPE, sidestep both issues at once. Instead of adding a position-dependent vector to the embedding, RoPE rotates the query and key vectors themselves, by an angle that grows with position. Rotation has a property addition doesn't: the angle between two rotated vectors depends only on how far apart their positions are, never on where they sit absolutely. Relative position falls out of the geometry for free, which is exactly the quantity attention scores actually need.

Think of It Like This

Two clock hands, always the same distance apart

Picture two clock hands that both started pointing at 12, and each one advances by a fixed number of minutes per tick — hand A ticks 2 minutes per step, hand B does too, they just started ticking at different times. At any moment, the angle between hand A and hand B depends only on how many ticks separate their starts, not on what time it currently is. Whether you check them at 1 o'clock or 11 o'clock, if hand A is always 6 ticks ahead of hand B, the angle between them is identical both times.

That's the whole trick behind RoPE. Rotate a query vector by an angle proportional to its position, and a key vector by an angle proportional to its position, and the angle between the two rotated vectors — which is exactly what a dot product measures — depends only on the difference between their positions. Shift both positions forward by the same amount, the way both clock hands keep ticking forward together, and the angle between them never changes.

How It Actually Works

Splitting the vector into rotating pairs

RoPE treats a query or key vector as a sequence of 2D pairs — dimensions $(0,1)$ , $(2,3)$ , and so on — and rotates each pair independently, by an angle that depends on the token's position $m$ and a frequency $\theta_i$ specific to that pair:

R(m, i) = \begin{pmatrix} \cos(m\theta_i) & -\sin(m\theta_i) \\ \sin(m\theta_i) & \cos(m\theta_i) \end{pmatrix}

Just like sinusoidal positional encoding, different pairs get different frequencies $\theta_i$ , so low-index pairs rotate quickly as position increases and high-index pairs rotate slowly. But instead of adding this to the embedding, RoPE multiplies each pair of the query and key vectors by this rotation matrix, at attention time, right before the dot product.

Why the dot product ends up depending only on distance

Here's the part that makes RoPE worth the extra machinery. Rotate a query $q$ at position $m$ by angle $m\theta$ , and a key $k$ at position $n$ by angle $n\theta$ , and take their dot product. Rotation preserves length and only changes angle, so the dot product between two rotated vectors works out to depend on the difference in their rotation angles, $(m - n)\theta$ , not on $m$ or $n$ individually. The diagram above shows exactly this: shift both positions forward together and the angle between the two vectors is unchanged, because only their difference entered the computation. That's a property additive positional encoding never guaranteed — you'd have to learn it approximately; here it's exact, by construction.

Where it actually gets applied

RoPE only touches queries and keys, never values. Values still carry whatever content the token contributes once it's found relevant; rotation is purely about making the score between a query and a key sensitive to relative position, not about changing what gets blended in once attention weights are computed. This is also why RoPE composes cleanly with everything else in the attention computation — it's a preprocessing step on $Q$ and $K$ , applied per attention head, before the usual $QK^\top / \sqrt{d_k}$ and softmax proceed exactly as in ordinary self-attention.

Show Me the Code

Rotating a toy 2D query and key vector at two different positions, confirming the resulting score depends only on how far apart those positions are.

1import numpy as np2
3
4def rotate(x: np.ndarray, theta: float, position: int) -> np.ndarray:5    angle = theta * position                     # angle grows with position6    cos, sin = np.cos(angle), np.sin(angle)7    return np.array([[cos, -sin], [sin, cos]]) @ x8
9
10theta = 0.1                                       # one fixed frequency11q, k = np.array([1.0, 0.0]), np.array([1.0, 0.0])  # same raw vector, two positions12
13q_rot, k_rot = rotate(q, theta, 2), rotate(k, theta, 5)14print(round(q_rot @ k_rot, 4))  # -> 0.9553 — depends only on 5 - 2 = 315
16q_rot2, k_rot2 = rotate(q, theta, 12), rotate(k, theta, 15)17print(round(q_rot2 @ k_rot2, 4))  # -> 0.9553 — same distance, same score

Positions 2 and 5 are three apart, and so are 12 and 15 — the score comes back identical both times, exactly the invariance the diagram illustrates.

Watch Out For

Assuming RoPE is applied to values

It's tempting to assume position gets baked into everything a token carries, the same way additive positional encoding touches the whole embedding. RoPE specifically rotates only $Q$ and $K$ , because those are what the score computation reads. If you're implementing this from scratch and rotate the value vectors too, the blended output stops meaning what it's supposed to mean, and the bug is easy to miss because shapes never change.

Expecting RoPE to generalize past its trained length for free

RoPE fixes the relative position problem, but it doesn't remove the trained-length wall entirely — a model trained with positions up to 4,096 still hasn't seen the specific angles that positions past 4,096 produce, and quality typically degrades past that point just as it did with sinusoidal encoding. RoPE's clean relative-distance property is exactly why techniques like position interpolation can rescale those angles after the fact, but that rescaling is a deliberate extra step, not something RoPE gives you automatically.

The Quick Version

RoPE rotates query and key vectors by an angle proportional to position, instead of adding a position vector to the embedding.
Vectors are treated as 2D pairs, each pair rotated at its own frequency, mirroring the frequency structure of sinusoidal encoding.
The dot product between a rotated query and a rotated key depends only on the difference between their positions, never on absolute position.
Only queries and keys are rotated; values carry content and are left untouched.
RoPE doesn't remove the trained-length wall on its own — extending past it still needs a deliberate rescaling technique.