Rotary Position Embeddings

Often abbreviated RoPE, this scheme rotates a query vector at position $m$ and a key vector at position $n$ by angles $m\theta$ and $n\theta$ before their dot product runs. Rotation preserves length and only changes angle, so the resulting score ends up depending on the difference $m - n$ rather than on either position individually — relative distance falls out of the geometry instead of having to be learned.

It's the default position scheme in most current large language models, including the LLaMA and Mistral families, and it's the mechanism that long-context extension techniques like position interpolation and YaRN rescale to push a model past its trained sequence length.