Core ML

Positional Encoding

Add a unique signal built from sine and cosine waves at different frequencies to each token's embedding, since self-attention has no way to tell one position from another on its own.

Each position gets a vector built from sine and cosine waves at different frequencies, so shuffling the tokens without adding this signal would leave attention unable to tell the order apart

Why Does This Exist?

Self-attention computes a weighted average of value vectors, and averaging has no concept of order — $\frac{1}{3}(a + b + c)$ is identical to $\frac{1}{3}(c + a + b)$ . Shuffle the tokens fed into a self-attention layer, and absent anything else, the same multiset of output vectors comes back, just reassigned to the shuffled positions. The mechanism that made every token visible to every other token in one step paid for that with a real cost: it threw away the information a vanilla RNN got automatically, just by processing tokens one at a time in order.

That's not a hypothetical gap — "the dog bit the man" and "the man bit the dog" contain the exact same multiset of tokens, and a model that can't distinguish word order can't distinguish those two sentences. Something has to tell attention which position is which, and it has to be added before attention runs, since attention itself has no mechanism for it.

Think of It Like This

Watermarking every page of a shuffled stack

Imagine a stack of loose pages that got dropped and shuffled, with no page numbers printed anywhere. There's no way to recover the original order — every page looks structurally identical to every other page, content aside. Now imagine each page instead carries a unique watermark pattern, visible under the right light, that encodes exactly where in the original sequence it belonged. Shuffle the pages all you like; the watermark survives, and anyone reading them can reconstruct the true order from the pattern alone.

Positional encoding is that watermark, added to each token's embedding before it ever reaches attention — a pattern unique enough that "position 5" and "position 47" are never confusable, no matter how attention subsequently mixes information across positions.

How It Actually Works

Building a unique signal per position

The original transformer's approach assigns each position $t$ a vector where each pair of dimensions oscillates at a different frequency:

PE_{(t, 2i)} = \sin\!\left(\frac{t}{10000^{2i/d}}\right), \qquad PE_{(t, 2i+1)} = \cos\!\left(\frac{t}{10000^{2i/d}}\right)

$t$ is the position, $i$ indexes the dimension pair, and $d$ is the embedding dimension. The diagram above shows what this actually looks like: low dimensions ( $i$ near 0) oscillate quickly as $t$ increases, high dimensions oscillate slowly. Reading across all dimensions at a fixed position gives a specific combination of "where in each wave's cycle" — and because the frequencies are all different, that combination is essentially unique to that position, the same way a combination lock's specific digit sequence identifies one specific unlocking state out of many.

This vector is added, elementwise, directly to the token's embedding before the first attention layer: $x_t' = x_t + PE_t$ . Attention then operates on $x_t'$ , which now carries positional information baked directly into the same vector as the token's content — attention itself never needs a separate "position" input, because position is already folded in.

Why sine and cosine specifically

Two properties make this particular construction useful beyond just "unique per position." First, it's bounded — sine and cosine never exceed $[-1, 1]$ , so positional signal never dominates or vanishes relative to the token embedding's own scale, regardless of how long the sequence gets. Second, the relationship between two positions' encodings is a smooth, learnable function of their distance: $PE_{t+k}$ can be expressed as a linear function of $PE_t$ for a fixed offset $k$ , which in principle helps a model generalize what it learns about nearby-position relationships to positions it saw less often during training.

What replaced it, briefly

Later architectures largely moved to rotary position embeddings, which inject position by rotating the query and key vectors by an angle proportional to position, rather than adding a separate vector to the embedding. The underlying goal is identical — give attention a way to recover relative position — but the mechanism differs enough to earn its own page. What doesn't change is the wall this page opens with: attention has no order of its own, and something external always has to supply it.

Show Me the Code

Building the sinusoidal encoding matrix and confirming two properties directly: it's bounded, and every position gets a distinct pattern.

1import numpy as np2
3
4def positional_encoding(seq_len: int, d_model: int) -> np.ndarray:5    position = np.arange(seq_len)[:, None]                       # (seq_len, 1)6    dim_pair = np.arange(0, d_model, 2)                            # even indices: 0, 2, 4, ...7    freqs = 1.0 / (10_000 ** (dim_pair / d_model))                  # one frequency per pair8    angles = position * freqs                                      # (seq_len, d_model/2)9    pe = np.zeros((seq_len, d_model))10    pe[:, 0::2] = np.sin(angles)11    pe[:, 1::2] = np.cos(angles)12    return pe13
14
15pe = positional_encoding(seq_len=6, d_model=8)16print(pe.max() <= 1.0 and pe.min() >= -1.0)             # -> True — bounded, regardless of position17print(np.allclose(pe[0], pe[3]))                         # -> False — every position is distinct

Every row of pe stays within $[-1, 1]$ no matter how large seq_len grows, and no two distinct positions produce the same row — both properties the argument above depends on, checked rather than assumed.

Watch Out For

Forgetting positional encoding entirely on a custom architecture

Because positional encoding is added once, quietly, before the first attention layer, it's easy to omit entirely when hand-assembling a transformer from parts — nothing crashes, shapes all still match, and the model trains, just noticeably worse on anything where word order carries meaning. The failure is silent and gradual rather than a hard error, which makes it a common first bug when implementing attention from scratch. If a from-scratch transformer trains but seems oddly insensitive to word order, check for this first.

Extrapolating a fixed encoding scheme past its trained length

A sinusoidal positional encoding is well-defined mathematically at any position, so it's tempting to assume a model trained on 2,048-token sequences will generalize cleanly to 8,000-token ones just because the formula still produces valid numbers past that length. In practice, the model has never seen those higher-position encodings during training and its attention patterns weren't shaped around them, so quality typically degrades past the trained context length even though nothing about the encoding itself breaks. This is exactly the problem long-context extension techniques exist to address.

The Quick Version

Self-attention is permutation-equivariant — it has no built-in sense of token order, since averaging discards it.
Positional encoding adds a per-position vector, built from sine and cosine waves at different frequencies, directly to each token's embedding.
The encoding is bounded in $[-1, 1]$ and essentially unique per position, so it never dominates or vanishes at scale.
Attention itself never needs a separate position input — position is folded into the same vector as the token's content before attention runs.
Rotary position embeddings later replaced this additive scheme with a rotation-based one, aimed at the same underlying gap.