Full Transformer Block

The Transformer block is the fundamental building block of modern large language models. It combines several critical layers into a single cohesive pipeline.

First, tokens are converted into embeddings, which are continuous vector representations of discrete words or subwords. Since transformers process all tokens in parallel, they have no inherent concept of order. To fix this, a positional encoding is added to each embedding, injecting a unique signal that depends on the token's position in the sequence.

The core of the block is multi-head attention. Each head projects the input into its own query, key and value vectors through a fixed linear projection, computes a weighted sum of values based on the similarity between queries and keys, and every head's output is then mixed back into one vector by a further output projection. Splitting this into multiple heads allows the model to attend to different types of relationships simultaneously, each in its own subspace.

After attention, a residual connection adds the original input back to the attention output. This helps preserve the original information and allows gradients to flow easily during training, preventing the vanishing gradient problem. The result is then normalized using Layer Normalization, which stabilizes the activations and speeds up convergence.

Next, a feed-forward network (FFN) independently processes the representation of each token. It expands each token's vector to a wider hidden layer, applies a non-linearity, and projects it back down to the original size. While attention mixes information across the sequence, the FFN transforms the information within each token's representation, with no visibility into any other token.

Finally, another residual connection and layer normalization step is applied to the FFN output, completing the Transformer block. This pipeline of Attention -> Add & Norm -> FFN -> Add & Norm can be stacked many times to build deep, powerful models like GPT and BERT.

Notice that nothing in attention or the feed-forward network reads a token's position — attention compares every pair of tokens the same way regardless of where they sit, and the FFN processes each token's vector independently of its neighbours. Positional encoding is the only signal in the whole block that breaks that symmetry: with it removed, permuting the input tokens permutes the output identically, and nothing else changes. Turn it off in the lab and swap two tokens to see the swap propagate through cleanly, with no side effects.