Gen AI

Context Windows

The context window is a fixed-size frame of tokens the model can actually see — anything outside it, on either end, simply isn't part of the input anymore.

The context window is a fixed-size sliding frame over tokens — everything inside it is visible to attention, and anything that falls outside it, on either end, simply isn't there anymore

Why Does This Exist?

Every transformer-based model has a hard architectural limit on how many tokens it can process in one forward pass — a number baked into the model at training time, not a setting a user can raise by asking nicely. This isn't an arbitrary product restriction; it follows directly from attention's quadratic cost: the compute and memory needed to run self-attention over $n$ tokens grows with $n^2$ , so a model has to be trained and served with some fixed ceiling on $n$ , or the cost of a single request becomes unbounded.

That ceiling is the context window, and understanding it precisely matters because two very different failure modes get conflated under the same word "context." One is a hard cutoff — text beyond the window's token limit is truncated and the model never sees it at all. The other is a soft quality problem — text technically inside the window but far from where the model is currently generating can still be attended to less effectively than nearby text, even though it's nominally "in context."

Think of It Like This

Reading through a fixed-size window cut into a scrolling scroll

Picture an old scroll being read through a small, fixed-size window cut into a board placed on top of it — you can only see whatever portion of the scroll currently sits behind the window, and as you scroll further along, text at the trailing edge disappears off one side while new text appears at the other. You never get less legible near the edges of the window; text either sits behind the window and is fully visible, or it's rolled past the edge and is gone completely.

The context window works the same way: every token currently inside it is available to attention, uniformly, by construction. There's no partial visibility for tokens still inside the frame — the boundary is binary, in or out — but as generation continues and the frame slides forward, the oldest tokens eventually roll off the trailing edge and stop being available at all.

How It Actually Works

A hard limit, set at training time

A model is trained with a specific maximum sequence length — positional encoding schemes and the attention computation itself are both built around processing sequences up to some fixed $n_{\max}$ . Advertising a "128K context window" means the model was trained (or later extended) to handle sequences up to 128,000 tokens; feeding it more than that either truncates the excess, fails outright, or — for models using a fixed sliding-window scheme — silently drops the oldest tokens as newer ones arrive, exactly as the diagram shows.

What "inside the window" actually guarantees

Every token inside the window is, structurally, available for self-attention to read from — nothing about the architecture prevents a query at any position from attending to any key still within the window. What the window does not guarantee is that attention will use distant context as effectively as nearby context; a model can have technically full access to a fact stated at token 50 while generating token 90,000, and still attend to it comparatively weakly in practice, a phenomenon often described loosely as "lost in the middle." That's a training and architectural-quality question, layered on top of the hard binary limit this page is about — the two are related but genuinely separate failure modes, and conflating them leads to misdiagnosing which problem you actually have.

The cost that scales with window size, not linearly

Because attention cost is quadratic, a model advertising twice the context length of another doesn't cost twice as much to run a full-context request through — it costs roughly four times as much, all else equal. This is the same relationship attention complexity covers directly, and it's the reason context length became a headline specification worth marketing and optimizing rather than a minor implementation detail: doubling it is a genuinely expensive decision, not a free upgrade.

Show Me the Code

A simple sliding-window truncation, showing exactly which tokens survive and which fall off the trailing edge as new tokens arrive.

1def apply_context_window(tokens: list[int], new_token: int, window_size: int) -> list[int]:2    tokens = tokens + [new_token]3    if len(tokens) > window_size:4        tokens = tokens[-window_size:]      # drop from the FRONT — oldest tokens leave first5    return tokens6
7
8window = list(range(1, 6))                    # tokens 1..5 already "in the window"9print(window)                                   # -> [1, 2, 3, 4, 5]10window = apply_context_window(window, new_token=6, window_size=5)11print(window)                                   # -> [2, 3, 4, 5, 6] — token 1 fell off the front12window = apply_context_window(window, new_token=7, window_size=5)13print(window)                                   # -> [3, 4, 5, 6, 7] — token 2 fell off next

Each new token pushes the oldest one out once the window is full — the list length never exceeds window_size, and the tokens that fall off are gone, not merely deprioritized.

Watch Out For

Assuming a longer advertised window means uniformly good use of all of it

A model's advertised maximum context length is the hard limit it was trained or extended to handle — it is not a guarantee that quality stays flat across the entire window. Models frequently perform measurably worse at retrieving or reasoning over information placed in the middle of a very long context compared to the beginning or end, even when every token is technically within the advertised limit. Treat "supports 200K tokens" and "uses all 200K tokens equally well" as two separate claims, and test the second one directly for the specific task at hand rather than assuming it from the first.

Budgeting context cost linearly with length

Because attention cost is quadratic in sequence length, doubling how much context a request uses does not double its cost — it roughly quadruples the attention computation specifically, on top of the separately linear cost of the KV cache. Teams estimating serving cost or latency by scaling linearly with token count routinely underestimate the cost of their longest requests, precisely because the dominant cost term doesn't scale the way intuition suggests.

The Quick Version

The context window is a hard limit on how many tokens a model can process at once, set at training time and following from attention's quadratic cost.
Every token inside the window is structurally available to attention; tokens outside it, past either edge, aren't part of the input at all.
Being "in the window" doesn't guarantee equally effective use of that context — retrieval and reasoning quality can still vary by position within it.
A sliding-window scheme drops the oldest tokens first as new ones arrive, once the window is full.
Doubling context length roughly quadruples attention cost, not doubles it, because of the underlying quadratic relationship.