Gen AI

Supervised Fine-Tuning

SFT keeps training a pretrained model on a curated set of example answers, so it reliably repeats that exact style instead of just guessing plausible text.

In one demonstration example, the prompt is context the model conditions on, while the loss and the gradient that updates the weights are computed only over the completion tokens

Why Does This Exist?

A pretrained model is good at one thing: continuing text in a way that looks plausible, given everything it read during pretraining. That's genuinely useful, but "plausible" and "what the company wants" are two different targets. Tell a base model to write a support reply and it might do it well once, ramble the next time, or skip the numbered steps entirely on the third try — because nothing ever taught it that this specific shape of answer is the one that counts.

Say you're building a support chatbot. The brief is exact: acknowledge the customer's problem in one line, lay out next steps as a numbered list, then close with a short reassurance. Prompting can nudge a model toward that shape. It can't lock it in. Across thousands of differently worded customer messages, a prompt-only setup drifts — the acknowledgment disappears, the list becomes a paragraph, the closing line vanishes. That's not a bug in the prompt. It's what happens when the desired behavior lives in an instruction the model reads, rather than in the weights themselves.

SFT moves the behavior from "instruction the model might follow" to "pattern it has actually learned." You show it real input-output pairs, and keep training the same pretrained model until producing that pattern is just what it does.

Think of It Like This

Training a new hire on real, already-answered emails

Imagine onboarding a new support agent. You could hand them a style guide: "acknowledge the issue, then list steps, then close warmly." Some new hires nail it immediately. Most don't — a written rule and consistent behavior under real pressure are different things.

Now imagine instead you hand them a folder of two hundred real customer emails, each with the ideal reply attached, written exactly the way you want every reply written. The new hire reads pair after pair — this question, this answer — and after enough of them, the pattern stops being a rule they're consciously following and becomes just how they write. That folder of worked examples is the demonstration set. Absorbing it is what SFT does to a model.

How It Actually Works

Demonstration pairs and the same objective as pretraining

Mechanically, SFT starts with a dataset of demonstrations: pairs of an input and the exact output you want for it. For the chatbot, an input is a real customer message; the output is a transcript written or curated in the target shape — brief acknowledgment, numbered next steps, closing line. Nothing about the training mechanism changes from pretraining: the model is still trained with next-token prediction, still scored on how much probability it assigns to the correct next token at each position. What changes is where the "correct" answer comes from. In pretraining, the next token is whatever happened to appear next in some scraped document. In SFT, it comes from a demonstration a person wrote or approved specifically because it's the answer they want reinforced. Same mechanism, curated target.

Masking the loss to completion tokens

Each training example is really one sequence: the prompt (the customer's message, plus whatever template wraps it) followed by the completion (the ideal reply). The model has to see the whole thing — it can't condition on a prompt it never reads. But the loss, the number that drives the weight updates, is typically computed over the completion tokens only. The prompt tokens are masked out.

Why bother? The goal isn't to make the model better at predicting customer messages — it'll never be asked to generate one. The goal is to shape what it produces once the message is already given. Spending gradient signal on predicting its own input would burn capacity on a skill nobody needs, and dilute the real objective: getting the completion right.

Why quality beats volume

This is the part that surprises people coming from a "more data is always better" instinct. A demonstration set teaches the model whatever pattern is actually in it — consistent or not. Two hundred transcripts that are all genuinely excellent, all following the acknowledgment-steps-closing shape precisely, teach that shape cleanly, because every single example reinforces the same thing. Five thousand scraped, inconsistent transcripts — some rambling, some missing the numbered list, some in the wrong tone — teach the model an average of all of that. At best you get a blurrier version of the pattern. At worst, the model faithfully reproduces whatever bad habits show up often enough in the noisy pile, because from the model's point of view, a frequent pattern in the data is the correct pattern to learn.

For the chatbot, 200 excellent, hand-checked transcripts reliably beat 5,000 mediocre ones scraped from old support logs. Volume doesn't buy consistency. Curation does.

Show Me the Code

A small function that builds the loss mask described above — 1 at completion positions, 0 at prompt positions — given only the two lengths.

1def completion_loss_mask(prompt_len: int, completion_len: int) -> list[int]:2    """Mark only completion-token positions as trainable.3
4    Position i is 1 if it falls in the completion span, else 0 --5    matching a loss function that ignores masked positions entirely.6    """7    total_len = prompt_len + completion_len8    return [0 if i < prompt_len else 1 for i in range(total_len)]9
10
11mask = completion_loss_mask(prompt_len=5, completion_len=3)12print(mask)  # -> [0, 0, 0, 0, 0, 1, 1, 1]

Watch Out For

Computing loss over the full sequence, prompt included

Skipping the mask and training on every token — prompt and completion alike — doesn't crash anything, so it's an easy mistake to miss. It just wastes a chunk of every training step teaching the model to predict input tokens it will never be asked to generate on its own, which slows learning on the completion pattern that's actually the point.

Scaling the dataset instead of curating it

Grabbing every past transcript to get a bigger dataset feels like progress, but an inconsistent pile trains an inconsistent model. If a fifth of the transcripts skip the numbered list, the model learns that skipping it is an acceptable fifth-of-the-time move. A smaller, hand-checked set with zero exceptions to the format teaches the format with zero exceptions.

The Quick Version

SFT keeps training a pretrained model on curated (input, output) demonstration pairs.
It uses the exact same next-token-prediction objective as pretraining — only the source of the "correct" next token changes.
Loss is usually masked to the completion tokens only; the prompt is context the model reads but isn't scored on predicting.
A small, consistent, high-quality demonstration set teaches the pattern more reliably than a large, noisy one.
For the support chatbot, 200 clean transcripts beat 5,000 scraped ones — because the model learns whatever's actually in the data, inconsistencies included.