Fine-Tuning Data Preparation

Why Does This Exist?

When a team runs their first Supervised Fine-Tuning (SFT) job and the model comes out generating gibberish or ignoring instructions, they usually start tweaking the learning rate or increasing the epoch count. Nine times out of ten, they are trying to solve a data pipeline bug with calculus.

A base model is fragile. It learns exactly what you show it. If your training data contains hidden whitespace, inconsistent prompt formatting, or if you accidentally penalize the model for failing to predict the question (instead of just the answer), the gradients will destroy the model's capabilities. Data preparation is the rigorous, non-glamorous engineering step that protects the model from garbage in.

Think of It Like This

Prepping the canvas before painting

If you paint a masterpiece on a canvas covered in grease, the paint will flake off. You don't need a more expensive paintbrush; you needed to gesso the canvas properly.

Data preparation is the gesso. The hyperparameter tuning is the brush. The brush does not matter if the canvas is ruined.

How It Actually Works

Data preparation for fine-tuning has four mandatory steps that cannot be skipped.

1. The Chat Template

You cannot just concatenate "Question: X \n Answer: Y" and feed it to the model. Every base model (Llama, Mistral, Qwen) was pretrained to recognize specific control tokens that delineate the user's turn from the assistant's turn.

For example, Llama 3 expects: <|start_header_id|>user<|end_header_id|>\n\nWhat is X?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n

If your data pipeline does not format the training data using the exact tokenizer chat template expected by the base model, you are essentially speaking to it in a broken alien language. It will fail to learn.

2. Loss Masking (Train on Completions Only)

This is the most common catastrophic bug. When you feed a Q&A pair to the model, you only want it to learn how to generate the Answer. You do not want it to update its weights trying to guess what the Question was going to be.

To fix this, data pipelines use loss masking. The entire conversation is tokenized, but the tokens belonging to the user's prompt are masked out (usually by setting their label to -100 in PyTorch). During backpropagation, the loss function ignores any token labeled -100. The model reads the prompt to gain context, but only optimizes its weights on the assistant's response.

3. Deduplication

If the same Q&A pair appears 50 times in a 10,000-example dataset, the model will memorize it. When confronted with a similar but distinct question in production, it will blindy repeat the memorized answer. Exact-string and semantic deduplication ensures the model learns the underlying logic, not a lookup table.

4. A Held-Out Test Split Before Touching the Data

If you clean your data, run deduplication, and then split 5% off for testing, you might accidentally leak information. The golden rule is to split your test set out immediately from the raw data. This is your only honest measure of whether the fine-tune actually generalized to unseen prompts.

Watch Out For

Training on the prompt

If you forget to mask the loss on the prompt tokens, the model will spend 50% of its gradient updates trying to learn how to generate user questions. This degrades its ability to actually answer them and causes severe performance drops compared to the base model.

The Quick Version

• Chat Templates: Must match the exact special tokens (e.g., <|user|>, <|assistant|>) the base model was trained on.
• Loss Masking: You must set the labels of the prompt tokens to -100 so the model only learns from the assistant's responses.
• Deduplication: Prevents the model from memorizing repeated examples.
• Most fine-tuning failures are formatting or masking bugs, not learning-rate problems.

What to Read Next

• Chat Templates dives specifically into the tokenizer's role in formatting.
• Instruction Tuning explains the actual training phase that uses this prepared data.