Hugging Face Fixed A Silent Bug In Packed Training
Hugging Face's SFT trainer supported packing — concatenating short training examples into full-length rows to avoid wasting compute on padding — for a long time before it correctly masked attention across the documents it packed together. The gap is documented plainly: packing existed, cross-document masking did not, until the capability landed in Transformers 4.44. Content was rephrased for compliance with licensing restrictions. That gap is worth a post on its own, because of what it did not do: it never raised an error.
What packing without masking actually does
Sequence packing concatenates multiple training examples into one row and cuts the stream into full context-length chunks, instead of padding every example out to the batch's longest member. The efficiency win is real and large — often a four-times reduction in wasted token slots, as the sequence-packing page's worked example shows. The correctness condition is that attention inside a packed row has to stay within each original document: token 300, in the middle of document two, must not attend to token 50, in the middle of document one.
Without that block-diagonal mask, attention runs all-to-all across the whole row by default. The forward pass computes something. The loss goes down. Nothing about the training run signals a defect, because there is no shape mismatch, no NaN, no exception — just a model quietly learning that unrelated documents flow into one another.
Why nobody caught it sooner
The reason this shipped and stayed unnoticed is the reason it's genuinely dangerous: the failure mode looks exactly like normal training. Loss curves for packed and properly-masked runs are not obviously distinguishable from loss curves for packed and leakily-masked runs, especially early in training. The symptom, when it shows up at all, is a trained model with a mild, hard-to-attribute tendency to drift topic mid-generation — plausibly explained away as a data-quality issue, a prompt-formatting issue, or simply "the model being the model," rather than traced back to a masking bug in the trainer.
This is precisely the failure the sequence-packing page names as the one worth checking for directly rather than trusting: run one document alone, run it packed alongside another, and compare the logits for that document's tokens. If they diverge, attention is leaking across the boundary.
What changed, and what to check now
Modern kernels take a cu_seqlens-style argument — cumulative sequence-length offsets — and enforce document boundaries natively rather than requiring a hand-built mask, which is both faster and harder to get wrong than materializing a full block-diagonal tensor yourself. That's the mechanism the fix relies on.
If your fine-tuning pipeline enables packing through Hugging Face's trainer, confirm you're on a version at or after the fix and that packing-aware masking is actually turned on for your configuration rather than assumed. If you're on an older pinned version for reproducibility reasons, that pin is now also a correctness liability, not just a compatibility one. And regardless of which trainer you use: the two-run logit comparison from the sequence-packing page costs a few minutes and is the only check that would have caught this bug the whole time it existed.