LoRA
Low-Rank Adaptation is a parameter-efficient fine-tuning technique that injects small, trainable rank-decomposition matrices into frozen model weights.
Think of It Like This
Like learning a new language by adding a tiny cheat sheet to a massive dictionary, rather than rewriting the entire dictionary.
LoRA drastically reduces the number of trainable parameters by assuming weight updates have a low intrinsic rank. This allows massive models to be fine-tuned on consumer hardware with minimal VRAM while achieving parity with full fine-tuning. The resulting LoRA weights can be easily hot-swapped during inference.