LoRA (Low-Rank Adaptation)

The problem with full fine-tuning

When a massive model like a Large Language Model (LLM) is pre-trained, it learns general patterns across billions of parameters, stored in massive weight matrices like the grey $W$ matrix you see here.

If you want to fine-tune it on a specific task, like writing code or speaking a specific language, the standard approach is to update every single parameter. This means you must load the entire model into GPU memory, compute gradients for every weight, and save a brand new multi-gigabyte copy of the model.

That is extremely expensive, slow, and often totally unnecessary.

Freezing the base

The first trick in LoRA is to stop updating the original weights altogether. We freeze $W$.

In the simulation, this is when the padlock appears and $W$ turns grey. The gradients stop flowing through the massive matrix. The original knowledge of the model is perfectly preserved, which prevents catastrophic forgetting (where a model forgets how to speak English because it was fine-tuned too aggressively on French).

But if $W$ is frozen, how does the model learn anything new?

The LoRA bottleneck

Instead of changing $W$, we add a new path next to it. We learn an additive update, $\Delta W$, such that the new output is $h = Wx + \Delta Wx$.

The magic of LoRA is that we do not make $\Delta W$ the same massive size as $W$. Instead, we decompose it into two thin matrices: $A$ and $B$.

If the original dimension is $d \times d$, then $A$ is $d \times r$ and $B$ is $r \times d$. The bottleneck dimension $r$ is the rank.

By forcing the information to squeeze through this narrow rank $r$, the number of parameters we have to train drops dramatically. For example, if $d=64$ and $r=4$:

• A full update would need $64 \times 64 = 4,096$ parameters.
• LoRA needs $(64 \times 4) + (4 \times 64) = 512$ parameters. That is an 87.5% reduction, while often achieving the exact same performance.

Editing a masterpiece

Imagine you want to correct a few specific details on a massive, highly detailed map of the world (the base model).

Instead of redrawing the entire map from scratch (full fine-tuning), you place a sheet of clear tracing paper over it and only draw the new roads and borders on the tracing paper. The tracing paper is much smaller and lighter to carry (the LoRA weights), but when overlaid on the map, you get the updated world.

The Rank Trade-off

The rank $r$ is the most important hyperparameter in LoRA.

Grab the rank slider and sweep it across its full range. As the rank increases, the matrices $A$ and $B$ physically widen, and the number of trainable parameters grows.

Simultaneously, watch the loss curve, which here is a real reconstruction error: the simulation fits $A$ and $B$ to an actual fixed target update via truncated SVD, so the number you see is not a hand-picked formula, it is however much of that target the current rank can still capture. Giving the model a higher rank gives it more "expressivity", more capacity to learn complex updates, so the achievable loss floor drops, until the rank reaches the target update's own intrinsic dimension and the reconstruction is exact.

There is also an $\alpha$ slider, which scales the applied update: $\Delta W = (\alpha / r) \cdot AB$. At $\alpha = 0$, the update is nullified no matter how high the rank is, the bottleneck may have plenty of capacity, but nothing is added to $W$. At $\alpha = r$, the update is applied completely unscaled.

Too low or too high?

At $r=0$ the bottleneck has no capacity at all: $\Delta W$ is exactly zero, and the loss sits at its worst value. Set the rank too low for a complex task (like $r=1$) and the matrices are barely more than vectors, lacking the mathematical capacity to express the necessary weight update. The loss plateaus early, leaving you with an underperforming model.

However, increasing the rank has diminishing returns. The difference between $r=1$ and $r=3$ is huge, but ranks well past a target update's own intrinsic rank barely move the loss further, while doubling your memory footprint. You always want to find the lowest rank that reaches your target loss.

What to take away

LoRA proves that the updates needed to fine-tune a model have a much lower intrinsic dimension than the model itself. We do not need to update billions of parameters to teach an LLM a new trick; a few million, intelligently arranged in a low-rank bottleneck, are more than enough.