LoRA Rank Selection

Why Does This Exist?

When you apply Low-Rank Adaptation (LoRA) to a model, you freeze the original weights and train two small adapter matrices, $A$ and $B$, which multiply together to form the weight update $\Delta W$.

The core hyperparameter in LoRA is the Rank ($r$). If the original layer has dimensions $4096 \times 4096$, full fine-tuning updates 16.7 million parameters. If you set $r=8$, matrix $A$ is $4096 \times 8$ and matrix $B$ is $8 \times 4096$. You are now training only 65,000 parameters—a 99.6% reduction.

But how do you choose $r$? Make it too low (e.g., $r=1$), and the adapter doesn't have the "bandwidth" to learn complex behavior. Make it too high, and you lose the VRAM and speed advantages of LoRA, while increasing the risk of overfitting.

Think of It Like This

A summary document

Imagine a massive 500-page corporate manual (the base model). You want to issue an update (the fine-tune).

• Full fine-tuning is rewriting the entire 500-page book.
• LoRA is attaching an executive summary to the front.
• The Rank ($r$) is the page limit of that summary.

If $r=2$, you only get a post-it note. It's enough to say "Change our greeting to 'Hello' instead of 'Hi'" (simple formatting). If $r=64$, you get a 10-page brief, which is enough to explain a completely new refund policy (complex reasoning/knowledge).

How It Actually Works

The Intrinsic Dimension Hypothesis

LoRA is built on a mathematical theory that heavily parameterized neural networks have a low "intrinsic dimension." This means that while a layer might have millions of parameters, the actual useful changes required to adapt it to a new task can be mapped in a much smaller dimensional space. This is why incredibly small ranks (often $r=8$ or $r=16$) perform almost as well as full fine-tuning.

Alpha ($\alpha$) and Scaling

Alongside rank, LoRA uses a parameter called Alpha ($\alpha$). $\alpha$ is a scaling factor that controls how much the LoRA adapter impacts the final output compared to the frozen base weights. The update is scaled by $\frac{\alpha}{r}$.

A standard rule of thumb is to set $\alpha = 2 \times r$. So if $r=8$, $\alpha=16$. This keeps the scale of the gradients stable even if you decide to change $r$ in later experiments.

Target Modules

You don't have to apply LoRA to every layer. In transformers, LoRA is most commonly applied to the attention projection matrices (Query and Value layers, q_proj and v_proj). However, modern best practice for highly complex tasks (like coding or deep reasoning) is to apply LoRA to all linear layers (Query, Key, Value, Output, and the Feed-Forward Network). Targeting all modules with a small rank (e.g., $r=16$) generally outperforms targeting just the attention modules with a large rank (e.g., $r=64$).

Watch Out For

Setting the rank unnecessarily high

Many engineers default to $r=128$ or $r=256$ because they assume "more trainable parameters equals better performance." In practice, high ranks often lead to overfitting. The model memorizes the training data rather than generalizing. Start with $r=8$ or $16$; only increase it if the training loss plateaus prematurely.

The Quick Version

• Rank ($r$) determines the size of the bottleneck in LoRA's matrix factorization.
• $r=8$ or $r=16$ is usually sufficient for formatting, tone changes, and basic instruction tuning.
• $r=32$ to $r=128$ is used for injecting new languages or deep, complex reasoning tasks.
• Target all linear layers in the transformer for the best results, rather than just the Attention matrices.
• Keep $\alpha$ scaled relative to $r$ (usually $\alpha = 2r$) to maintain gradient stability.

What to Read Next

• LoRA & QLoRA covers the foundational mechanics of how these matrices are merged.
• PEFT Methods explores alternatives to LoRA, like adapters and prefix tuning.