When should I use LoRA instead of full fine-tuning?

Use LoRA when your task primarily involves formatting or representation (structured output, style adaptation, domain terminology), when you need swappable task-specific adapters, when memory constraints prevent full fine-tuning, or when you're iterating quickly. Consider full fine-tuning when mathematical or logical reasoning is central to performance, when building continual learning pipelines with sequential adaptations, or when that final 5-10% accuracy justifies the compute cost.

Why does LoRA underperform on reasoning tasks?

LoRA training creates 'intruder dimensions': novel singular vectors that interfere with pre-trained capabilities. For tasks that primarily format existing knowledge (classification, structured output), this interference is minimal. But tasks requiring extended reasoning capabilities see larger gaps because LoRA's low-rank updates can't fully capture the weight changes needed to enhance reasoning abilities. The 95% vs 97% accuracy gap on representation tasks widens significantly on benchmarks like GSM8k mathematical reasoning.

What LoRA hyperparameters should I start with?

Start with rank 16-32, alpha equal to rank (or 2× rank), and learning rate around 3e-5 (lower than the standard 1e-4 which often causes instability). Target all linear layers including MLP projections, not just attention. Train for 2-3 epochs maximum. Watch for training loss below 0.2, which signals overfitting even when validation metrics look stable.

LoRA Fine-Tuning: A Practical Decision Framework

Low-Rank Adaptation (LoRA) solves a practical problem: full fine-tuning of a 7B parameter model requires updating billions of weights, consuming memory and compute that most teams don't have. LoRA freezes the base model and trains small adapter matrices instead. You get a customized model at a fraction of the cost.

The efficiency story is real. But three years of production experience has revealed something the original paper didn't anticipate: LoRA's performance depends heavily on what you're fine-tuning for. Get the task match wrong and you'll spend weeks debugging a model that was never going to work well.

The core mechanism

Standard fine-tuning modifies the model's weight matrices directly. If a layer has a 4096×4096 weight matrix, you're training 16 million parameters for that single layer.

LoRA's insight is that most weight updates during fine-tuning are low-rank. Instead of modifying the full matrix, you decompose the update into two smaller matrices: a down-projection A (4096×r) and an up-projection B (r×4096), where r is your chosen rank (typically 16-32). During inference, you compute the original output plus the LoRA adjustment (A×B), which approximates what a full fine-tuning update would have produced. The original weights stay frozen, and only the adapter matrices train.

The math is elegant. A rank-32 adapter on that same layer trains about 260,000 parameters instead of 16 million. Scale that across an entire model and you're training 1-5% of the parameters while storing adapter weights you can swap at runtime.

Where practitioners diverge from the paper

The original LoRA paper targeted only attention layers (the query, key, and value projections). Subsequent experiments from Sebastian Raschka and others found that applying LoRA to all linear layers, including the MLP projections, significantly boosts performance. Yes, this increases trainable parameters by roughly 5×. It's still a tiny fraction of full fine-tuning, and the performance gain is worth it.

Anyscale's systematic comparison on Llama 2 found task-dependent gaps that matter for deployment decisions:

Representation tasks (structured output, classification): LoRA hit 95% accuracy versus 97% for full fine-tuning. A 2% gap most production systems can absorb.
SQL generation: Near-equivalent performance between methods.
Mathematical reasoning (GSM8k): LoRA significantly underperformed on 7B and 13B models.

The task-type pattern is stark. Tasks that primarily need the model to format its existing knowledge (follow a schema, adopt a tone, output structured data) work great with LoRA. Tasks that require extending the model's reasoning capabilities see larger gaps.

A NeurIPS 2024 paper explained why. LoRA training creates what the authors call "intruder dimensions": novel high-ranking singular vectors that aren't present in full fine-tuning. These intruder dimensions don't just represent new task knowledge; they actively interfere with pre-trained capabilities. For single-task fine-tuning on representation-heavy work, this interference is minimal. But intruder dimensions accumulate during sequential fine-tuning. If you're planning continual adaptation (fine-tune for task A, then B, then C), LoRA's knowledge degradation compounds in ways full fine-tuning avoids.

Our read: this finding matters most for teams building adaptive systems that learn from deployment data. If you're doing one-shot fine-tuning for a specific use case, the intruder dimension effect is a footnote. If you're planning an ongoing adaptation pipeline, it's a potential blocker.

Hyperparameters worth caring about

The practical literature has converged on reasonable defaults.

Rank (r): 16 or 32 for most applications. Higher ranks increase capacity but risk overfitting. Raschka found r=256 with alpha=512 worked in some experiments, but this is the exception rather than the rule.

Alpha: Set equal to rank, or 2× rank. Alpha scales the magnitude of LoRA updates; the alpha/rank ratio determines effective learning rate contribution. Start with alpha=rank and adjust if training is unstable.

Target modules: All linear layers, not just attention. Include q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj. The original paper's attention-only approach is outdated advice.

Learning rate: This is where people get burned. The standard 1e-4 learning rate caused instability in Anyscale's experiments; 3e-5 stabilized training. LoRA's optimization landscape is trickier than full fine-tuning. Start lower than you think.

Epochs: Training beyond 3 epochs shows diminishing returns. Multi-epoch training on static datasets can actually degrade performance, contrary to traditional deep learning intuition. Watch your loss curves.

Training loss below 0.2 signals overfitting. This is more reliable than validation metrics for catching the problem early.

If you hit this threshold, reduce your learning rate, limit epochs, or increase weight decay. Raschka's experiments showed that continuing to train past this point hurts generalization even when validation loss looks stable.

Dataset quality matters more than size here. 1,000 curated examples (the LIMA approach) can outperform 50,000 noisier examples. LoRA's limited capacity means garbage in, garbage out hits faster than full fine-tuning. One additional caution: models can "unlearn" capabilities not represented in fine-tuning data. If your training set lacks arithmetic examples and the base model was good at arithmetic, you may degrade that capability. Include representative examples of skills you want to preserve.

QLoRA: when memory is the constraint

QLoRA combines LoRA with 4-bit quantization of base weights. You save roughly 33% memory at the cost of about 39% longer training time. Accuracy impact is minimal for most tasks.

When does the tradeoff make sense? Teams with limited GPU memory who need to fine-tune larger models. Iteration speed during development (faster to start runs). Inference deployment where quantization was planned anyway. It makes less sense when training time is your bottleneck, you're targeting the reasoning tasks where LoRA already underperforms, or you need maximum accuracy and have the compute budget for full fine-tuning.

Making the call

Use LoRA when your task is primarily representation or formatting (structured output, style adaptation, domain terminology), when you need multiple task-specific adapters you can swap at runtime, when memory constraints prevent full fine-tuning, or when you're iterating quickly and need faster experiment cycles.

Consider full fine-tuning when mathematical or logical reasoning is central to performance, when you're building a continual learning pipeline with sequential adaptations, when that final 5-10% of accuracy justifies the compute cost, or when you're training once for long-term deployment.

The practitioner's rule of thumb: start with LoRA. If performance gaps appear on reasoning-heavy tasks after hyperparameter tuning, graduate to full fine-tuning. You'll have saved significant compute on tasks where LoRA was sufficient, and you'll know exactly which capabilities need the full treatment.

Sources cited: Claims as analysis: