Skip to content
Lesson 5 of 12

LoRA and QLoRA Deep Dive

7 min read

Understanding LoRA From First Principles

LoRA (Low-Rank Adaptation of Large Language Models) is the technique that made fine-tuning accessible to everyone. Before diving into training code, you need to understand how it works — not just at a high level, but deeply enough to make informed decisions about rank, alpha, target modules, and when to choose QLoRA over LoRA. This lesson gives you that understanding.

How LoRA Works

The Core Idea: Low-Rank Decomposition

In a pretrained transformer, each layer contains weight matrices that transform inputs. For a linear layer with weight matrix W of dimensions (d_out x d_in), a standard update during training produces a new matrix W' = W + delta_W, where delta_W has the same dimensions as W.

LoRA's key insight: the update delta_W for task adaptation has a low intrinsic rank. Instead of learning a full (d_out x d_in) matrix, you can decompose the update into two smaller matrices:

delta_W = B @ A

Where:
  A has dimensions (r x d_in)   — the down-projection
  B has dimensions (d_out x r)  — the up-projection
  r << min(d_in, d_out)         — the rank

For a typical attention layer where d_in = d_out = 4096 and rank r = 16:

  • Full update: 4096 x 4096 = 16,777,216 parameters
  • LoRA update: (4096 x 16) + (16 x 4096) = 131,072 parameters
  • Reduction: 99.2%

During inference, the LoRA matrices are merged: W_final = W + (alpha/r) * B @ A. The merged model has the exact same architecture and speed as the original — there is zero inference overhead.

Initialization

Matrix A is initialized with random Gaussian values. Matrix B is initialized to zeros. This means the LoRA update starts as zero (B @ A = 0), so training begins from the pretrained weights and gradually moves away from them. This initialization is critical for stability.

Key Parameters

Rank (r)

The rank determines the expressiveness of the adaptation. Higher rank means more parameters and more capacity to learn complex adaptations.

# Rank comparison for a 4096-dim layer
# r=8:   2 * 4096 * 8  = 65,536 params per layer
# r=16:  2 * 4096 * 16 = 131,072 params per layer
# r=32:  2 * 4096 * 32 = 262,144 params per layer
# r=64:  2 * 4096 * 64 = 524,288 params per layer

Practical guidelines:

  • r=8: Minimal adaptation. Good for simple style changes or format compliance with large datasets.
  • r=16: The default starting point. Sufficient for most tasks including instruction following, domain adaptation, and output formatting.
  • r=32: When r=16 shows underfitting (training loss plateaus too high). Good for complex reasoning tasks.
  • r=64: Maximum you should typically need. Only when the task requires significant behavioral change. Diminishing returns beyond this.

How to choose: Start with r=16. If validation loss is still decreasing when training ends, try r=32. If r=16 trains well but eval quality is insufficient, try r=32 or r=64. If r=8 works as well as r=16, use r=8 for smaller adapter size.

Alpha (lora_alpha)

The alpha parameter scales the LoRA update: effective_update = (alpha / r) * B @ A. It controls the learning rate for the LoRA weights relative to the base model.

Common settings:

  • alpha = r (e.g., alpha=16, r=16): Scaling factor of 1.0. Conservative. Good default.
  • alpha = 2*r (e.g., alpha=32, r=16): Scaling factor of 2.0. More aggressive adaptation. Often works well in practice.
  • alpha = r/2: Very conservative. Use when you want minimal deviation from the base model.

Tip: The ratio alpha/r matters more than the individual values. An alpha=32 with r=16 is equivalent to alpha=64 with r=32 in terms of scaling. Many practitioners set alpha = 2 * r and adjust the learning rate instead.

Target Modules

LoRA can be applied to any linear layer, but the most common targets are attention layers:

from peft import LoraConfig

# Conservative: attention query and value projections only
config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
)

# Recommended: all attention projections
config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
)

# Aggressive: all linear layers (attention + MLP)
config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules="all-linear",  # Shorthand in recent PEFT versions
)

What the research says: Applying LoRA to all linear layers (including gate_proj, up_proj, down_proj in the MLP) consistently produces better results than attention-only, with moderate increase in trainable parameters. For QLoRA on memory-constrained hardware, start with attention-only and expand if quality is insufficient.

Dropout (lora_dropout)

LoRA dropout randomly zeroes elements in the LoRA layers during training, acting as regularization.

  • 0.0: No dropout. Use for larger datasets where overfitting is less likely.
  • 0.05: Light regularization. Good default.
  • 0.1: Moderate regularization. Use for small datasets (under 500 examples).

QLoRA: 4-Bit Quantization

QLoRA extends LoRA by quantizing the base model to 4-bit precision, dramatically reducing memory.

NF4 (NormalFloat 4-bit)

QLoRA introduces the NF4 data type, specifically designed for normally-distributed neural network weights. NF4 provides better information-theoretic representation than standard INT4 quantization.

from transformers import BitsAndBytesConfig
import torch

# QLoRA quantization configuration
quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",           # NormalFloat 4-bit
    bnb_4bit_compute_dtype=torch.bfloat16, # Compute in bfloat16
    bnb_4bit_use_double_quant=True,        # Double quantization
)

Double Quantization

Standard quantization stores quantization constants (scaling factors) in FP32. Double quantization quantizes these constants too, saving an additional ~0.4 bits per parameter. For a 7B model, this saves roughly 350MB of VRAM.

Paged Optimizers

QLoRA uses paged optimizers that offload optimizer states to CPU memory when GPU memory runs low. This prevents out-of-memory crashes during training spikes:

from transformers import TrainingArguments

training_args = TrainingArguments(
    optim="paged_adamw_8bit",  # 8-bit Adam with paging
    # ... other arguments
)

Memory Savings Calculation

Let us calculate the exact memory requirements for fine-tuning Llama 3.1 8B:

Full fine-tuning (FP16):
  Model weights:     8B * 2 bytes = 16 GB
  Gradients:         8B * 2 bytes = 16 GB
  Optimizer (Adam):  8B * 8 bytes = 64 GB  (2 states * 4 bytes each)
  Total:             ~96 GB

LoRA (FP16 base, r=16, attention layers):
  Model weights:     8B * 2 bytes = 16 GB (frozen, no gradients)
  LoRA params:       ~20M * 2 bytes = 40 MB
  LoRA gradients:    ~20M * 2 bytes = 40 MB
  Optimizer:         ~20M * 8 bytes = 160 MB
  Total:             ~17 GB

QLoRA (4-bit base, r=16, attention layers):
  Model weights:     8B * 0.5 bytes = 4 GB (4-bit quantized)
  LoRA params:       ~20M * 2 bytes = 40 MB
  LoRA gradients:    ~20M * 2 bytes = 40 MB
  Optimizer (8-bit): ~20M * 4 bytes = 80 MB
  Total:             ~5 GB

These are approximate — actual usage varies with batch size, sequence length, and activation memory. But the ratios are clear: QLoRA uses roughly 5% of the memory required for full fine-tuning.

LoRA vs QLoRA: When to Use Each

| Factor | LoRA (FP16) | QLoRA (4-bit) | |--------|-------------|---------------| | Memory | ~17GB for 8B | ~5GB for 8B | | Training speed | Faster | ~10-15% slower | | Quality | Slightly better | Very close | | Hardware | 24GB+ GPU | 16GB GPU works | | Best for | When you have the VRAM | When memory-constrained |

Practical recommendation: If you have enough VRAM for LoRA, use LoRA — it is slightly faster and avoids quantization artifacts. If you are memory-constrained (which most people are), QLoRA produces results that are within 1-2% of LoRA quality. The quality gap is small enough that your dataset quality matters far more than the choice between LoRA and QLoRA.

Complete Configuration Example

from peft import LoraConfig, TaskType
from transformers import BitsAndBytesConfig
import torch

# QLoRA: 4-bit quantization for the base model
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

# LoRA configuration
lora_config = LoraConfig(
    r=16,                          # Rank
    lora_alpha=32,                 # Alpha (2x rank)
    target_modules=[               # Which layers to adapt
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"
    ],
    lora_dropout=0.05,             # Light regularization
    bias="none",                   # Don't train bias terms
    task_type=TaskType.CAUSAL_LM,  # Causal language modeling
)

This configuration is a strong starting point for most fine-tuning projects. In the next lesson, we will use it to train a model end-to-end with Hugging Face's ecosystem.