Fine-Tuning LLMs with LoRA

Training a large language model from scratch is out of reach for an individual — it costs millions of dollars in compute. But adapting an existing open model to your own data is surprisingly accessible, and the technique that makes it possible is LoRA (Low-Rank Adaptation). I've been training a custom model I call KING2 this way, and this post explains what LoRA actually does and the full train → merge → upload pipeline I use.

The Problem with Full Fine-Tuning

A modern open model has billions of parameters. Full fine-tuning updates all of them, which means you need enough GPU memory to hold the weights, their gradients, and the optimizer state — typically several times the size of the model itself. A 7-billion-parameter model can demand 60+ GB of VRAM to fine-tune fully. That's data-center territory, not a single consumer GPU.

The LoRA Insight

LoRA starts from a clever observation: the change a fine-tune makes to a weight matrix tends to be "low rank" — it can be approximated by the product of two much smaller matrices. So instead of updating a giant weight matrix W, LoRA freezes it and learns a small additive update:

W_effective = W (frozen) + (B × A) × (alpha / r)

where:
  A : shape (r, d)   — small, trainable
  B : shape (d, r)   — small, trainable
  r : the rank (e.g. 8, 16, 32) — tiny
  alpha : a scaling factor

Because r is small (often 8 or 16), the two adapter matrices have a fraction of the parameters of the original. You might train less than 1% of the total parameter count and still capture most of the benefit. The frozen base never changes, so you only store and update the adapters.

Why This Saves So Much Memory

Two big wins stack up:

Far fewer trainable parameters means far less gradient and optimizer memory — often the dominant cost.
QLoRA goes further by quantizing the frozen base model to 4-bit. The base sits in memory compressed while the adapters train in higher precision on top.

Together these bring fine-tuning a 7B model down to something that fits on a single 16 GB GPU — the kind you can rent by the hour or find in a free notebook environment.

The Key Hyperparameters

Three knobs matter most when configuring a LoRA run:

Rank (r) — capacity of the adapter. Higher means more expressive but more parameters and more overfitting risk. 8–16 is a sensible starting range.
Alpha — scales how strongly the adapter affects the output. A common rule of thumb is alpha = 2 × r.
Target modules — which layers get adapters. Attention projections (q_proj, k_proj, v_proj, o_proj) are the usual targets; adding the MLP layers increases capacity at extra cost.

Preparing the Data

The model is only as good as the dataset. For instruction-style tuning I format each example as a consistent prompt/response template and keep it strict — mixing formats confuses the model. A typical record:

{
  "instruction": "Explain what a closure is in JavaScript.",
  "input": "",
  "output": "A closure is a function bundled together with ..."
}

Quality beats quantity. A few thousand clean, on-distribution examples outperform tens of thousands of noisy ones. I deduplicate, strip broken samples, and hold out a small validation set to watch for overfitting.

Merging the Adapter

After training you have two artifacts: the original base model and a small adapter. You can serve them separately — load the base, then attach the adapter at runtime — which is great for swapping behaviors. But for a single distributable model, I merge the adapter back into the weights:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained(BASE_ID)
model = PeftModel.from_pretrained(base, ADAPTER_DIR)

# fold B×A back into W
merged = model.merge_and_unload()
merged.save_pretrained("KING2-merged")

Merging removes the runtime dependency on the adapter library and produces a standalone model that any standard loader can run. The trade-off is you lose the ability to detach the adapter, so I keep the raw adapter archived separately.

Shipping the Model

The final step is uploading the merged weights to a model hub so they can be pulled by ID anywhere. I script this so the whole pipeline — train, merge, push — runs unattended from a notebook:

merged.push_to_hub("MOT1209/KING2")
tokenizer.push_to_hub("MOT1209/KING2")

Always upload the tokenizer alongside the weights — a model with a mismatched tokenizer produces garbage, and it's the most common "why is my model broken" mistake.

What I'd Tell My Past Self

Start small. Validate the whole pipeline end-to-end on a tiny model and a handful of examples before burning GPU hours on the real run.
Watch the validation loss. If training loss keeps dropping while validation loss rises, you're overfitting — lower the rank, add data, or stop earlier.
Version everything. Dataset, hyperparameters, and base model ID. Reproducibility is what turns a lucky run into a repeatable process.

Conclusion

LoRA democratized model customization. By learning a tiny low-rank update on top of frozen weights, it turned fine-tuning from a data-center luxury into something you can do on a single rented GPU. If you have a focused task and a clean dataset, you can have a custom model trained, merged, and published in an afternoon — and understand exactly why every step is there.

Fine-Tuning LLMs with LoRA: A Practical Walkthrough