Fine-Tuning a Large Language Model: A Realistic Guide

Fine-tuning adapts a base model to your domain by continuing training on your data. It's powerful, but it's not the default answer — often retrieval (RAG) gets you further, cheaper. Here's how to decide, and how to do it.

Fine-tune vs RAG

• RAG — model looks up answers in your documents at query time. Best for factual Q&A over changing knowledge.
• Fine-tuning — model learns a style, format or behaviour. Best for a fixed voice, JSON output, or a domain vocabulary.

Rule of thumb: if your data changes constantly, use RAG. If you need the model to behave differently, fine-tune. They compose — a fine-tuned model can still use RAG.

Prepare your data

[
  {
    "instruction": "Summarise the following contract in one sentence.",
    "input": "Party A agrees to deliver 200 units by Q3.",
    "output": "Party A must deliver 200 units by Q3."
  }
]

Quality beats quantity. A few hundred carefully written examples can move a model more than tens of thousands of noisy ones.

Fine-tune with LoRA

# peft + transformers — LoRA trains a tiny adapter
# while the base weights stay frozen.
from peft import LoraConfig, get_peft_model, TaskType
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")

lora = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=["q_proj", "v_proj"],
)
model = get_peft_model(model, lora)
model.print_trainable_parameters()   # ~0.1% of weights trainable

Train the adapter

trainer = Trainer(
    model=model,
    args=TrainingArguments(
        output_dir="./adapter",
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        learning_rate=2e-4,
        num_train_epochs=3,
        logging_steps=10,
    ),
    train_dataset=dataset,
)
trainer.train()
model.save_pretrained("./adapter")

The adapter is small — megabytes, not gigabytes — which is exactly why LoRA is now the standard way to fine-tune.

Evaluate honestly

• Hold out examples the model never trained on.
• Compare against the base model, not just against nothing.
• Watch for catastrophic forgetting — did it get worse at general tasks?

Fine-tuning is a precision tool. Aim it at behaviour, not at facts — and let retrieval handle the facts.