LLM Fine-Tuning with LoRA & PEFT – Practical Python Guide 2026

Fine-tuning a large language model used to require dozens of A100 GPUs and weeks of compute time. LoRA (Low-Rank Adaptation) changed that — it makes fine-tuning a 7B parameter model possible on a single consumer GPU with 16 GB VRAM in a few hours. This guide covers the practical workflow for fine-tuning open-source LLMs on custom datasets using the HuggingFace PEFT library.

Why Fine-Tune Instead of Prompting?

Prompt engineering works well for general tasks, but fine-tuning is better when: you need consistent output formatting the model doesn’t naturally produce, you want to inject domain-specific knowledge (legal documents, medical terminology, internal company style), you need to reduce hallucinations on a specific topic, or you’re reducing inference costs by using a small fine-tuned model instead of a large general one. Fine-tuning a 7B model to be a medical QnA bot often outperforms GPT-4 with prompting alone on that narrow domain.

How LoRA Works

Full fine-tuning updates all model weights — billions of parameters. LoRA freezes all original weights and adds small trainable “adapter” matrices to the attention layers. Each adapter is a low-rank decomposition: instead of a full d×d weight update matrix, LoRA uses two small matrices A (d×r) and B (r×d) where rank r is typically 4–64. This reduces trainable parameters by 99%+. QLoRA (Quantized LoRA) goes further by loading the base model in 4-bit precision, reducing VRAM requirements from 28 GB to ~6 GB for a 7B model.

Setup

pip install transformers peft bitsandbytes datasets trl accelerate

Preparing Your Dataset

from datasets import Dataset
import pandas as pd

# Format: instruction-response pairs
data = [
    {"instruction": "Summarise this medical report in plain English.",
     "input": "Patient presents with...",
     "output": "The patient has..."},
    # ... more examples
]

def format_prompt(example):
    return f'''### Instruction:
{example['instruction']}

### Input:
{example.get('input', '')}

### Response:
{example['output']}'''

dataset = Dataset.from_pandas(pd.DataFrame(data))
dataset = dataset.map(lambda x: {"text": format_prompt(x)})

Loading the Base Model with 4-bit Quantization (QLoRA)

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

model_name = "meta-llama/Llama-3.1-8B"  # or mistralai/Mistral-7B-v0.3

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True)

tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

Attaching LoRA Adapters

from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training, TaskType

# Prepare model for k-bit training
model = prepare_model_for_kbit_training(model)

lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,                 # LoRA rank (higher = more capacity, more VRAM)
    lora_alpha=32,        # scaling factor (usually 2*r)
    lora_dropout=0.05,
    bias="none",
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"]  # attention layers
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Trainable params: 41,943,040 || All params: 8,072,220,672 || 0.52%

Training with SFTTrainer

from trl import SFTTrainer
from transformers import TrainingArguments

args = TrainingArguments(
    output_dir="./llm-finetuned",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,   # effective batch = 16
    learning_rate=2e-4,
    fp16=False, bf16=True,           # use bfloat16 on Ampere GPUs
    logging_steps=10,
    save_strategy="epoch",
    warmup_ratio=0.03,
    lr_scheduler_type="cosine",
    report_to="none")

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=2048,
    args=args)

trainer.train()
trainer.save_model("./llm-finetuned")

Merging and Saving the Full Model

from peft import AutoPeftModelForCausalLM

# Load fine-tuned model with LoRA adapters
peft_model = AutoPeftModelForCausalLM.from_pretrained(
    "./llm-finetuned", torch_dtype=torch.bfloat16, device_map="auto")

# Merge LoRA weights into base model and save
merged_model = peft_model.merge_and_unload()
merged_model.save_pretrained("./llm-merged")
tokenizer.save_pretrained("./llm-merged")

Inference

from transformers import pipeline

pipe = pipeline("text-generation", model="./llm-merged",
                torch_dtype=torch.bfloat16, device_map="auto")

prompt = '''### Instruction:
Explain machine learning in one paragraph suitable for a business executive.

### Response:'''

output = pipe(prompt, max_new_tokens=256, do_sample=True, temperature=0.7)
print(output[0]["generated_text"][len(prompt):])

VRAM Requirements by Model Size

With QLoRA (4-bit), a 7B model needs ~6 GB VRAM (fits on RTX 3060/4060), a 13B model needs ~10 GB (RTX 3080), a 34B model needs ~20 GB (RTX 3090/4090), and a 70B model needs ~40 GB (requires multi-GPU or A100). For training (not inference), add 2-4× for gradients and optimizer states. On Google Colab Pro, you get access to A100 (40 GB) which can handle 7B–13B models comfortably.

Conclusion

LoRA and QLoRA have democratised LLM fine-tuning. What required 8 A100s in 2022 runs on a single RTX 4090 in 2026. The workflow — quantize the base model, attach LoRA adapters, train with SFTTrainer, merge and deploy — is standardized and well-supported by the HuggingFace ecosystem. Fine-tuning a 7B model on 1,000 domain-specific examples takes 2-3 hours and often outperforms GPT-4 prompting on narrow tasks. If you work with LLMs, this skill is worth investing in.