Transfer Learning and Fine-Tuning – Complete Guide for Machine Learning 2026

Transfer learning is the single most impactful technique in modern deep learning — enabling state-of-the-art performance on tasks with limited labelled data by leveraging knowledge learned from large datasets. The shift from training models from scratch to fine-tuning pretrained models has democratised deep learning: a team with 500 labelled medical images can achieve results that would have required 500,000 images just five years ago. This guide covers the complete transfer learning landscape — from classical feature extraction to modern parameter-efficient fine-tuning (PEFT) for large language models.

Transfer learning spans all deep learning modalities: computer vision (covered in our Computer Vision Interview Q&A), natural language processing (covered in our NLP Interview Q&A), and the neural network architectures that make it possible (see our Neural Network Architectures guide). The fine-tuning decisions described here directly affect the model evaluation and hyperparameter tuning process, and deploying fine-tuned models raises the MLOps challenges of versioning, drift monitoring, and serving.

Why Transfer Learning Works

Deep neural networks learn hierarchical representations. In a CNN trained on ImageNet: early layers detect edges, corners, and colour gradients — low-level features universally present in any natural image. Middle layers detect textures, patterns, and object parts — moderately transferable. Late layers detect high-level semantic features specific to ImageNet categories (dog ears, car wheels) — less transferable to very different domains.

This hierarchy means that early and middle layer weights can be directly reused for new visual tasks — a model trained to classify 1000 ImageNet categories has already learned to see. For NLP, BERT’s pretraining on Wikipedia and BookCorpus gives it deep understanding of English grammar, semantics, and world knowledge — fine-tuning on a downstream task (sentiment analysis, NER, QA) adapts this knowledge to the specific task with as few as 100-1000 examples.

The data-availability decision matrix:

Feature Extraction — Frozen Backbone

Feature extraction uses the pretrained model as a fixed feature extractor — all pretrained weights are frozen (no gradient updates) and only the new task-specific head is trained. This is appropriate when your dataset is small, your domain is similar to the pretraining data, and you need fast training.

import torch
import torch.nn as nn
from torchvision import models, transforms
from torch.utils.data import DataLoader

# Load pretrained ResNet-50
backbone = models.resnet50(weights=models.ResNet50_Weights.IMAGENET1K_V2)

# Freeze ALL pretrained parameters
for param in backbone.parameters():
    param.requires_grad = False

# Replace final layer with task-specific head
n_classes = 5
backbone.fc = nn.Sequential(
    nn.Dropout(0.3),
    nn.Linear(backbone.fc.in_features, 256),
    nn.ReLU(),
    nn.Linear(256, n_classes)
)
# Only the new head has requires_grad=True
# Verify: only head parameters will be updated
trainable = sum(p.numel() for p in backbone.parameters() if p.requires_grad)
total     = sum(p.numel() for p in backbone.parameters())
print(f'Trainable: {trainable:,} / {total:,} ({trainable/total*100:.1f}%)')

# Preprocessing must match what the pretrained model expects
preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],  # ImageNet mean
                         std= [0.229, 0.224, 0.225])   # ImageNet std
])

Fine-Tuning — Unfreezing Pretrained Layers

Fine-tuning unfreezes some or all pretrained layers and trains them alongside the new head — allowing the pretrained representations to adapt to the new task and domain. The critical challenge: the pretrained weights are carefully calibrated; training them with a high learning rate destroys the useful representations (catastrophic forgetting). Use a very small learning rate for pretrained layers.

Discriminative learning rates (ULMFiT strategy, fastai): Assign different learning rates to different layer groups — lower rates for earlier (more general) layers, higher rates for later layers and the head. Earlier layers are adapted gently; the head is trained aggressively.

# Fine-tuning with discriminative learning rates
backbone = models.resnet50(weights=models.ResNet50_Weights.IMAGENET1K_V2)
backbone.fc = nn.Linear(backbone.fc.in_features, n_classes)

# Group parameters by layer depth
early_layers  = list(backbone.layer1.parameters()) + list(backbone.layer2.parameters())
middle_layers = list(backbone.layer3.parameters())
late_layers   = list(backbone.layer4.parameters())
head_params   = list(backbone.fc.parameters())

# Discriminative learning rates: early = 1e-5, head = 1e-3
optimizer = torch.optim.AdamW([
    {'params': early_layers,  'lr': 1e-5, 'weight_decay': 1e-4},
    {'params': middle_layers, 'lr': 3e-5, 'weight_decay': 1e-4},
    {'params': late_layers,   'lr': 1e-4, 'weight_decay': 1e-4},
    {'params': head_params,   'lr': 1e-3, 'weight_decay': 1e-4},
])

# Learning rate warmup + cosine annealing
scheduler = torch.optim.lr_scheduler.OneCycleLR(
    optimizer,
    max_lr=[1e-5, 3e-5, 1e-4, 1e-3],
    steps_per_epoch=len(train_loader),
    epochs=10,
    pct_start=0.1   # 10% warmup
)

Gradual unfreezing: Start with all pretrained layers frozen; train the head for 1-2 epochs until it converges. Then unfreeze the last pretrained block and train for 1-2 more epochs. Continue unfreezing one block at a time. This prevents early overfitting by the randomly initialised head disrupting the pretrained weights before it produces reasonable gradients.

Parameter-Efficient Fine-Tuning (PEFT) for LLMs

Full fine-tuning of large language models (GPT-4: ~1.8T parameters, LLaMA-3: 70B parameters) requires enormous GPU memory — a 7B parameter model in float16 requires ~14GB just to store the weights, plus optimiser state and gradients during training. PEFT methods fine-tune a tiny fraction of parameters while keeping the rest frozen, achieving comparable performance at a fraction of the cost.

LoRA (Low-Rank Adaptation, 2021): The most widely-used PEFT method. The key insight: weight updates during fine-tuning have low intrinsic rank — they lie in a low-dimensional subspace. Instead of updating the full weight matrix W (d×k), LoRA approximates the update as two small matrices: ΔW = A·B where A is d×r and B is r×k, with r << min(d,k). Typical r=8 or r=16. Only A and B are trained (and initialised: A~N(0,σ), B=0 so ΔW=0 at the start). At inference, ΔW is merged into W: W’ = W + αΔW/r (α controls the scaling). For a 7B LLM, LoRA with r=16 adds only ~0.1% additional parameters.

from peft import LoraConfig, get_peft_model, TaskType
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = 'meta-llama/Llama-3-8B'
tokenizer  = AutoTokenizer.from_pretrained(model_name)
model      = AutoModelForCausalLM.from_pretrained(
    model_name, torch_dtype=torch.float16, device_map='auto'
)

lora_config = LoraConfig(
    r=16,                      # LoRA rank
    lora_alpha=32,             # scaling factor (alpha/r = 2.0)
    target_modules=['q_proj', 'v_proj'],  # apply LoRA to attention
    lora_dropout=0.05,
    bias='none',
    task_type=TaskType.CAUSAL_LM
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 4,194,304 || all params: 8,030,261,248 || 0.052%

Domain Adaptation and Common Pitfalls

Domain shift: When the target domain is very different from the pretraining domain (e.g., applying ImageNet-pretrained models to medical X-rays, or BERT to legal documents), transfer learning may be limited. Solutions: domain-adaptive pretraining (continue pretraining on unlabelled target-domain data before task-specific fine-tuning); intermediate task fine-tuning (fine-tune on a related labelled task first); and domain-specific pretrained models (BioBERT for biomedical, LegalBERT for legal, FinBERT for finance).

Catastrophic forgetting: Fine-tuning on a new task can erase knowledge needed for the original task. Relevant when you need the model to perform multiple tasks. Solutions: Elastic Weight Consolidation (EWC) — penalises changes to weights most important for the original task; multi-task learning — jointly train on original and new task; LoRA — since the original weights are frozen, catastrophic forgetting is impossible (only the small adapters change).

Always freeze BatchNorm during fine-tuning: BatchNorm layers contain running statistics (mean and variance) computed during ImageNet training on millions of images. Fine-tuning with a small dataset would corrupt these statistics. Set all BN layers to eval mode: for m in model.modules(): if isinstance(m, nn.BatchNorm2d): m.eval().

Transfer learning is the foundation of modern applied deep learning — it underpins the BERT fine-tuning described in our NLP Interview Q&A, the ResNet transfer learning in our Computer Vision Interview Q&A, and the RLHF fine-tuning in our Reinforcement Learning Explained guide. For interview questions specifically on transfer learning trade-offs, our Deep Learning Interview Q&A covers when to freeze layers, what learning rate to use, and how to diagnose transfer learning failures.