Model Compression – Pruning, Quantisation and Knowledge Distillation

Training a state-of-the-art model and deploying it in production are two fundamentally different engineering problems. A fine-tuned 70B parameter language model may achieve impressive benchmark scores, but deploying it to serve thousands of requests per second requires either hundreds of thousands of pounds of GPU hardware or a compression strategy that makes the model smaller and faster without sacrificing meaningful accuracy. Model compression — the set of techniques for reducing the size, memory footprint, and inference cost of ML models — has become a standard phase in the production ML lifecycle. The field has advanced rapidly: modern LLMs can be quantised to 4-bit precision with less than 1% accuracy degradation; structured pruning can reduce transformer models to 50% of their original size with minimal performance loss; knowledge distillation can train a 7B student model to match a 70B teacher on specific tasks. This guide covers the four main compression approaches — quantisation, pruning, knowledge distillation, and efficient architecture design — with practical guidance on when to use each and how they interact with the deployment infrastructure.

Quantisation — Trading Precision for Efficiency

Neural network weights are typically stored and computed in 32-bit floating point (FP32) or 16-bit floating point (FP16/BF16). Quantisation maps these continuous-valued weights to lower-precision discrete representations — 8-bit integer (INT8), 4-bit integer (INT4), or even lower. The size reduction is proportional to the precision reduction: INT8 uses 4× less memory than FP32; INT4 uses 8× less. Inference speed improvements are even greater on hardware with native INT8 or INT4 support (modern NVIDIA GPUs, Apple Neural Engine, Qualcomm Hexagon DSP), because integer arithmetic is faster and uses less power than floating-point arithmetic.

Post-training quantisation (PTQ) quantises a pre-trained model without retraining — it is fast (minutes) and requires only a small calibration dataset. PTQ using a round-to-nearest scheme degrades accuracy substantially below INT8 for most models. The better approach is GPTQ (Frantar et al., 2022) for LLMs: it quantises weights layer by layer, using the Hessian of the reconstruction error to correct for quantisation-induced distortions via a second-order update. GPTQ achieves INT4 quantisation of 7B–70B LLMs with less than 1% perplexity degradation, enabling these models to run on consumer GPUs. AWQ (Lin et al., 2023) achieves similar results by protecting the small fraction of weights with large activation magnitudes from aggressive quantisation — these “salient” weights contribute disproportionately to model quality. The GGUF format (used by llama.cpp) implements mixed-precision quantisation where different layers use different bit widths based on their sensitivity, and is the dominant format for local LLM inference. The mathematical connection to the Hessian matrix and second-order optimisation used in GPTQ is covered in our linear algebra and calculus for ML guides.

Quantisation-aware training (QAT) simulates quantisation during the forward pass (using fake quantisation operators that round weights during forward but pass gradients through in the backward pass — the straight-through estimator), training the model to be robust to quantisation error. QAT typically achieves better accuracy than PTQ, especially at very low bit widths (INT4 and below), at the cost of a full training run. For models that will be deployed at large scale, QAT’s higher quality justifies its computational cost. The gradient flow through the straight-through estimator is a notable example of non-differentiable quantisation being made compatible with backpropagation.

Pruning — Removing Redundant Capacity

Pruning is based on the observation that trained neural networks are massively over-parameterised: a large fraction of weights are close to zero and contribute negligibly to the model’s output. Removing these weights reduces model size and, for structured pruning, inference cost. The key question is which weights to remove — the pruning criterion — and whether to prune all at once (one-shot pruning) or gradually during training (iterative pruning).

Magnitude pruning removes weights with the smallest absolute values, on the assumption that small-magnitude weights contribute least to the model’s output. It is fast and simple, but not always the most accurate criterion — a small weight with a large input activation can contribute substantially to the output. Gradient-based pruning (e.g. movement pruning, used in BERT compression) removes weights whose gradient-weighted magnitude is smallest — weights that would not change much if training continued. Structured pruning removes entire structural units: attention heads in Transformers, filters in CNNs, neurons in fully-connected layers, or entire layers. Structured pruning produces dense (non-sparse) weight matrices in the remaining structure, which benefit directly from standard hardware acceleration — unlike unstructured pruning, which produces sparse matrices that require specialised sparse computation support.

The lottery ticket hypothesis (Frankle and Carlin, 2019) proposed that large networks contain sparse subnetworks (“winning tickets”) that can be trained in isolation to the same accuracy as the full network. This finding motivated iterative magnitude pruning: train the full network, prune the smallest weights, reset remaining weights to their initialisation values, retrain — repeat. Each cycle identifies more of the winning ticket subnetwork. The connection to L1 regularisation (which shrinks weights toward zero, making them easier to prune) and dropout (which also randomly zeroes weights during training) is direct, and our regularisation guide covers these overlapping techniques.

Knowledge Distillation — Learning From a Teacher

Knowledge distillation (Hinton et al., 2015) trains a small student model to replicate the output of a large, pre-trained teacher model. Rather than training the student on hard labels (one-hot class assignments), distillation trains it to match the teacher’s soft predictions — the full probability distribution over classes. The soft predictions contain “dark knowledge”: a teacher that assigns 0.7 probability to class A, 0.2 to class B, and 0.1 to class C is revealing that A and B are more similar to each other than to C, even though the hard label is simply “A.” This inter-class similarity information — which is discarded when only the hard label is used — helps the student generalise better from limited training data.

The distillation loss combines the standard cross-entropy against hard labels with a cross-entropy (or KL divergence) against the teacher’s softened predictions at temperature T: L = α × CE(y, student) + (1−α) × CE(teacher/T, student/T). Temperature T > 1 softens the teacher’s distribution, giving more weight to low-probability classes and transferring more inter-class similarity information. The KL divergence between teacher and student distributions is the information-theoretic lens on distillation — the student minimises the information gap between its distribution and the teacher’s.

Distillation is the primary method for creating efficient NLP models: DistilBERT achieves 97% of BERT’s performance with 40% fewer parameters and 60% faster inference; TinyBERT uses intermediate-layer distillation (matching attention matrices and hidden states, not just final predictions) to achieve even better compression. The transfer learning connection is significant: distillation can be viewed as a form of transfer learning where the teacher is the “pre-trained model” and the student is the “fine-tuned model” — but the knowledge transfer happens through soft labels rather than weight initialisation. For gradient boosting and ensemble models, distillation trains a single model (often a neural network) to replicate an ensemble’s outputs, enabling the accuracy of an ensemble with the inference speed of a single model.

✦ SUMMARIZE THIS ARTICLE WITH AI

ChatGPT Perplexity Claude Grok

The neural network architectures that are the primary targets of compression — Transformers, CNNs, LSTMs — are in our Neural Network Architectures guide. Transfer learning and fine-tuning — complementary to compression for efficient model reuse — is in our Transfer Learning guide. The Hessian-based second-order optimisation used in GPTQ quantisation is in our Calculus for ML and Linear Algebra guides. Regularisation techniques (L1 pruning, dropout) that overlap with compression are in our Regularisation guide. Deploying compressed models to REST APIs and edge hardware is in our ML Model Deployment guide.