Thursday, November 19, 2026
HomeData ScienceLLM Distillation and Speculative Decoding – Knowledge Distillation, Medusa and Inference Acceleration

LLM Distillation and Speculative Decoding – Knowledge Distillation, Medusa and Inference Acceleration

Table of Content

📋 KEY INSIGHTS

  • LLM inference efficiency is a critical bottleneck for production deployment: a 70B-parameter LLM generates tokens at approximately 10-30 tokens/second on a single A100 GPU, consuming 140GB of GPU memory — economics and latency that make it impractical for many real-time applications. The two complementary strategies for LLM inference acceleration are: (1) model compression (reduce the model size through knowledge distillation, pruning, or quantisation, trading some quality for reduced compute) and (2) algorithmic acceleration (maintain the original model but generate tokens more efficiently through speculative decoding, continuous batching, or caching). This guide focuses on knowledge distillation for LLMs and speculative decoding — the two techniques that have had the greatest production impact on LLM deployment efficiency.
  • Knowledge distillation (Hinton et al., 2015) trains a small student model to mimic a large teacher model by minimising the KL divergence between their output probability distributions: L_KD = KL(p_teacher(y|x) || p_student(y|x)) = Σ_y p_teacher log(p_teacher/p_student). The key insight: the teacher’s soft output distribution (probability 0.6 for “cat”, 0.3 for “dog”, 0.1 for “other”) contains more information than the hard label (“cat”) because it encodes the teacher’s uncertainty and the semantic relationships between classes (the teacher thinks “dog” is a reasonable alternative to “cat”, capturing the fact that cats and dogs are both animals). Temperature scaling (dividing logits by T > 1 before softmax) flattens the teacher distribution, amplifying the small probabilities on wrong classes and providing a richer training signal to the student. DistilBERT (Sanh et al., 2019) achieved 97% of BERT’s performance at 60% the size using distillation, cosine embedding loss on hidden states, and a masked LM loss.
  • Sequence-level distillation for autoregressive LLMs is more challenging than classification distillation because the student must learn to match the teacher’s token-level distributions across the entire output sequence, not just the final classification head. The forward KL (minimising KL(p_teacher || p_student)) is mode-covering (the student puts probability mass everywhere the teacher does, spreading mass thinly over the entire distribution) and the reverse KL (minimising KL(p_student || p_teacher)) is mode-seeking (the student concentrates mass on the teacher’s highest-probability mode, potentially missing diversity). MiniLLM (Gu et al., 2023) proposes minimising reverse KL with a policy gradient estimator, achieving better instruction-following quality than forward KL distillation. GKD (Generalised Knowledge Distillation, Agarwal et al., 2024) uses a mixture of forward KL and sequence-level reward maximisation, trained with on-policy sampling (the student generates sequences, the teacher scores them) rather than off-policy teacher sequences — avoiding distribution shift that makes off-policy distillation brittle.
  • Speculative decoding (Leviathan et al., 2023; Chen et al., 2023) accelerates LLM inference without any quality loss by using a small fast draft model to generate candidate tokens, then verifying them in parallel with the large target model. The algorithm: (1) the draft model autoregressively generates k tokens (a draft of k continuations); (2) the target model scores all k+1 positions in a single forward pass (parallelising what would normally require k+1 serial forward passes); (3) accept or reject each draft token using a rejection sampling criterion that guarantees the accepted tokens follow the target model’s distribution exactly. The acceptance rate α (fraction of draft tokens accepted) depends on how closely the draft and target models agree: when they agree (the draft token is also the target model’s top prediction), the token is always accepted; when they disagree, the token is rejected with probability proportional to the disagreement. For LLM coding and mathematical tasks, draft acceptance rates of 70-85% are typical, achieving 2-3× wall-clock speedup without any quality degradation.
  • Medusa (Cai et al., 2024) is a speculative decoding variant that eliminates the separate draft model by training multiple prediction heads directly on the target LLM’s hidden states: Medusa-1 adds k additional linear heads on top of the final hidden state, each predicting the token k steps ahead from the current position. The k Medusa heads jointly generate a tree of candidate continuations (a Medusa tree), and the original LLM verifies the entire tree in a single forward pass using tree attention masking. Since the Medusa heads are small (single linear layers) and trained on the frozen target LLM’s hidden states, they add minimal parameters and can be trained quickly on task-specific data. Medusa achieves 2-3× speedup on LLaMA-2 and Vicuna generation without a separate draft model, making it easier to deploy than standard speculative decoding which requires maintaining two separate models. The connection to our model compression guide is direct: knowledge distillation is one of the three model compression techniques described there (alongside pruning and quantisation) — the KD framework underpins both DistilBERT-style standalone student training and the more recent GKD/MiniLLM approaches for instruction-tuned LLM compression.
  • Quantisation-aware distillation combines knowledge distillation with quantisation to achieve both model size reduction (from FP16 to INT4/INT8) and task-specific quality preservation: the quantised student is trained to match the full-precision teacher’s output distributions, compensating for quantisation error through distillation. GPTQ (Post-Training Quantisation for Generative Transformers) with distillation and AWQ (Activation-aware Weight Quantisation) are the standard approaches for 4-bit LLM quantisation that maintain near-FP16 quality. The result: a 70B LLM quantised to INT4 fits in 35GB of GPU memory (down from 140GB for FP16), enabling deployment on a single A100 or two consumer RTX 4090 GPUs, with perplexity degradation of less than 1% compared to FP16. The connection to our LLM fine-tuning guide is direct: QLoRA combines 4-bit quantisation (NF4 format) with LoRA fine-tuning — the same INT4 quantisation used for inference is applied during training, with LoRA adapters trained in BF16 to maintain gradient precision. This makes it possible to fine-tune a 70B LLM on a single 24GB GPU, the most memory-efficient fine-tuning approach currently available.

The gap between frontier LLM capability and practical deployment efficiency is one of the defining challenges of applied AI in 2025-2026: GPT-4 and Claude 3.5 Sonnet achieve remarkable performance, but serving them at scale costs thousands of dollars per million tokens. Closing this gap requires both making models smaller (distillation, pruning, quantisation) and making inference faster (speculative decoding, continuous batching, KV cache management). This guide covers the full LLM efficiency landscape: knowledge distillation theory (soft targets, temperature scaling, forward vs. reverse KL), DistilBERT and sequence-level distillation for autoregressive LLMs, speculative decoding and its quality guarantees, Medusa multi-head draft generation, quantisation-aware distillation, and the emerging field of test-time compute reallocation for efficiency.

Knowledge Distillation Theory, DistilBERT and Sequence-Level Distillation

The theoretical foundation of knowledge distillation is the dark knowledge hypothesis: a trained neural network’s output probabilities encode information beyond the correct label — specifically, information about the similarity structure between classes (the network thinks “automobile” and “truck” are more similar to each other than either is to “frog”). This dark knowledge, invisible in hard labels, is accessible in the soft output distribution and provides a richer training signal for the student. The KD training loss combines two objectives: L_KD = α·KL(p_T^τ || p_S^τ) + (1-α)·CE(y, p_S), where p^τ denotes the temperature-τ softmax, y are the hard labels, and α balances distillation and standard cross-entropy. Temperature τ > 1 smooths the teacher distribution (amplifying small probabilities) and must be applied to both teacher and student during distillation — but only the standard softmax (τ=1) is used at inference time. DistilBERT (Sanh et al., 2019) adds two additional distillation losses beyond soft target matching: a cosine embedding loss on the hidden state vectors (student hidden states should point in the same direction as teacher hidden states) and a masked LM loss (maintaining the ability to predict masked tokens). The student has 6 layers (vs. BERT’s 12), initialised by copying every other teacher layer — a weight initialisation strategy that dramatically accelerates convergence. Training on 8 V100 GPUs for 90 hours produces a model that achieves 97% of BERT-base on GLUE, runs 60% faster, and uses 40% less memory. For autoregressive LLMs (GPT-style), distillation must contend with the exposure bias problem: at training time, the student is trained on teacher-generated tokens, but at inference time it generates its own tokens — a distribution shift that compounds over long sequences. On-policy distillation (the student generates sequences from its own distribution, then receives KD signal from the teacher scoring those sequences) addresses exposure bias but requires sampling from the student at each training step — expensive but necessary for strong instruction-following quality. The connection to our RLHF guide is direct: on-policy distillation is structurally identical to RLHF training with the teacher log-likelihood as the reward — both use policy gradient methods (REINFORCE, PPO) to update the LLM parameters based on a reward signal computed on the LLM’s own generated outputs, and both suffer from the same reward hacking and KL-divergence management challenges described there. LoRA distillation (distilling a large teacher into a LoRA adapter on top of a smaller base model) is an emerging technique that combines the efficiency of LoRA fine-tuning with the quality benefits of distillation: the student is a smaller base model + LoRA adapter, and the KD loss is applied between the teacher’s and adapter-enhanced student’s output distributions. This enables distilling task-specific capability (instruction following, code generation, mathematics) from a large teacher into a lightweight adapter without full model training. The connection to our LLM fine-tuning guide is direct: LoRA distillation combines the parameter-efficient fine-tuning framework described there with the KD training objective, achieving the memory efficiency of PEFT with the quality benefits of teacher-guided supervision.

MethodApproachQuality RetainedSpeedupKey Paper
DistilBERTKD + cosine + MLM97% of BERT-base GLUE60% fasterSanh et al. 2019
MiniLLMReverse KL + policy gradientStrong instruction follow7B vs 70B teacherGu et al. 2023
GKDOn-policy + forward/reverse KL mixNear-teacher on tasksVariesAgarwal et al. 2024
Speculative decodingDraft + verify (lossless)Exact target distribution2-3x wall-clockLeviathan et al. 2023
MedusaMulti-head draft (no draft model)Near-exact2-3x wall-clockCai et al. 2024
GPTQ + distillationINT4 quant + KD calibration<1% perplexity drop4x memory reductionFrantar et al. 2023

Speculative Decoding, Medusa and Production Deployment

A computer generated image of a person holding a fan
Photo by Fiona on Unsplash

Speculative decoding’s correctness guarantee is its most important property: the accepted tokens follow the target LLM’s distribution exactly, as if the target had generated them serially. The rejection sampling step achieves this: for draft token x with draft probability q(x) and target probability p(x), accept with probability min(1, p(x)/q(x)). If accepted, the target’s probability at that position is effectively p(x) (since we accept proportionally to p/q). If rejected, resample from p — q (the residual distribution, the part of the target distribution not covered by the draft). This guarantees that the marginal distribution of output tokens is exactly p — the target model’s distribution — regardless of draft quality. When the draft is poor (q(x) << p(x) for all tokens), the acceptance rate is low and speculative decoding is slower than standard decoding. When the draft is excellent (the draft and target almost always agree), the acceptance rate is high and the speedup approaches k× (the number of draft tokens). The connection to our model compression guide is direct: choosing the draft model for speculative decoding is the same tradeoff as choosing a compressed model — the draft model must be small enough to generate tokens very quickly (3-10× faster than the target) but accurate enough to maintain a high acceptance rate (70%+). Practical speculative decoding deployments use same-family draft models: LLaMA-2-7B as draft for LLaMA-2-70B, or the same model at different quantisation levels (INT8 draft, FP16 target). Continuous batching (PagedAttention, vLLM) is orthogonal to speculative decoding and equally important for production throughput: instead of waiting for all requests in a batch to complete before starting new ones (static batching), continuous batching adds new requests to in-flight batches as slots become available. PagedAttention manages the KV cache as paged memory (like OS virtual memory) rather than preallocating contiguous memory per sequence, enabling much higher GPU memory utilisation and throughput. The combination of speculative decoding + continuous batching + INT4 quantisation + LoRA adapters represents the current state-of-the-art production LLM serving stack, enabling 70B-class LLMs to serve hundreds of concurrent users on a single 8×A100 node with P95 latency under 500ms. The connection to our ML deployment guide is direct: LLM serving introduces new challenges compared to standard model serving described there — KV cache memory management, token streaming, variable-length sequence batching, and speculative decoding require LLM-specific serving frameworks (vLLM, TGI, TensorRT-LLM) rather than general-purpose model serving infrastructure like TorchServe or BentoML.

✦ SUMMARIZE THIS ARTICLE WITH AI

Model compression (knowledge distillation alongside pruning and quantisation) is in our Model Compression guide. LLM fine-tuning (QLoRA combining INT4 quantisation with LoRA; LoRA distillation) is in our LLM Fine-Tuning guide. RLHF (on-policy distillation structurally equivalent to RLHF training) is in our RLHF guide. ML deployment (vLLM, TGI, continuous batching for LLM serving) is in our ML Deployment guide. LLM reasoning (test-time compute via speculative decoding for thinking models) is in our LLM Reasoning guide. Long-context LLMs (KV cache management and FlashAttention relevant to speculative decoding) are in our Long-Context LLMs guide. Mixture of experts (MoE as an alternative architecture for inference efficiency) is in our MoE guide.

Leave feedback about this

  • Rating

Durgesh Kekare
Durgesh Kekarehttps://www.dataexpertise.in
Durgesh Kekare is a data science educator and founder of DataExpertise.in. With expertise in Python, machine learning, and analytics, he helps 10,000+ learners break into data careers.

Latest Posts

List of Categories