Deep Learning Interview Questions and Answers – Top 40 for 2026

Deep learning interviews at AI companies, research labs, and top tech firms go deeper than general ML interviews. You need to understand not just how to use PyTorch or TensorFlow, but how and why the architectures work, what their failure modes are, and how to debug training. This guide covers the 40 most commonly asked deep learning interview questions with thorough answers for 2026 — covering everything from backpropagation to Transformers and LLMs.

Neural Network Fundamentals

Q1. What is a neural network and why are multiple layers useful?
A neural network is a composition of learnable linear transformations (weight matrices) and non-linear activation functions. A single linear layer can only learn linear decision boundaries — adding a non-linear activation after each linear layer enables the network to learn arbitrary functions. The universal approximation theorem states that a single hidden layer with enough neurons can approximate any continuous function — but in practice, depth is far more parameter-efficient than width. Multiple layers allow the network to learn hierarchical representations: early layers learn simple features (edges, character strokes), middle layers learn combinations (shapes, word parts), and deep layers learn high-level concepts (objects, semantics).

Q2. Explain activation functions. Why can’t we just use linear activations?
Activation functions introduce non-linearity. Without them, stacking linear layers collapses to a single linear transformation regardless of depth: W₃(W₂(W₁x)) = Wx. The network would be equivalent to a single layer, unable to learn non-linear patterns. Sigmoid σ(x) = 1/(1+e^-x): outputs (0,1), historically used for output layers and gates. Problem: saturates (gradient ≈ 0) for large |x|, causing vanishing gradients. Tanh: outputs (-1,1), zero-centred (better than sigmoid), still saturates. ReLU max(0,x): non-saturating for positive inputs, computationally simple, most widely used. Problem: “dying ReLU” — neurons that receive only negative inputs always output zero and stop learning. Leaky ReLU: small negative slope (0.01x) for x < 0, fixes dying ReLU. GELU (Gaussian Error Linear Unit): used in Transformers (BERT, GPT), smoother than ReLU, often outperforms it on NLP tasks.

Q3. What is the difference between a loss function and an optimiser?
The loss function measures how wrong the model’s predictions are — it defines what “wrong” means for your task. Common losses: Binary Cross-Entropy for binary classification; Categorical Cross-Entropy for multiclass; MSE (Mean Squared Error) for regression; Focal Loss for class imbalance. The loss function does not change model parameters — it only computes a scalar error. The optimiser uses the gradient of the loss to update parameters. Common optimisers: SGD (Stochastic Gradient Descent) — simple, requires careful learning rate tuning. Adam — adaptive learning rates per parameter (maintains running averages of gradients and their squares), converges faster and with less tuning. AdamW — Adam with decoupled weight decay, the standard for Transformer training. RMSprop — similar to Adam, often preferred for RNNs.

Q4. What is batch normalisation and why does it help training?
Batch normalisation (BatchNorm) normalises the inputs of each layer to have zero mean and unit variance across the mini-batch, then applies learnable scale (γ) and shift (β) parameters. Benefits: (1) Reduces internal covariate shift — the distribution of each layer’s inputs does not drift as earlier layers change, allowing higher learning rates. (2) Acts as a regulariser — the noise introduced by batch statistics reduces reliance on individual training examples, often allowing dropout to be removed. (3) Makes training less sensitive to weight initialisation. (4) Smooths the loss landscape, enabling faster convergence. Limitation: behaviour differs between training (uses batch statistics) and inference (uses running average statistics) — remember to call model.eval() before inference in PyTorch.

Q5. What is dropout and how does it prevent overfitting?
Dropout randomly sets a fraction p of neuron outputs to zero during each training step, independently for each forward pass. During inference, dropout is disabled and all neurons are active — their outputs are scaled by (1-p) to account for the higher activation. Mechanism: by randomly disabling neurons, dropout prevents any single neuron from becoming overly specialised or from co-adapting with specific other neurons — each neuron must learn to be useful on its own. This forces the network to learn redundant representations. Conceptually equivalent to training an ensemble of 2^n different sub-networks (where n is the number of dropout neurons) and averaging their predictions at inference. Typical dropout rates: 0.2-0.5 for dense layers, 0.0-0.1 for convolutional layers, 0.1 for Transformer attention.

CNN Interview Questions

Q6. How does a convolutional layer work? What are its advantages over fully-connected layers?
A convolutional layer applies learnable filters (kernels) by sliding them across the input, computing a dot product at each position. A 3×3 filter applied to a 224×224 image produces a 222×222 feature map (with no padding). Stacking convolutional layers builds hierarchical features. Advantages over fully-connected: (1) Parameter sharing — the same filter is applied everywhere, vastly reducing parameters (a 3×3 filter on a 224×224 image uses 9 weights vs 224×224=50,176 for FC). (2) Translation equivariance — a cat detector in the top-left corner works in the bottom-right. (3) Sparse connectivity — each output neuron connects to only a small receptive field, not all inputs. These properties make CNNs highly efficient for spatial data.

Q7. What is the receptive field in a CNN?
The receptive field is the region of the input image that influences a particular neuron’s output. A single 3×3 convolutional layer has a receptive field of 3×3. Stacking two 3×3 layers gives an effective receptive field of 5×5. Three 3×3 layers give 7×7. This is why deep networks with small filters can capture large spatial context without the computational cost of large filters. Dilated (atrous) convolutions expand the receptive field without increasing parameters by inserting zeros between filter weights. Global Average Pooling collapses the entire feature map to a single value, giving a receptive field covering the entire input.

Q8. What is the difference between max pooling and average pooling?
Both reduce spatial dimensions (downsampling), but max pooling takes the maximum value in each pooling window while average pooling takes the mean. Max pooling: preserves the strongest feature activations; more robust to small spatial translations; creates sharper feature maps. Average pooling: smoother, loses less information; Global Average Pooling (GAP) averages each entire feature map to a single value and is standard before the final classification layer in modern CNNs (replacing large FC layers), significantly reducing parameters.

Q9. What are the key CNN architectures and what innovation did each introduce?
LeNet (1998): first practical CNN for digit recognition. AlexNet (2012): first deep CNN to win ImageNet — introduced ReLU, dropout, data augmentation, GPU training. VGG (2014): showed that depth with very small (3×3) filters beats large filters. GoogLeNet/Inception (2014): Inception modules that apply multiple filter sizes in parallel, then concatenate — efficient. ResNet (2015): residual connections (skip connections) allowing training of very deep networks (152+ layers) without vanishing gradients. DenseNet (2017): each layer connects to all subsequent layers. EfficientNet (2019): systematic scaling of width, depth, and resolution using neural architecture search. Vision Transformer (ViT, 2020): applies Transformer self-attention to image patches — now dominant for large-scale vision.

RNN and Sequence Model Questions

Q10. What is the vanishing gradient problem in RNNs and how do LSTMs solve it?
Plain RNNs process sequences step by step, multiplying the hidden state by the same weight matrix at each step. Over long sequences, gradients either vanish (product of values < 1 shrinks exponentially) or explode (product of values > 1 grows exponentially), making it impossible to learn long-range dependencies. LSTM (Long Short-Term Memory) introduces a cell state — a “memory highway” that runs through the entire sequence with only additive interactions (no repeated multiplication). Three gates control information flow: the forget gate decides what to discard from cell state; the input gate decides what new information to add; the output gate decides what to output. Because the cell state is updated additively, gradients can flow over long distances without vanishing. GRUs are a simpler alternative with two gates, fewer parameters, and comparable performance.

Transformer and Attention Mechanism

Q11. Explain the self-attention mechanism in Transformers.
Self-attention allows each token in a sequence to attend to all other tokens simultaneously, computing a weighted combination of all value vectors. For each token, three vectors are computed: Query (Q), Key (K), and Value (V) via learnable linear projections. Attention score for token i over token j = softmax(Qᵢ · Kⱼ / √d_k), where √d_k prevents dot products from growing too large in high dimensions. The output for token i is the weighted sum of all value vectors: output_i = Σ_j (attention_score_ij × V_j). Multi-head attention runs h parallel attention operations with different projections, then concatenates and projects their outputs. This allows the model to attend to information from different “representation subspaces” simultaneously.

Q12. What is the difference between BERT and GPT architectures?
Both are Transformer-based language models but with key architectural and training differences. BERT uses the Transformer encoder — bidirectional attention, each token attends to all other tokens in both directions. Trained with masked language modelling (predict masked tokens from context) and next sentence prediction. Best for understanding tasks: classification, NER, question answering. GPT uses the Transformer decoder — causal (left-to-right) attention, each token attends only to previous tokens. Trained with autoregressive language modelling (predict the next token). Best for generation tasks: text generation, summarisation, code completion. In 2026, instruction-tuned variants (fine-tuned with RLHF/DPO) of GPT-style models dominate general-purpose AI assistants.

Q13. What is positional encoding and why do Transformers need it?
Unlike RNNs, Transformers process all tokens in parallel — they have no inherent notion of sequence order. Positional encodings inject position information by adding a vector to each token embedding that encodes its position in the sequence. The original Transformer uses sinusoidal positional encodings: PE(pos, 2i) = sin(pos/10000^(2i/d_model)); PE(pos, 2i+1) = cos(pos/10000^(2i/d_model)). Learnable absolute positional embeddings are also common. Modern LLMs use Rotary Positional Embeddings (RoPE) or ALiBi — relative positional encodings that generalise better to sequence lengths not seen in training, which is why GPT-4 and Llama can handle long contexts.

Training Techniques and Debugging

Q14. What causes training loss to diverge and how do you fix it?
Symptoms: loss suddenly spikes to infinity (exploding gradients) or NaN. Causes and fixes: (1) Learning rate too high — most common cause. Fix: reduce by 10x. (2) Exploding gradients — gradients grow uncontrollably in RNNs and deep networks. Fix: gradient clipping (clip the gradient norm to a maximum value, typically 1.0). (3) Bad weight initialisation — Fix: use Xavier (Glorot) for tanh/sigmoid or He initialisation for ReLU. (4) Numerical instability — e.g., log(0). Fix: add small epsilon, use log-softmax instead of log(softmax(x)). (5) NaN in input data — Fix: check for NaN/Inf before training. Always start with a small learning rate, use gradient clipping for RNNs, and monitor gradient norms during training.

Q15. What is the difference between fine-tuning and training from scratch?
Training from scratch initialises weights randomly and trains on your dataset alone. It requires large amounts of data and compute, and is only appropriate when your domain differs substantially from available pretrained models. Fine-tuning starts from pretrained weights (ImageNet for vision, Wikipedia/books for NLP) and continues training on your task-specific dataset with a low learning rate. The pretrained weights encode general knowledge (features, grammar, world knowledge) that transfers to your task. Fine-tuning is almost always superior to training from scratch when pretrained models are available. Variants: full fine-tuning (update all parameters), linear probing (freeze pretrained layers, train only a new head), and parameter-efficient fine-tuning (LoRA, prefix tuning, adapters — update a small subset of parameters).

Q16. What is learning rate scheduling and what are common schedules?
Learning rate scheduling adjusts the learning rate during training. Starting high enables fast progress; reducing it later enables fine-grained convergence. Common schedules: Step decay — reduce by a factor every N epochs. Cosine annealing — reduces from max to min following a cosine curve, optionally with restarts (SGDR). Linear warmup + cosine decay — standard for Transformer training: gradually increase from 0 to max_lr over the first few thousand steps, then decay with cosine. OneCycleLR — increases then decreases in one cycle, often allows higher peak LR. Reduce on Plateau — reduces when validation loss stops improving (useful when you do not know the right schedule in advance).

Q17. What is knowledge distillation?
Knowledge distillation transfers knowledge from a large, accurate teacher model to a smaller, faster student model. The student is trained to match not just the hard labels but the teacher’s soft probability outputs — the full distribution over classes contains more information than a one-hot label (e.g., a model might output 80% cat, 18% leopard, 2% other — revealing that cats and leopards look similar). Loss = α × cross_entropy(hard_labels) + (1-α) × KL_divergence(student_probs, teacher_probs). The student, trained on these “dark knowledge” soft targets, often approaches teacher performance with 5-10x fewer parameters. Used to deploy models on mobile and edge devices. DistilBERT achieved 97% of BERT’s performance with 40% fewer parameters using distillation.

Q18–30 (Rapid fire deep learning):

Q18. What is gradient clipping? Rescaling the gradient vector to have a maximum L2 norm, preventing exploding gradients. Common threshold: 1.0. Standard practice in RNN and Transformer training.

Q19. What is weight initialisation and why does it matter? Poor initialisation causes vanishing/exploding activations before training begins. Xavier/Glorot (σ = √(2/(fan_in + fan_out))) for tanh/sigmoid. He initialisation (σ = √(2/fan_in)) for ReLU.

Q20. What is the difference between SGD and Adam? SGD updates all parameters with the same learning rate using the full gradient. Adam maintains per-parameter adaptive learning rates using exponential moving averages of gradients (m) and squared gradients (v). Adam converges faster with less tuning; SGD with momentum sometimes generalises slightly better for vision tasks.

Q21. What is an autoencoder? A neural network trained to reconstruct its input. An encoder compresses input to a latent representation; a decoder reconstructs input from latent. Applications: dimensionality reduction, anomaly detection (high reconstruction error = anomaly), denoising, and as pretraining for generative models.

Q22. What is a GAN and what is mode collapse? Generative Adversarial Network: a generator creates fake samples, a discriminator tries to distinguish real from fake. They compete until the generator produces indistinguishable samples. Mode collapse: the generator learns to produce only one or a few types of samples (ignoring diversity) because they consistently fool the discriminator.

Q23. What is a variational autoencoder (VAE)? An autoencoder with a probabilistic latent space. The encoder outputs a mean and variance; the latent vector is sampled from this distribution. The KL divergence loss regularises the latent space to be smooth and continuous, enabling interpolation and generation.

Q24. What is data augmentation and why does it help? Applying random transformations (flip, rotate, crop, colour jitter, cutout) to training images increases effective dataset size and teaches invariance to irrelevant variations. Reduces overfitting significantly for small datasets.

Q25. What is curriculum learning? Training on easy examples first, gradually introducing harder ones — mimicking human learning. Can improve convergence speed and final performance, particularly in NLP (short sentences first, then long) and RL.

Q26. What is LoRA (Low-Rank Adaptation)? A parameter-efficient fine-tuning method. Instead of updating the full weight matrix W, LoRA adds a low-rank decomposition ΔW = AB (where A and B are small matrices). Only A and B are trained, reducing trainable parameters by 10-1000x. Dominant method for fine-tuning LLMs.

Q27. What is RLHF? Reinforcement Learning from Human Feedback. After pretraining, a reward model is trained on human preference rankings. The language model is then fine-tuned with PPO (Proximal Policy Optimisation) to maximise the reward model’s score. Used to align ChatGPT, Claude, and Gemini.

Q28. What is the attention mask in Transformers? A binary mask that prevents attention to certain positions. Causal (decoder) mask: prevents attending to future tokens. Padding mask: prevents attending to padded positions in batches with variable-length sequences.

Q29. What is the difference between model parallelism and data parallelism? Data parallelism: same model replicated on multiple GPUs, each processes a different mini-batch, gradients are averaged. Model parallelism: the model itself is split across GPUs (too large to fit on one). Tensor parallelism splits individual layers. Pipeline parallelism splits layers into stages.

Q30. What is quantisation in deep learning? Reducing the numerical precision of model weights from 32-bit float to 16-bit, 8-bit (INT8), or 4-bit (INT4). Reduces model size and inference latency with minimal accuracy loss. INT8 quantisation typically loses < 1% accuracy. 4-bit quantisation (GPTQ, AWQ) enables running 70B parameter LLMs on a single consumer GPU.

Quick Reference Code

import torch
import torch.nn as nn

class TransformerBlock(nn.Module):
    def __init__(self, d_model=512, n_heads=8, d_ff=2048, dropout=0.1):
        super().__init__()
        self.attn  = nn.MultiheadAttention(d_model, n_heads, dropout=dropout, batch_first=True)
        self.ff    = nn.Sequential(nn.Linear(d_model, d_ff), nn.GELU(), nn.Linear(d_ff, d_model))
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        self.drop  = nn.Dropout(dropout)

    def forward(self, x, mask=None):
        # Pre-norm + residual (modern style)
        attn_out, _ = self.attn(x, x, x, attn_mask=mask)
        x = self.norm1(x + self.drop(attn_out))
        x = self.norm2(x + self.drop(self.ff(x)))
        return x

# Training loop essentials
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)
scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=100)

for epoch in range(100):
    model.train()
    for X, y in train_loader:
        optimizer.zero_grad()
        logits = model(X)
        loss   = nn.CrossEntropyLoss()(logits, y)
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)  # gradient clipping
        optimizer.step()
    scheduler.step()

Conclusion

Deep learning interviews in 2026 increasingly test knowledge of Transformers (self-attention, positional encoding, BERT vs GPT), LLM training (RLHF, fine-tuning, LoRA), and practical debugging skills (diagnosing diverging loss, fixing vanishing gradients). The best preparation is building models from scratch — implement a simple Transformer block, train an image classifier from scratch, and fine-tune a pretrained BERT for a classification task. Hands-on experience reveals gaps in understanding that reading alone cannot. Combine that with the theoretical depth in this guide, and you will be well-prepared for any deep learning interview role.