Neural Network Architectures Explained – MLP, CNN, RNN, LSTM, Attention and Transformers

The choice of neural network architecture is the most important design decision in deep learning. Using a CNN for text, an MLP for images, or an RNN for non-sequential data leads to dramatically inferior results — not because of poor hyperparameter tuning, but because the architectural inductive biases are fundamentally mismatched to the data structure. This guide explains every major neural network architecture, how it works, and when to use it — covering the full spectrum from MLPs to Transformers with practical interview Q&A throughout.

This guide is the theoretical companion to our Deep Learning Interview Q&A (40 interview questions with answers), our Computer Vision Interview Q&A (CNNs applied to images), our NLP Interview Q&A (Transformers applied to text), and our Time Series Forecasting Interview Q&A (LSTMs applied to sequential data). For RL-specific architectures (value networks, policy networks), see our Reinforcement Learning Explained guide.

Multi-Layer Perceptron (MLP) — The Foundation

The MLP (also called a fully connected or dense network) is the simplest neural network — stacked layers of neurons where each neuron in layer L is connected to every neuron in layer L+1. The forward pass: for each layer, compute Z = W·X + b (linear transformation), then apply an activation function A = f(Z) (non-linearity). Without activation functions, stacking linear layers produces another linear function — the whole network could be replaced by a single matrix multiplication. Non-linearities are what give neural networks their expressive power.

Activation functions and when to use each:

Backpropagation and gradient descent: Training neural networks via backpropagation computes the gradient of the loss with respect to every weight using the chain rule, then updates weights in the direction that decreases the loss (gradient descent). The vanishing gradient problem occurs when gradients become exponentially small as they are backpropagated through many layers — earlier layers receive near-zero gradient updates and fail to learn. Solutions: ReLU activations (gradients do not vanish in the positive region), residual connections (skip connections provide gradient highways), batch normalisation, and gradient clipping. Our Deep Learning Interview Q&A covers backpropagation, vanishing gradients, and optimisers (Adam, SGD, RMSprop) in detail.

Convolutional Neural Networks (CNN)

CNNs exploit the spatial structure of images through two inductive biases: local connectivity (each neuron connects only to a local region — a 3×3 patch — not the full image) and weight sharing (the same filter is applied at every location, detecting the same feature regardless of where it appears in the image). These biases make CNNs dramatically more parameter-efficient than MLPs for image data.

Key CNN components:

Convolutional layer: applies K filters of size F×F×C (where C is input channels), each producing a feature map detecting a specific pattern (edge, texture, shape). Strided convolution (stride > 1) downsamples the spatial dimensions. Padding (same/valid) controls whether output size matches input size. Pooling layer: Max pooling takes the maximum value in each local region — provides translation invariance and reduces spatial dimensions. Average pooling is used in some architectures (global average pooling replaces the final dense layer in modern CNNs, reducing parameters dramatically). Batch Normalisation: normalises activations within a mini-batch, stabilising training, enabling higher learning rates, and acting as a regulariser. Now standard in every CNN layer.

CNN architecture evolution and when to use each:

For CNN applications to computer vision tasks (object detection with YOLO, segmentation with U-Net), see our comprehensive Computer Vision Interview Q&A.

Recurrent Neural Networks — RNN, LSTM, GRU

Recurrent neural networks process sequential data by maintaining a hidden state that is updated at each time step: hₜ = f(Wxₜ + Uhₜ₋₁ + b). The hidden state acts as memory — it carries information from previous time steps to influence current predictions. Unlike CNNs (which process all inputs simultaneously) and MLPs (which treat each input independently), RNNs naturally handle variable-length sequences and temporal dependencies.

The vanishing gradient problem in RNNs: During backpropagation through time (BPTT), gradients are multiplied by the recurrent weight matrix at each time step. If the largest eigenvalue of this matrix is < 1, gradients shrink exponentially over long sequences — the network cannot learn dependencies spanning more than ~10 steps. If > 1, gradients explode (handled by gradient clipping). This fundamental limitation motivated the development of LSTM.

LSTM (Long Short-Term Memory, 1997): Introduces a cell state cₜ — a separate memory channel with additive (not multiplicative) updates — alongside three gating mechanisms that control information flow:

Forget gate fₜ = σ(Wf·[hₜ₋₁, xₜ] + bf): decides what information to erase from cell state. A value near 1 means “keep this”, near 0 means “forget this.” Input gate iₜ = σ(Wi·[hₜ₋₁, xₜ] + bi): decides which new information to add. Candidate cell g̃ₜ = tanh(Wg·[hₜ₋₁, xₜ] + bg): the new candidate values. Cell update: cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ g̃ₜ. Output gate oₜ = σ(Wo·[hₜ₋₁, xₜ] + bo): controls what part of cell state becomes the hidden state. The additive cell update cₜ = fₜ ⊙ cₜ₋₁ + … provides a gradient highway that mitigates vanishing gradients — gradients can flow back through the cell state with minimal attenuation.

GRU (Gated Recurrent Unit, 2014): A simplified LSTM with two gates instead of three (reset gate and update gate), merging the cell state and hidden state. GRU has ~25% fewer parameters than LSTM and trains slightly faster, with comparable performance on most tasks. Use LSTM when sequences are very long and memory capacity matters; use GRU for shorter sequences or when training speed is critical. Both are largely superseded by Transformers for NLP — see our NLP Interview Q&A — but remain relevant for time series forecasting as covered in our Time Series Interview Q&A.

import torch
import torch.nn as nn

class LSTMForecaster(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, output_size, dropout=0.2):
        super().__init__()
        self.lstm = nn.LSTM(
            input_size=input_size,
            hidden_size=hidden_size,
            num_layers=num_layers,
            batch_first=True,       # input shape: (batch, seq_len, features)
            dropout=dropout if num_layers > 1 else 0,
            bidirectional=False
        )
        self.dropout = nn.Dropout(dropout)
        self.fc      = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        # x: (batch, seq_len, input_size)
        out, (h_n, c_n) = self.lstm(x)
        # Use last hidden state for prediction
        out = self.dropout(out[:, -1, :])   # (batch, hidden_size)
        return self.fc(out)                  # (batch, output_size)

model = LSTMForecaster(input_size=10, hidden_size=64, num_layers=2, output_size=1)
print(f'Parameters: {sum(p.numel() for p in model.parameters()):,}')

Attention Mechanism and the Transformer

Attention (Bahdanau et al., 2014) was introduced to solve a fundamental limitation of encoder-decoder RNNs: the entire input sequence must be compressed into a single fixed-size context vector, which becomes a bottleneck for long sequences. Attention allows the decoder to directly “look at” relevant parts of the encoder’s hidden states when generating each output token.

Scaled Dot-Product Attention — the core operation:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) · V

Q (Queries): what we are looking for. K (Keys): what each position offers. V (Values): the content to retrieve. QKᵀ computes compatibility scores — how relevant each key is to each query. Dividing by √dₖ prevents the dot products from growing large (which would push softmax into saturation). Softmax converts scores to attention weights (sum to 1). Multiplying by V retrieves a weighted sum of values — the attended representation.

Multi-Head Attention: Instead of one attention function, run h=8 or h=16 attention operations in parallel, each in a lower-dimensional subspace (dmodel/h). This allows the model to simultaneously attend to information from different representation subspaces — one head might capture syntactic relationships while another captures semantic similarity. Outputs are concatenated and linearly projected.

The Transformer Architecture (Vaswani et al., 2017): “Attention Is All You Need” replaced recurrence entirely with self-attention. The encoder processes the full input sequence in parallel (no sequential dependency), applying self-attention where each token attends to all other tokens — capturing global dependencies with O(1) path length between any two positions (vs O(n) for RNNs). Positional encodings (sinusoidal or learned) are added to embeddings to inject positional information lost by the non-sequential attention operation. The decoder uses masked self-attention (cannot attend to future tokens during training), cross-attention to the encoder, and autoregressive generation.

For BERT, GPT, and other Transformer-based language models, see our NLP Interview Q&A. For Vision Transformers (ViT) and how Transformers are applied to images, see our Computer Vision Interview Q&A.

Architecture Selection Guide

Choosing the right architecture for your task is one of the most important decisions in applied deep learning — and a frequent interview question:

Key Interview Q&A — Neural Architectures

Q: What problem do residual connections (ResNet) solve? Residual connections add the input directly to the output of a layer: F(x) + x. This allows gradients to flow directly through the skip connection during backpropagation without passing through the layer’s weights — solving the vanishing gradient problem in very deep networks (50-200 layers). The layer only needs to learn the “residual” difference F(x) = H(x) – x rather than the full desired mapping H(x), which is easier. Our Deep Learning Q&A covers ResNet in detail including the mathematical justification.

Q: Why do Transformers outperform RNNs for NLP? Three key advantages: (1) Parallelism — self-attention processes all tokens simultaneously, enabling GPU utilisation and faster training vs sequential RNN processing. (2) Long-range dependencies — attention directly connects any two tokens with O(1) operations; RNNs must carry information through O(n) sequential steps (vanishing gradient risk). (3) Scalability — Transformers scale effectively with more data and parameters (GPT-3: 175B parameters), while RNNs are harder to scale.

Q: What is the difference between encoder-only, decoder-only, and encoder-decoder Transformers? Encoder-only (BERT): bidirectional attention — each token sees all other tokens. Best for understanding tasks: classification, NER, question answering over a given context. Decoder-only (GPT): causal (masked) attention — each token only sees previous tokens. Best for generation: text completion, code generation, conversational AI. Encoder-decoder (T5, BART): encoder processes input bidirectionally; decoder generates output autoregressively using cross-attention to the encoder. Best for sequence-to-sequence tasks: translation, summarisation, abstractive QA.

Q: How do you handle the O(n²) memory cost of attention for long sequences? Flash Attention (Dao et al., 2022): reorders attention computation to avoid materialising the full n×n attention matrix in GPU HBM — achieves same result in O(n) memory. Sliding window attention (Longformer): attend only within a window of nearby tokens + a few global tokens. Sparse attention (BigBird): structured sparsity patterns. Linear attention approximations (Performer). These are covered in our NLP Interview Q&A.

Q: What regularisation techniques are specific to neural networks? Dropout: randomly zeroes activations during training (p=0.1-0.5), forcing redundant representations. Disabled at inference. Batch Normalisation: normalises activations, acts as an implicit regulariser (and enables higher LR). Weight Decay (L2 on weights): penalises large weights, equivalent to a Gaussian prior on weights. Early Stopping: monitor validation loss and stop when it starts increasing. Data Augmentation: generates additional training examples (for images, text, time series). These complement the general regularisation approaches discussed in our Model Evaluation guide.

Understanding neural network architectures at this depth — not just what they are but why they work, what problems they solve, and when not to use them — is what distinguishes candidates who pass deep learning interviews from those who impress. For hands-on practice, implement a small CNN for image classification (PyTorch torchvision), fine-tune BERT for sentiment analysis (Hugging Face), and build an LSTM forecaster (as shown above) — these three projects cover the spectrum of architectures that appear in ML engineering interviews.