Dimensionality Reduction Explained – PCA, t-SNE, UMAP and Autoencoders

High-dimensional data is the norm in modern machine learning — genomics datasets with 20,000 gene features, NLP embeddings with 768 dimensions, image datasets with millions of pixels. Dimensionality reduction compresses this into a lower-dimensional representation while preserving the structure that matters most. It is essential for visualisation, denoising, speeding up downstream models, and understanding latent data structure. This guide explains PCA, t-SNE, UMAP, and autoencoders with intuition and maths needed for both practice and interviews.

If you’re preparing for interviews, our Machine Learning Interview Q&A includes dimensionality reduction questions, and our Feature Engineering Interview Q&A covers when to apply these techniques as preprocessing steps.

Why Dimensionality Reduction?

The curse of dimensionality: As dimensionality increases, data becomes increasingly sparse. Euclidean distance becomes meaningless — all points are approximately equidistant, destroying the concept of nearest neighbours. A k-NN classifier on 100D data may need exponentially more training examples than the same classifier on 10D data.

When it helps: Visualisation (compress to 2D/3D for human inspection), feature extraction (compact representation), noise reduction (principal components of noise have small variance), computational efficiency (training SVM on 50 features vs 5000 is dramatically faster), and storage/bandwidth savings.

Principal Component Analysis (PCA)

PCA finds the directions of maximum variance in the data and projects onto them. It is linear — new features (principal components) are linear combinations of the original features.

How PCA works step by step:

1. Standardise the data (subtract mean, divide by std per feature). 2. Compute the covariance matrix of standardised data. 3. Compute eigendecomposition — eigenvectors = principal components, eigenvalues = variance explained by each PC. 4. Sort eigenvectors by eigenvalue, descending. 5. Project data onto the top k eigenvectors.

How to choose k? Plot explained variance ratio vs k (scree plot). Choose k where the plot elbows, or where cumulative explained variance reaches 95%.

from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

pca_full = PCA().fit(X_scaled)

cumvar = pca_full.explained_variance_ratio_.cumsum()
plt.figure(figsize=(8, 4))
plt.plot(range(1, len(cumvar)+1), cumvar, marker='o')
plt.axhline(0.95, color='r', linestyle='--', label='95% threshold')
plt.xlabel('Number of Components')
plt.ylabel('Cumulative Explained Variance')
plt.title('PCA Scree Plot')
plt.legend(); plt.tight_layout(); plt.show()

k = (cumvar < 0.95).sum() + 1
pca = PCA(n_components=k)
X_reduced = pca.fit_transform(X_scaled)
print(f'Reduced from {X.shape[1]}D to {k}D — {cumvar[k-1]*100:.1f}% variance retained')

Limitations of PCA: Linear only — cannot capture non-linear manifolds. Sensitive to scale (always standardise first). Principal components may not be interpretable. Assumes high-variance directions are the most informative — true for many tasks, not all.

t-SNE

t-SNE (van der Maaten & Hinton, 2008) is a non-linear technique designed specifically for visualisation. It excels at revealing cluster structure that PCA cannot separate. In high-D space: define probability P where close points have high p_{ij}. In low-D: define Q using a Student's t-distribution (heavy tails solve the crowding problem). Minimise KL divergence between P and Q. The result: the 2D layout matches neighbourhood structure of the high-D data.

Perplexity: The single most important hyperparameter — controls effective number of neighbours. Low (5-10): very local structure. High (50-100): more global. Try multiple values (5, 30, 50) for every dataset.

Critical limitations — interviewers test this: Global structure is NOT preserved — distances between clusters in a t-SNE plot are meaningless. Cluster sizes are meaningless — t-SNE expands dense clusters and compresses sparse ones. Not reproducible without fixing random seed. Very slow for large datasets. Cannot embed new points without re-running. t-SNE is for visualisation only — never use its embeddings as features for downstream models.

UMAP

UMAP (McInnes et al., 2018) is the modern replacement for t-SNE — based on Riemannian geometry and topological data analysis.

How UMAP differs from t-SNE: Preserves both local AND global structure better. Relative distances between clusters are more meaningful. Dramatically faster (minutes vs hours for 100K points). Supports out-of-sample embedding (transform new data without refitting). Can embed into any number of dimensions — suitable for feature extraction, not just visualisation.

import umap

reducer = umap.UMAP(
    n_components=2,      # 2D for visualisation; 10-50 for feature extraction
    n_neighbors=15,      # local neighbourhood size
    min_dist=0.1,        # minimum distance between points in embedding
    metric='euclidean',  # or 'cosine' for text embeddings
    random_state=42
)
X_umap = reducer.fit_transform(X_scaled)

# Embed new points without refitting
# X_new_umap = reducer.transform(X_new_scaled)

n_neighbors: Small (5-10) → fine local structure. Large (50-200) → more global structure. min_dist: Small (0.0-0.1) → tight clusters. Large (0.5-1.0) → spread-out topology.

Autoencoders

An autoencoder compresses data to a low-dimensional bottleneck (encoder) and reconstructs from it (decoder). The bottleneck activations are the learned embeddings.

import torch
import torch.nn as nn

class Autoencoder(nn.Module):
    def __init__(self, input_dim, latent_dim):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(input_dim, 256), nn.ReLU(),
            nn.Linear(256, 64),        nn.ReLU(),
            nn.Linear(64, latent_dim)
        )
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 64), nn.ReLU(),
            nn.Linear(64, 256),        nn.ReLU(),
            nn.Linear(256, input_dim), nn.Sigmoid()
        )
    def forward(self, x):
        z = self.encoder(x)
        return self.decoder(z), z

Advantages over PCA: captures non-linear relationships, handles any data type. Disadvantage: requires training. Variational Autoencoders (VAE) encode to a distribution (mean + std), regularise the latent space to be Gaussian — used for anomaly detection and generation.

Choosing the Right Method

PCA: default first choice — fast, deterministic, interpretable. Use for preprocessing before ML models, noise reduction, multicollinearity. t-SNE: exploratory visualisation only — never use embeddings downstream. UMAP: superior to t-SNE for visualisation (faster, better global structure) and suitable for feature extraction. Autoencoders: when PCA is insufficient and you have enough data. Kernel PCA: non-linear extension of PCA via the kernel trick — middle ground between PCA and neural autoencoders.

The most common interview trap: "Can you interpret the distances between clusters in a t-SNE plot?" The answer is no — and knowing why demonstrates deeper understanding than just being able to run the code.