High-dimensional data is everywhere in machine learning — image pixels, word embeddings, sensor readings. Dimensionality reduction compresses that data into fewer dimensions while preserving the structure that matters. This guide covers the three most important techniques: PCA, t-SNE, and UMAP.
Why Dimensionality Reduction?
Working with high-dimensional data creates several problems. The curse of dimensionality means that as features increase, the data becomes increasingly sparse — distances lose meaning, and models need exponentially more data to generalise. Reducing dimensions speeds up training, reduces overfitting, and makes visualization possible (humans can only see 2D or 3D). It also removes redundant or noisy features, which often improves model performance.
PCA – Principal Component Analysis
PCA is a linear technique that finds the directions (principal components) of maximum variance in the data and projects it onto those axes.
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import numpy as np
# Always scale before PCA
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
# Reduce to 2 components for visualization
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)
print("Variance explained:", pca.explained_variance_ratio_)
print("Total variance retained:", pca.explained_variance_ratio_.sum())
A common workflow is to plot the cumulative explained variance to choose the right number of components:
pca_full = PCA()
pca_full.fit(X_scaled)
import matplotlib.pyplot as plt
plt.plot(np.cumsum(pca_full.explained_variance_ratio_))
plt.xlabel("Number of components")
plt.ylabel("Cumulative explained variance")
plt.axhline(0.95, color='r', linestyle='--', label='95% threshold')
plt.legend()
plt.show()
When to Use PCA
PCA works best when the relationships in your data are approximately linear. Use it for preprocessing before a linear model, for reducing collinearity between features, and for compressing image data. Its limitations: it only captures linear structure, so non-linear manifolds (like the Swiss roll dataset) are poorly represented.
t-SNE – t-Distributed Stochastic Neighbor Embedding
t-SNE is a non-linear technique designed specifically for visualization. It places similar points close together in 2D/3D space by minimising the KL-divergence between high-dimensional and low-dimensional probability distributions.
from sklearn.manifold import TSNE
tsne = TSNE(n_components=2, perplexity=30, random_state=42, n_iter=1000)
X_tsne = tsne.fit_transform(X_scaled)
import matplotlib.pyplot as plt
plt.scatter(X_tsne[:, 0], X_tsne[:, 1], c=y, cmap='tab10', alpha=0.6)
plt.colorbar()
plt.title("t-SNE 2D Embedding")
plt.show()
Key hyperparameter: perplexity (5–50) controls the balance between local and global structure. Try several values.
t-SNE Limitations
t-SNE is slow (O(n²) without approximations), doesn’t scale beyond ~50k samples easily, and — critically — distances between clusters in a t-SNE plot are NOT meaningful. Cluster sizes and inter-cluster gaps are artefacts of the perplexity setting. Use t-SNE for visualization only, never as a preprocessing step for a model.
UMAP – Uniform Manifold Approximation and Projection
UMAP is the current state-of-the-art for both visualization and preprocessing. It’s much faster than t-SNE, scales to millions of points, and better preserves global structure.
pip install umap-learn
import umap
reducer = umap.UMAP(n_components=2, n_neighbors=15, min_dist=0.1, random_state=42)
X_umap = reducer.fit_transform(X_scaled)
plt.scatter(X_umap[:, 0], X_umap[:, 1], c=y, cmap='tab10', alpha=0.6)
plt.title("UMAP 2D Embedding")
plt.show()
Key hyperparameters: n_neighbors (5–50) controls local vs global balance; min_dist (0.0–1.0) controls how tightly points are packed in the embedding.
PCA vs t-SNE vs UMAP — Quick Comparison
PCA is linear, interpretable, fast, and reversible — ideal as a preprocessing step and for datasets where linear structure dominates. t-SNE captures non-linear clusters beautifully but is slow and only suitable for visualization. UMAP is fast, non-linear, preserves both local and global structure, and can be used both for visualization and as a preprocessing step before clustering or classification. In 2026, UMAP has largely replaced t-SNE for most practitioners except in legacy codebases.
Practical Workflow
A common production workflow is to apply PCA first to reduce to 50 dimensions (removing noise and speeding things up), then UMAP to reduce to 2-10 dimensions for visualization or further modeling. This PCA→UMAP pipeline is used by teams at Spotify and Google for embedding visualization.
Conclusion
Dimensionality reduction is a foundational skill for any data scientist working with complex, high-dimensional data. Start with PCA for linear data and interpretability, use t-SNE for quick cluster visualizations, and reach for UMAP when you need speed, scalability, and the ability to handle non-linear structure. All three are available in Python with just a few lines of code.
