Most real-world datasets have a hidden structure problem: they contain far more features than they need. Customer behaviour datasets might have 200 columns, but many are highly correlated — customers who buy product A almost always buy product B. Gene expression datasets might have 50,000 features, but genes that belong to the same biological pathway rise and fall together. You are carrying redundant information everywhere.
Principal Component Analysis (PCA) is the standard solution to this problem. It identifies the underlying structure of your data — the directions where genuine variation lives — and gives you a compressed representation that preserves as much information as possible in fewer dimensions. This speeds up model training, often improves accuracy by reducing noise, and makes visualisation of high-dimensional data possible.
Understanding PCA properly requires understanding what “variance” means in a dataset and why the directions of maximum variance are meaningful. This guide builds that understanding step by step before moving into Python implementation.
What Is Variance Telling Us?
Variance measures spread. A feature with high variance is doing something interesting — it is taking different values across your samples, which means it carries information. A feature with zero variance is constant across all samples and tells you nothing at all about the differences between them.
Now extend this to two dimensions. If you plot two features on a scatter plot, you can ask: what direction in this 2D space has the most spread? What direction captures the most variation across all your data points? That direction is your first principal component. It is a weighted combination of your original features, chosen specifically to capture as much information as possible.
The second principal component is the direction with the most remaining variance, constrained to be perpendicular to the first. The third is perpendicular to the first two. This continues until you have as many components as original features — but the first few components typically capture 90-95% of all the variance. The rest is mostly noise that you can safely discard.
What PCA Actually Does to Your Data
PCA performs a rotation of your data. It finds a new coordinate system — the principal components — where the axes point in the directions of maximum variance. When you transform your data into this new space, the first coordinate (PC1) captures the most variation, the second captures the next most, and so on.
This is why PCA requires feature scaling first. If one feature has values in the millions and another in the range 0–1, the large-scale feature dominates the variance calculation entirely. PCA would essentially ignore the small-scale features. StandardScaler normalises all features to the same scale before PCA runs, ensuring every feature gets a fair chance to contribute to the components.
Python Implementation: Step by Step
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
import matplotlib.pyplot as plt
import numpy as np
# Load data
X, y = load_breast_cancer(return_X_y=True)
print(f'Original shape: {X.shape}') # (569, 30) — 30 features
# Step 1: Scale before PCA — mandatory
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
# Step 2: Fit PCA on all components first to inspect variance
pca_full = PCA()
pca_full.fit(X_scaled)
# Step 3: Plot explained variance
exp_var = pca_full.explained_variance_ratio_
cum_var = np.cumsum(exp_var)
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
axes[0].bar(range(1, 11), exp_var[:10], color='steelblue', alpha=0.8)
axes[0].set_xlabel('Principal Component')
axes[0].set_ylabel('Proportion of Variance Explained')
axes[0].set_title('Individual Explained Variance')
axes[1].plot(range(1, len(cum_var)+1), cum_var, 'bo-', markersize=4)
axes[1].axhline(y=0.95, color='red', linestyle='--', label='95% threshold')
axes[1].axhline(y=0.99, color='orange', linestyle='--', label='99% threshold')
axes[1].set_xlabel('Number of Components')
axes[1].set_ylabel('Cumulative Variance Explained')
axes[1].set_title('How Many Components Do We Need?')
axes[1].legend()
plt.tight_layout()
plt.show()
n_95 = np.argmax(cum_var >= 0.95) + 1
n_99 = np.argmax(cum_var >= 0.99) + 1
print(f'Components for 95% variance: {n_95}')
print(f'Components for 99% variance: {n_99}')
print(f'Reduction from 30 features to {n_95} captures 95% of information')
# Step 4: Apply PCA and compare model performance
pca = PCA(n_components=n_95)
X_reduced = pca.fit_transform(X_scaled)
print(f'
Reduced shape: {X_reduced.shape}')
rf = RandomForestClassifier(n_estimators=100, random_state=42)
acc_original = cross_val_score(rf, X_scaled, y, cv=5, scoring='roc_auc').mean()
acc_pca = cross_val_score(rf, X_reduced, y, cv=5, scoring='roc_auc').mean()
print(f'
ROC-AUC — Original 30 features: {acc_original:.4f}')
print(f'ROC-AUC — After PCA ({n_95} components): {acc_pca:.4f}')
Visualising High-Dimensional Data with PCA
One of the most common uses of PCA is collapsing any number of features down to 2 components for a scatter plot. This gives you an immediate visual sense of how well separated your classes are:
pca_2d = PCA(n_components=2)
X_2d = pca_2d.fit_transform(X_scaled)
plt.figure(figsize=(9, 6))
colors = ['coral', 'steelblue']
labels = ['Malignant', 'Benign']
for cls, (color, label) in enumerate(zip(colors, labels)):
mask = y == cls
plt.scatter(X_2d[mask, 0], X_2d[mask, 1],
c=color, label=label, alpha=0.6, s=40, edgecolors='none')
pct = pca_2d.explained_variance_ratio_ * 100
plt.xlabel(f'PC1 ({pct[0]:.1f}% of variance)', fontsize=12)
plt.ylabel(f'PC2 ({pct[1]:.1f}% of variance)', fontsize=12)
plt.title('Breast Cancer Dataset — 2D PCA Projection', fontsize=14)
plt.legend(fontsize=11)
plt.tight_layout()
plt.show()
If the two classes separate clearly in this 2D projection, your classification task is tractable. If they overlap heavily, you can expect all classifiers to struggle.
Understanding Component Loadings
Each principal component is a weighted combination of your original features. The loadings tell you which original features contribute most to each component — useful for interpreting what each component represents:
import pandas as pd
feature_names = load_breast_cancer().feature_names
# PC1 loadings
pc1_loadings = pd.Series(pca_full.components_[0], index=feature_names)
pc1_loadings.abs().sort_values(ascending=False).head(10).plot(
kind='barh', figsize=(8, 5)
)
plt.title('Features Contributing Most to PC1')
plt.xlabel('|Loading|')
plt.tight_layout()
plt.show()
When PCA Helps and When It Hurts
PCA adds the most value when your features are highly correlated — check this with a correlation heatmap first. If features are mostly independent, PCA compresses little and gains nothing. PCA also only captures linear relationships. Non-linear structure in your data requires non-linear methods like UMAP or t-SNE for visualisation, or kernel PCA for preprocessing.
The most important limitation: PCA destroys interpretability. After transformation, you cannot say “feature income was important.” Your components are weighted combinations of all original features. If you ever need to explain which specific input variables drove a decision — common in regulated industries — PCA is incompatible with that requirement. Use feature selection or tree-based importance instead.
Frequently Asked Questions
How many principal components should I keep?
The standard approach is to keep enough components to explain 95% of cumulative variance. This is not a hard rule — sometimes 90% gives a smaller model with nearly the same accuracy, and the noise reduction from discarding the last 10% of variance actually improves performance. Always compare model accuracy with different numbers of components using cross-validation. For visualisation specifically, use exactly 2 components (for scatter plots) or 3 components (for 3D plots), regardless of how much variance they explain.
Should I always apply PCA before training a machine learning model?
No — this is one of the most common misunderstandings about PCA. Tree-based models (Random Forest, XGBoost, LightGBM) handle high-dimensional, correlated features very well on their own. Applying PCA before these models often hurts performance and always hurts interpretability. Reserve PCA for linear models (logistic regression, linear SVM), neural networks where reducing input size speeds up training, and k-nearest neighbours where high dimensionality causes the “curse of dimensionality” problem. When in doubt, compare model performance with and without PCA using cross-validation.
What is the difference between PCA, t-SNE, and UMAP?
PCA is a linear dimensionality reduction method designed for both preprocessing and visualisation. It preserves global structure — points far apart in the original space stay far apart after PCA. t-SNE and UMAP are non-linear methods designed specifically for visualisation. They excel at revealing local cluster structure that PCA misses, because they optimise for preserving relationships between nearby points rather than global variance. However, t-SNE and UMAP coordinates are not meaningful for training downstream models — they are purely visual tools. In 2026, UMAP is generally preferred over t-SNE for visualisation because it is faster, scales to larger datasets, and better preserves global structure alongside local clusters.
My PCA model was fit on training data — how do I apply it to test data correctly?
Always fit PCA on training data only, then use the fitted PCA to transform both training and test data. Never fit PCA on the combined training plus test data — this would leak information from the test set into your transformation and invalidate your evaluation. In scikit-learn, this means: pca.fit_transform(X_train) then pca.transform(X_test). The same rule applies to the StandardScaler you use before PCA. This is best handled by wrapping both steps in a scikit-learn Pipeline, which enforces correct fit/transform separation automatically.
