You trained a classification model. Accuracy is 95%. Your manager is thrilled. But when you dig deeper, you find the model predicted “not fraud” for every single transaction — and 95% of transactions genuinely are not fraud. Accuracy is 95%, but your fraud detector has caught exactly zero frauds.
This scenario, real enough to have destroyed actual ML projects, illustrates why accuracy alone is a dangerously incomplete metric for most classification problems. The confusion matrix and the metrics derived from it — precision, recall, and F1 score — give you the full picture. This guide explains each concept clearly, shows you when to use which metric, and walks through Python implementation.
What Is a Confusion Matrix?
A confusion matrix is a table that summarises a classification model’s predictions against the actual labels. For binary classification (two classes: positive and negative), it has four cells representing four possible outcomes.
True Positives (TP): The model predicted positive and it actually was positive. Correct fraud predictions. Correct cancer detections. These are what you want to maximise.
True Negatives (TN): The model predicted negative and it actually was negative. Correctly cleared transactions. Correctly identified healthy patients. Also correct predictions.
False Positives (FP): The model predicted positive but it was actually negative. Flagged a legitimate transaction as fraud. Diagnosed a healthy patient with cancer. Also called a Type I error.
False Negatives (FN): The model predicted negative but it was actually positive. Missed a real fraud. Missed a real cancer. Also called a Type II error. In high-stakes applications, this is often the most dangerous error type.
from sklearn.metrics import (confusion_matrix, classification_report,
accuracy_score, precision_score,
recall_score, f1_score, roc_auc_score)
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
# Create imbalanced dataset (5% positive class — like fraud)
X, y = make_classification(n_samples=10000, n_features=20,
weights=[0.95, 0.05], random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y
)
clf = RandomForestClassifier(n_estimators=100, random_state=42)
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
# Confusion matrix
cm = confusion_matrix(y_test, y_pred)
print('Confusion Matrix:')
print(cm)
print(f'
TN={cm[0,0]} FP={cm[0,1]}')
print(f'FN={cm[1,0]} TP={cm[1,1]}')
# Visualise
plt.figure(figsize=(7, 5))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues',
xticklabels=['Predicted Negative', 'Predicted Positive'],
yticklabels=['Actual Negative', 'Actual Positive'])
plt.title('Confusion Matrix', fontsize=13)
plt.ylabel('Actual'); plt.xlabel('Predicted')
plt.tight_layout(); plt.show()
The Four Key Metrics — Explained Simply
Accuracy = (TP + TN) / Total. The percentage of all predictions that were correct. Intuitive but misleading on imbalanced datasets — a model that predicts the majority class for everything will have high accuracy while being completely useless for detecting the minority class.
Precision = TP / (TP + FP). Of all the times the model predicted positive, what fraction were actually positive? Precision answers the question: “When the model raises an alarm, how often is it right?” High precision means few false alarms. This is critical when false positives are costly — sending promotional emails to non-interested customers, flagging legitimate transactions as fraud, or making unnecessary medical referrals.
Recall (Sensitivity) = TP / (TP + FN). Of all the actual positives, what fraction did the model find? Recall answers: “What fraction of the real positives did the model catch?” High recall means few missed cases. This is critical when false negatives are costly — missing a cancer diagnosis, failing to detect a cyberattack, not flagging a loan default.
F1 Score = 2 × (Precision × Recall) / (Precision + Recall). The harmonic mean of precision and recall. It balances both concerns and is useful when you need a single metric for an imbalanced classification problem. Unlike the arithmetic mean, the harmonic mean is dragged down by whichever of the two is lower — so a model cannot achieve a high F1 by optimising one at the expense of the other.
print(f'Accuracy: {accuracy_score(y_test, y_pred):.4f}')
print(f'Precision: {precision_score(y_test, y_pred):.4f}')
print(f'Recall: {recall_score(y_test, y_pred):.4f}')
print(f'F1 Score: {f1_score(y_test, y_pred):.4f}')
print(f'ROC-AUC: {roc_auc_score(y_test, clf.predict_proba(X_test)[:,1]):.4f}')
print()
print(classification_report(y_test, y_pred,
target_names=['Not Fraud', 'Fraud']))
The Precision-Recall Trade-off
Precision and recall are in tension. Increasing one typically decreases the other. You can control this trade-off by adjusting the classification threshold — the probability cutoff above which the model predicts positive.
from sklearn.metrics import precision_recall_curve
y_proba = clf.predict_proba(X_test)[:, 1]
precisions, recalls, thresholds = precision_recall_curve(y_test, y_proba)
plt.figure(figsize=(9, 5))
plt.plot(thresholds, precisions[:-1], label='Precision', color='steelblue', linewidth=2)
plt.plot(thresholds, recalls[:-1], label='Recall', color='coral', linewidth=2)
plt.axvline(0.5, color='grey', linestyle='--', alpha=0.5, label='Default threshold (0.5)')
plt.xlabel('Threshold'); plt.ylabel('Score')
plt.title('Precision-Recall Trade-off vs Threshold')
plt.legend(); plt.tight_layout(); plt.show()
# Find threshold that maximises F1
f1_scores = 2 * precisions[:-1] * recalls[:-1] / (precisions[:-1] + recalls[:-1] + 1e-9)
best_thresh = thresholds[np.argmax(f1_scores)]
print(f'Best threshold for F1: {best_thresh:.3f}')
print(f'Precision at best threshold: {precisions[np.argmax(f1_scores)]:.4f}')
print(f'Recall at best threshold: {recalls[np.argmax(f1_scores)]:.4f}')
When to Prioritise Which Metric
The right metric depends on what kind of mistake is more costly in your application. In medical diagnosis, a missed cancer (false negative) is far more dangerous than a false alarm (false positive) — you optimise for recall even at the cost of precision. In spam filtering, sending legitimate emails to spam (false positive) is more frustrating than letting spam through — precision matters more. In most fraud detection systems, both are important but recall is weighted more heavily — a missed fraud is costlier than an incorrectly flagged legitimate transaction.
ROC-AUC (Area Under the ROC Curve) is a threshold-independent metric that measures the model’s overall ability to discriminate between classes across all possible thresholds. It ranges from 0.5 (random guessing) to 1.0 (perfect). It is the standard metric for comparing models on imbalanced datasets when you have not yet decided on an operating threshold. See how we use it in our hyperparameter tuning guide.
Frequently Asked Questions
When is accuracy a reliable metric?
Accuracy is reliable only when your dataset is roughly balanced — when each class has a similar number of examples. If you are classifying dog breeds from 100 images each of 10 breeds, accuracy is a fair measure. If you are detecting rare diseases where 99% of patients are healthy, accuracy is misleading — a model that always predicts “healthy” achieves 99% accuracy while being completely useless. The rule of thumb: when your minority class makes up less than 20% of the data, use precision, recall, F1, or ROC-AUC instead of accuracy. Always look at the full confusion matrix, not just the overall accuracy score.
What is the difference between micro, macro, and weighted F1 score?
These terms apply when you have more than two classes. Micro F1 calculates TP, FP, and FN across all classes combined before computing the F1 — it gives more weight to classes with more samples. Macro F1 computes F1 for each class independently and then takes the unweighted average — it treats all classes equally regardless of size. Weighted F1 computes F1 per class and takes a weighted average by class support (number of true instances per class). For imbalanced datasets, macro F1 is often most informative because it highlights poor performance on minority classes that weighted F1 would underweight. Scikit-learn’s classification_report shows all three automatically.
What is ROC-AUC and when should I use it instead of F1?
ROC-AUC measures a model’s discriminative ability across all possible classification thresholds, making it threshold-independent. It is the right choice when you want to compare two models’ overall performance without committing to a specific operating threshold. F1 is threshold-dependent — it is calculated at a specific cutoff (usually 0.5) and tells you performance at that specific operating point. Use ROC-AUC for model comparison and selection. Once you have chosen your best model and decided on your business constraints, use the precision-recall threshold curve to find the optimal threshold for your use case, then report the F1 at that threshold as your production performance metric.
How do I handle the confusion matrix for multi-class problems?
For N classes, the confusion matrix becomes an N×N table where each row represents the actual class and each column represents the predicted class. The diagonal cells are correct predictions; all off-diagonal cells are errors. Reading the confusion matrix tells you not just how many errors the model makes, but which classes it confuses with which — invaluable for understanding failure modes. For example, if a digit recognition model confuses 4s and 9s frequently, you know to focus improvement efforts on those pairs. Scikit-learn handles multi-class confusion matrices with the same confusion_matrix() function, and classification_report() automatically provides per-class precision, recall, and F1 for all classes.
