Every machine learning model starts with data exploration. Skip this step and you will build models on assumptions that are not true, miss relationships that are obvious once you look, and deliver results that confuse rather than inform. Exploratory Data Analysis — EDA — is the discipline of understanding your data before doing anything else with it. It is equal parts statistics, visualisation, and detective work.
This guide walks through a complete EDA process in Python: from loading raw data to producing the kind of structured analysis that data science teams and stakeholders actually find useful. We use a realistic dataset and cover every major technique you will need in practice.
What EDA Is Actually For
EDA serves three purposes that are easy to state but surprisingly deep in practice. First, it helps you understand the data’s structure — how many rows and columns, what types they are, what the distributions look like. Second, it surfaces data quality issues that would silently corrupt any analysis built on top: missing values, impossible values, duplicates, and inconsistently encoded categories. Third, it uncovers patterns and relationships that guide everything that follows — which features correlate with the target, which variables have outliers, which distributions violate the assumptions of statistical tests you might want to run.
A good EDA is also a communication tool. A clear, structured notebook that shows your data’s key properties, highlights surprising findings, and explains important caveats is invaluable for collaborating with colleagues, getting stakeholder buy-in, and documenting your work for reproducibility.
Step 1: Load and Inspect
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
plt.style.use('seaborn-v0_8-whitegrid')
pd.set_option('display.max_columns', 50)
# Load data
df = pd.read_csv('your_dataset.csv')
# First look
print(f'Shape: {df.shape}') # rows x columns
print(f'
Column types:
{df.dtypes}')
print(f'
First 5 rows:')
display(df.head())
print(f'
Basic statistics:')
display(df.describe(include='all'))
The describe() output immediately surfaces obvious problems: a column that should range 0–100 showing a max of 9999, a categorical column with far more unique values than expected, a numeric column with a mean and median so different that extreme outliers are likely.
Step 2: Assess Data Quality
# Missing values — percentage and absolute count
missing = pd.DataFrame({
'missing_count': df.isnull().sum(),
'missing_pct': (df.isnull().sum() / len(df) * 100).round(2)
}).sort_values('missing_pct', ascending=False)
print(missing[missing.missing_count > 0])
# Duplicates
print(f'
Duplicate rows: {df.duplicated().sum()}')
# Cardinality of categorical columns
cat_cols = df.select_dtypes(include='object').columns
for col in cat_cols:
print(f'{col}: {df[col].nunique()} unique values')
if df[col].nunique() < 15:
print(f' Values: {df[col].value_counts().to_dict()}')
Missing value patterns matter as much as counts. Are values missing completely at random, or is missingness correlated with other variables? A column where missing income always corresponds to customers who declined to answer is very different from one where missing income correlates with high-value customers who were never asked.
Step 3: Univariate Analysis
Univariate analysis examines each variable independently. For numeric variables, you want to understand central tendency, spread, and shape. For categorical variables, you want to understand the distribution of categories.
num_cols = df.select_dtypes(include=np.number).columns
# Distribution plots for all numeric features
n_cols = 3
n_rows = (len(num_cols) + n_cols - 1) // n_cols
fig, axes = plt.subplots(n_rows, n_cols, figsize=(15, n_rows * 4))
axes = axes.flatten()
for i, col in enumerate(num_cols):
axes[i].hist(df[col].dropna(), bins=30,
color='steelblue', edgecolor='white', alpha=0.8)
axes[i].axvline(df[col].mean(), color='red', linestyle='--',
linewidth=1.5, label=f'Mean: {df[col].mean():.2f}')
axes[i].axvline(df[col].median(), color='orange', linestyle='--',
linewidth=1.5, label=f'Median: {df[col].median():.2f}')
axes[i].set_title(col); axes[i].legend(fontsize=8)
for j in range(i+1, len(axes)):
axes[j].set_visible(False)
plt.suptitle('Numeric Feature Distributions', fontsize=14, y=1.01)
plt.tight_layout()
plt.show()
Key patterns to look for: highly skewed distributions (mean much larger than median) that may need log transformation before modelling; bimodal distributions suggesting two distinct populations in your data; very high frequency of specific values (often a sentinel for "unknown" or "not applicable").
Step 4: Bivariate Analysis — Relationships Between Variables
# Correlation matrix — numeric features
corr = df[num_cols].corr()
mask = np.triu(np.ones_like(corr, dtype=bool))
plt.figure(figsize=(12, 10))
sns.heatmap(corr, mask=mask, annot=True, fmt='.2f',
cmap='RdBu_r', center=0, vmin=-1, vmax=1,
square=True, linewidths=0.5, cbar_kws={'shrink': 0.8})
plt.title('Feature Correlation Matrix', fontsize=14)
plt.tight_layout()
plt.show()
# High correlations (potential multicollinearity)
high_corr = []
for i in range(len(corr.columns)):
for j in range(i+1, len(corr.columns)):
if abs(corr.iloc[i, j]) > 0.8:
high_corr.append((corr.columns[i], corr.columns[j],
corr.iloc[i, j]))
if high_corr:
print('Highly correlated feature pairs (|r| > 0.8):')
for a, b, r in sorted(high_corr, key=lambda x: -abs(x[2])):
print(f' {a} × {b}: r = {r:.3f}')
# Target variable analysis — how features relate to what you are predicting
target = 'your_target_column'
# For a numeric target: scatter plots
for col in num_cols[:6]:
if col != target:
plt.figure(figsize=(6, 4))
plt.scatter(df[col], df[target], alpha=0.3, s=20, color='steelblue')
plt.xlabel(col); plt.ylabel(target)
plt.title(f'{col} vs {target}')
plt.tight_layout(); plt.show()
Step 5: Outlier Detection
# Box plots reveal outliers visually
fig, axes = plt.subplots(1, len(num_cols[:6]), figsize=(16, 5))
for i, col in enumerate(num_cols[:6]):
axes[i].boxplot(df[col].dropna(), vert=True)
axes[i].set_title(col, fontsize=9)
plt.suptitle('Outlier Detection — Box Plots')
plt.tight_layout(); plt.show()
# IQR method to quantify outliers
for col in num_cols:
Q1 = df[col].quantile(0.25)
Q3 = df[col].quantile(0.75)
IQR = Q3 - Q1
outliers = df[(df[col] < Q1 - 1.5*IQR) | (df[col] > Q3 + 1.5*IQR)]
if len(outliers) > 0:
pct = len(outliers) / len(df) * 100
print(f'{col}: {len(outliers)} outliers ({pct:.1f}%)')
The key question with outliers is not "should I remove them?" but "what caused them?" A house price of ₹50 crore in a dataset of typical homes might be a legitimate luxury property or a data entry error. The answer changes whether you remove, cap, or investigate further. Outliers that are genuine extreme values often carry important information and should not be removed automatically.
Frequently Asked Questions
How long should EDA take before I start building models?
There is no fixed rule, but a common mistake is either rushing EDA to "get to the interesting part" or over-engineering beautiful visualisations while avoiding actual modelling. For a new dataset, a thorough first pass of EDA — covering all five steps in this guide — typically takes two to four hours of focused work for a dataset of moderate size (tens of thousands of rows, dozens of columns). After that, you will have formed hypotheses about the data that guide your feature engineering and model selection. EDA is not a one-time step either: as you build models and encounter unexpected behaviour, you return to EDA to investigate. Think of it as iterative investigation rather than a sequential phase.
What is the difference between EDA and data cleaning?
They are deeply intertwined but conceptually distinct. EDA is about understanding — discovering what your data contains, what its properties are, and what patterns exist. Data cleaning is about fixing — resolving the quality issues that EDA surfaces. In practice, EDA reveals that you have 3% missing values in the income column (that is EDA), and data cleaning decides to impute them with the median stratified by job category (that is cleaning). You cannot clean data well without first understanding it through EDA, and thorough EDA will naturally surface cleaning tasks. Most experienced data scientists move fluidly between the two rather than treating them as separate phases.
Should I always use a correlation matrix to find important features?
Correlation matrices are a useful starting point but have important limitations. They only measure linear relationships — a feature could have a strong non-linear relationship with your target and show near-zero correlation. They also do not account for interactions between features. A feature that has weak correlation with the target on its own might become highly predictive when combined with another feature. For these reasons, correlation analysis should be treated as hypothesis generation rather than feature selection. Always validate feature importance using your actual model's feature importance scores or permutation importance after training, rather than relying solely on pre-modelling correlation analysis.
What tools can automate EDA?
Several Python libraries automate parts of EDA. ydata-profiling (formerly pandas-profiling) generates a comprehensive HTML report with distributions, correlations, missing value analysis, and duplicate detection in a single function call: ProfileReport(df).to_notebook_iframe(). sweetviz creates visual reports comparing two datasets — useful for comparing training and test sets or before and after data cleaning. dtale provides an interactive web-based EDA interface. These tools are excellent for getting a quick overview, but they do not replace the judgment required to interpret what you find and decide how to handle it. Use them as a starting point, not a replacement for understanding your data.
