Pandas and NumPy Mastery – Complete Guide for Data Scientists 2026

Pandas and NumPy are the two most foundational Python libraries for data science. Virtually every data processing pipeline, machine learning preprocessing step, and exploratory analysis starts with these two libraries. Yet most practitioners only use a fraction of their capabilities — relying on slow loops where vectorised operations would be 100x faster, or missing powerful functions that would reduce 20 lines of code to one. This guide covers the patterns, functions, and performance techniques that distinguish expert pandas and NumPy users from beginners.

These libraries are tested directly in Python data science interviews and underpin the feature engineering techniques covered in our Feature Engineering guide. The time series patterns in this guide connect to our Time Series Forecasting Interview Q&A, and the performance techniques are essential for building the data pipelines described in our Data Engineering Interview Q&A.

NumPy Fundamentals — Vectorisation and Broadcasting

NumPy’s power comes from two concepts: vectorisation (applying operations to entire arrays without Python loops) and broadcasting (applying operations between arrays of different shapes). Both rely on NumPy’s C-backed contiguous memory arrays — ndarray — which enable BLAS/LAPACK linear algebra routines and are orders of magnitude faster than Python lists for numerical computation.

Why Python loops over arrays are slow: Python is dynamically typed — every operation on a Python list requires type checking, reference counting, and memory allocation overhead. A NumPy add operation on a 1M-element array executes in compiled C with a single function call; a Python loop over the same array executes 1M Python function calls. The speedup is typically 50-200x.

import numpy as np

# --- Vectorisation ---
arr = np.random.randn(1_000_000)

# Slow: Python loop
result_loop = [x**2 for x in arr]          # ~400ms

# Fast: NumPy vectorised
result_vec = arr ** 2                       # ~2ms  (200x faster)

# --- Broadcasting ---
# Shapes (3,4) and (4,) → broadcast (4,) to (3,4)
matrix = np.ones((3, 4))
row    = np.array([1, 2, 3, 4])
result = matrix + row          # adds row to every row of matrix

# Normalise each column of a matrix (broadcasting trick)
X = np.random.randn(1000, 20)
X_normalised = (X - X.mean(axis=0)) / X.std(axis=0)  # axis=0 = column-wise

# --- Universal functions (ufuncs) ---
# np.exp, np.log, np.sqrt, np.abs are all vectorised ufuncs
# Always prefer ufuncs over math module for array operations
log_arr  = np.log1p(np.abs(arr))     # log(1 + |x|) -- safe for zeros

# --- Fancy indexing and boolean masks ---
prices   = np.array([10.5, 23.1, 8.7, 45.2, 12.0])
mask     = prices > 15
filtered = prices[mask]                # [23.1, 45.2]
prices[mask] *= 0.9                    # 10% discount on high-price items

# --- Structured arrays for mixed-type tabular data ---
dtype = np.dtype([('name', 'U20'), ('age', 'i4'), ('score', 'f8')])
data  = np.array([('Alice', 28, 92.5), ('Bob', 35, 88.1)], dtype=dtype)
print(data['age'].mean())

Memory layout — C-order vs Fortran-order: NumPy arrays are stored as contiguous blocks of memory. C-order (row-major, default): elements of a row are contiguous — efficient for row-wise operations. Fortran-order (column-major): elements of a column are contiguous — efficient for column-wise operations. Operating along the contiguous axis is cache-efficient and significantly faster. Always check arr.flags if performance-sensitive. For matrix operations passed to BLAS, Fortran-order arrays avoid an internal copy.

Key NumPy operations every data scientist must know:

Pandas — Beyond the Basics

Pandas is built on NumPy and provides the DataFrame — a 2D labelled data structure with heterogeneous column types, powerful indexing, and a rich API for data manipulation. The most important mindset shift for pandas expertise: think in terms of operations on entire Series/DataFrames, not row-by-row loops. The apply() function is often a code smell — it is essentially a Python loop in disguise.

The pandas performance hierarchy (fastest to slowest):

import pandas as pd
import numpy as np

# --- Common anti-patterns and their vectorised fixes ---

# BAD: apply with lambda for arithmetic
df['profit_margin'] = df.apply(lambda r: r['profit'] / r['revenue']
                               if r['revenue'] > 0 else np.nan, axis=1)

# GOOD: vectorised with np.where
df['profit_margin'] = np.where(df['revenue'] > 0,
                               df['profit'] / df['revenue'], np.nan)

# BAD: loop to create a flag column
flags = []
for _, row in df.iterrows():
    flags.append('High' if row['score'] > 80 else 'Low')
df['tier'] = flags

# GOOD: np.select for multiple conditions
conditions = [df['score'] > 80, df['score'] > 60]
choices    = ['High', 'Medium']
df['tier'] = np.select(conditions, choices, default='Low')

# --- String operations via .str accessor ---
df['email_domain']    = df['email'].str.split('@').str[-1]
df['name_upper']      = df['name'].str.upper()
df['has_india']       = df['address'].str.contains('India', case=False, na=False)
df['phone_clean']     = df['phone'].str.replace(r'[\s\-\(\)]', '', regex=True)

# --- Datetime operations via .dt accessor ---
df['date'] = pd.to_datetime(df['date'])
df['year']       = df['date'].dt.year
df['month']      = df['date'].dt.month
df['day_of_week']= df['date'].dt.dayofweek   # 0=Monday
df['is_weekend'] = df['date'].dt.dayofweek >= 5
df['quarter']    = df['date'].dt.quarter

GroupBy — The Swiss Army Knife of Pandas

GroupBy is the most important pandas operation for data analysis — it implements the split-apply-combine pattern: split the DataFrame by one or more keys, apply a function to each group, and combine results. Understanding its internals makes you dramatically more effective.

# --- Multiple aggregations in one pass ---
agg_result = df.groupby(['region', 'category']).agg(
    total_revenue = ('revenue', 'sum'),
    avg_order_val = ('revenue', 'mean'),
    n_orders      = ('order_id', 'count'),
    unique_customers = ('customer_id', 'nunique'),
    p95_revenue   = ('revenue', lambda x: x.quantile(0.95))
).reset_index()

# --- Transform: group-level statistics at row level ---
# (doesn't collapse rows — preserves original index)
df['revenue_zscore'] = df.groupby('category')['revenue'].transform(
    lambda x: (x - x.mean()) / x.std()
)
df['pct_of_category_total'] = (df['revenue'] /
    df.groupby('category')['revenue'].transform('sum') * 100)

# --- Filter: keep only groups meeting a condition ---
# Keep only categories with more than 100 orders
df_filtered = df.groupby('category').filter(lambda g: len(g) > 100)

# --- Cumulative operations within groups ---
df = df.sort_values(['user_id', 'order_date'])
df['cumulative_spend'] = df.groupby('user_id')['revenue'].cumsum()
df['order_number']     = df.groupby('user_id').cumcount() + 1

Merge, Join, and Reshape

# --- Merge types (SQL equivalents) ---
# INNER join: only matching keys
inner = pd.merge(orders, customers, on='customer_id', how='inner')

# LEFT join: all orders, matched customer info (NaN if no match)
left  = pd.merge(orders, customers, on='customer_id', how='left')

# Merge on multiple keys
merged = pd.merge(sales, targets,
                  on=['region', 'product_category', 'year'],
                  how='left', suffixes=('_actual', '_target'))

# --- Merge diagnostics: detect duplicates and mismatches ---
merged = pd.merge(df_left, df_right, on='id', how='left', indicator=True)
print(merged['_merge'].value_counts())
# both / left_only / right_only

# --- Pivot table ---
pivot = df.pivot_table(
    values='revenue', index='region', columns='quarter',
    aggfunc='sum', fill_value=0, margins=True
)

# --- Melt (wide to long) ---
long_df = pd.melt(wide_df,
    id_vars=['user_id', 'date'],
    value_vars=['feature_1', 'feature_2', 'feature_3'],
    var_name='feature_name', value_name='feature_value'
)

Performance Optimisation for Large DataFrames

When DataFrames grow to millions of rows, memory and speed become critical. The Data Engineering Interview Q&A covers distributed alternatives (Spark, Dask) for truly large data, but pandas itself has significant optimisation headroom.

Memory reduction via dtype optimisation:

def optimise_dtypes(df):
    for col in df.select_dtypes('integer').columns:
        df[col] = pd.to_numeric(df[col], downcast='integer')
    for col in df.select_dtypes('float').columns:
        df[col] = pd.to_numeric(df[col], downcast='float')
    for col in df.select_dtypes('object').columns:
        if df[col].nunique() / len(df) < 0.1:   # < 10% unique = categorical
            df[col] = df[col].astype('category')
    return df

# Before optimisation: 500MB
df = optimise_dtypes(df)
# After optimisation: ~120MB (4x reduction is typical)
print(df.memory_usage(deep=True).sum() / 1e6, 'MB')

Additional performance patterns: use query() for filtering (often 2-3x faster than boolean indexing on large DataFrames); use eval() for arithmetic expressions (avoids creating intermediate arrays); read CSVs with usecols to load only needed columns; use chunksize for processing files larger than RAM; store intermediate results in Parquet (10x faster to read than CSV, 5x smaller). These patterns directly apply to the feature engineering pipelines that feed into machine learning models and the time series feature creation workflows.

For interview questions specifically on pandas and Python data manipulation, our Python Interview Q&A covers 50 questions including common pandas gotchas (chained indexing, copy vs view, SettingWithCopyWarning) that appear frequently in technical rounds. The SQL equivalents of these operations are covered in our Advanced SQL for Data Scientists guide.