📋 KEY INSIGHTS
- Data science for retail addresses decisions across the retail value chain: pricing (setting and updating prices to maximise revenue or profit), assortment (deciding which products to carry in which stores), inventory (setting safety stock and reorder points), promotions (planning markdowns and trade promotions), and personalisation (product recommendations and targeted marketing). Each problem has a distinct modelling approach, but all share the need for causal inference — naive correlations between prices and sales reflect both demand elasticity (causal) and endogeneity (prices are raised during high-demand periods, confounding the correlation).
- Price elasticity of demand quantifies how sensitive sales volume is to price changes: elasticity ε = (% change in quantity) / (% change in price). Estimating elasticity from historical pricing data requires causal inference to avoid endogeneity bias: prices and demand are jointly determined (prices rise in peak demand periods), so naive OLS regression of log(sales) on log(price) overestimates elasticity. The standard solution is an instrumental variable approach — using competitor prices, cost shocks, or rule-based price changes as instruments for price — or a quasi-experimental design exploiting natural variation in prices across stores or time. Causal inference and uplift modelling methodology directly applies here.
- Demand forecasting for retail differs from generic time series forecasting in three important ways: (1) intermittent demand — many SKUs have zero sales on most days, requiring specialised models (Croston method, zero-inflated distributions); (2) hierarchical structure — forecasts are needed at store × SKU level but training data is sparse, requiring temporal hierarchical reconciliation; (3) causal features matter enormously — promotions, holidays, weather, and competitor actions cause demand spikes far larger than the autoregressive component. The state-of-the-art approach combines gradient boosting (LightGBM with promotional, calendar, and lagged sales features) with deep learning (Temporal Fusion Transformer for capturing complex promotional lift patterns) and probabilistic outputs for safety stock setting.
- Markdown optimisation (clearance pricing) determines the sequence of price reductions to sell a fixed inventory of seasonal products by end-of-season at a target clearance rate, maximising total revenue. The problem is a sequential decision process (the markdown schedule is a sequence of price levels over T time periods) with uncertain demand. Dynamic programming and reinforcement learning are the natural formulations; practical approaches use a demand model (estimating sales rate as a function of current price and remaining inventory) combined with a lookahead optimisation that selects the markdown schedule maximising expected revenue. The connection to our RL for real-world applications guide is direct: markdown optimisation is one of the highest-value real-world RL deployments in retail, with documented revenue uplifts of 3-8% over rule-based markdown schedules.
- Assortment optimisation decides which products to include in the store offer (and in what quantities) given shelf space, supplier constraints, and customer preference data. Customer choice models — Multinomial Logit (MNL), Mixed Logit, Nested Logit — model the probability that a customer chooses product j from assortment S as P(j|S) = exp(V_j) / Σ_{k ∈ S} exp(V_k), where V_j is the utility of product j estimated from transaction data. The optimal assortment that maximises expected revenue given shelf space constraint is a combinatorial optimisation problem (NP-hard in general) solved approximately by greedy algorithms, branch and bound, or linear programming relaxations with column generation.
- Market basket analysis and association rules identify products frequently purchased together, enabling cross-selling, store layout optimisation, and bundle pricing. The Apriori algorithm and FP-Growth are the classical methods for mining frequent itemsets; the connection metrics are support (fraction of transactions containing the itemset), confidence (P(B|A) = P(A ∪ B) / P(A)), and lift (confidence / P(B) — values > 1 indicate the items are purchased together more often than by chance). Modern retail analytics has largely moved from rule mining to embedding-based approaches: product embeddings trained by treating transaction baskets as sentences (item2vec, an application of Word2Vec from our word embeddings guide) capture rich co-purchase similarity that generalises to unseen product combinations.
Retail is one of the highest-data-intensity industries: a mid-size grocery chain generates millions of transactions per day, maintains tens of thousands of active SKUs across hundreds of stores, and must make pricing, inventory, and assortment decisions at the store × SKU level on a daily or weekly cadence. The scale and velocity of retail data make machine learning not just beneficial but operationally necessary — no human team can manually set prices and safety stocks for 50,000 SKUs in 500 stores. This guide covers the core data science problems in retail: demand forecasting (including promotional lift modelling and intermittent demand), price elasticity estimation with causal inference, dynamic pricing and markdown optimisation, assortment optimisation with choice models, inventory optimisation, and customer lifetime value and segmentation.
Demand Forecasting and Promotional Lift Modelling
Retail demand forecasting operates at the intersection of time series modelling and causal feature engineering: the autoregressive component (last week sold X units) is dwarfed by causal components (a 20% promotion doubles sales; Christmas week triples sales) for most SKUs. The standard production pipeline combines: a base demand model (capturing trend and seasonality, typically using our Prophet or ARIMA models), a promotional lift model (estimating the multiplicative demand uplift from promotions, using gradient boosting with promotion type, depth, and duration features), and a price elasticity model (estimating the demand change from price moves, using causal regression with instrument variables). LightGBM with 200+ features — lag sales at multiple windows (1, 2, 4, 8, 13, 26, 52 weeks), promotional indicators, holiday flags, weather features, and rolling means — is the backbone of most production retail forecasting systems due to its combination of tabular accuracy, speed, and feature interpretability via SHAP. The Temporal Fusion Transformer adds value for SKUs with complex multi-period promotional patterns that gradient boosting misses, particularly for multi-week promotional campaigns where demand is elevated for the entire campaign duration and there is a demand pull-forward/pull-back effect before and after the promotion. Intermittent demand (many zero-sales days) requires specialised models: the Croston method (exponential smoothing applied separately to demand intervals and demand sizes) and the ADIDA framework (temporal aggregation to reduce intermittency before applying standard forecasters) outperform standard exponential smoothing on slow-moving SKUs.
| Retail DS Problem | Modelling Approach | Key Features | Metric | Key Package |
|---|---|---|---|---|
| Demand forecasting | LightGBM + TFT ensemble | Lags, promo, holiday, weather | WMAPE, wMASE | LightGBM, GluonTS |
| Price elasticity | IV regression / DML | Price, competitor price, cost | Elasticity SE, coverage | EconML, linearmodels |
| Markdown optimisation | DP / RL + demand model | Price, inventory, days remaining | Revenue, clearance rate | Stable-Baselines3, cvxpy |
| Assortment optimisation | MNL choice model + IP | Utility features, shelf space | Expected revenue per sq ft | scipy, PuLP |
| Basket analysis | FP-Growth / item2vec | Transaction items | Lift, revenue per basket | mlxtend, gensim |
Price Elasticity with Causal Inference and Dynamic Pricing
Price elasticity estimation is a canonical causal inference problem in retail: we want to know the causal effect of a price change on demand, but the data we observe is generated by a process where prices and demand are jointly determined. High-demand periods (Christmas, back-to-school) see higher prices because retailers raise prices during peak demand — this creates a positive correlation between price and sales in the raw data, biasing naive OLS estimates of elasticity toward zero (prices are set high, sales are high, so price appears not to decrease sales). The instrumental variable (IV) approach resolves endogeneity by finding a variable that affects price but affects demand only through price: competitor prices and input cost indices are standard instruments. The Double Machine Learning (DML) framework from our causal inference guide extends IV to high-dimensional settings: partial out the effect of control variables (seasonality, promotions, store characteristics) from both the price and sales variables using ML models, then estimate elasticity from the residual-on-residual regression. The EconML library implements DML for retail elasticity estimation. Dynamic pricing sets prices in real time based on current inventory, time remaining, competitor prices, and demand signals, optimising a revenue or margin objective over a planning horizon. Rule-based dynamic pricing (adjusting price by fixed amounts when inventory thresholds are crossed) is the production baseline; ML-based dynamic pricing (using a demand model to simulate the revenue-optimal price at each timestep) outperforms rules by 2-5% revenue lift in controlled experiments. The connection to our multi-armed bandits guide is direct: contextual bandits with continuous action spaces (the price is the arm) are the natural formulation for real-time pricing with limited historical data, enabling online learning of price-demand relationships while balancing exploration (testing prices to learn elasticity) and exploitation (setting the revenue-maximising price given current estimates).
Customer Lifetime Value, Segmentation and Personalisation
Customer lifetime value (CLV) prediction estimates the total future revenue a customer will generate, enabling customer-level investment decisions (how much to spend on retention for each customer segment). The BG/NBD model (Buy-Till-You-Die with Negative Binomial Distribution) is the standard probabilistic CLV model for non-contractual settings: it jointly models purchase frequency (customers make purchases at a Poisson rate γ while active) and customer lifetime (customers permanently churn at rate μ, modelled as an exponential lifetime). The connection to our survival analysis guide is direct: CLV modelling is essentially survival analysis for customer lifetimes with a purchasing process overlaid. The CLV-based segmentation that follows (high-CLV loyalists, medium-CLV occasional shoppers, low-CLV deal-seekers) connects to our clustering guide: RFM (Recency, Frequency, Monetary) k-means clustering is the operational simplification of full probabilistic CLV modelling used in production systems due to its interpretability. Personalisation at scale — serving individualised product recommendations, promotions, and email content to millions of customers — requires the deep recommender systems architecture: a Two-Tower retrieval model to identify the top-K relevant products from a catalogue of millions, followed by a ranking model that scores the retrieved candidates using customer context, purchase history, and promotional features. The connection to our marketing analytics guide is direct: retail personalisation and marketing attribution share the same infrastructure (customer feature store, A/B testing framework, campaign measurement) and the same causal inference challenges (isolating the incremental lift of a promotion from selection effects).
✦ SUMMARIZE THIS ARTICLE WITH AI
ChatGPTDemand forecasting models (Prophet, ARIMA, Temporal Fusion Transformer) are in our Time Series Analysis and Time Series Deep Learning guides. Causal inference for price elasticity estimation (DML, IV regression) is in our Causal Inference guide. RL for markdown optimisation is in our RL for Real-World Applications guide. Deep recommendation systems (Two-Tower, ranking models) for retail personalisation are in our Deep Recommenders guide. Marketing analytics (attribution, CLV, cohort analysis) is in our Marketing Analytics guide. Supply chain and inventory optimisation is in our Data Science for Supply Chain guide.
