Data Quality and Data Contracts – Validation, Monitoring and Great Expectations

Data quality is not a peripheral concern — it is the foundation that every ML model and analytics product rests on. A model trained on biased, incomplete, or mislabelled data will systematically produce wrong predictions regardless of its architecture or hyperparameters. A dashboard reading from a pipeline with silent data corruption will misinform decisions for days before the error is noticed. Yet despite its centrality, data quality is systematically underinvested in most data science organisations: it is unglamorous, its absence is often invisible until something goes badly wrong, and its value is harder to measure than model accuracy improvements. This guide covers the framework for thinking about data quality, the technical tools (Great Expectations, Soda, dbt tests, Monte Carlo Data) for implementing automated data quality checks, the data contract pattern for enforcing quality agreements across teams, and the ML-specific challenge of detecting and responding to distribution shift in production data.

A Framework for Data Quality — Four Dimensions

Data quality problems fall into four categories, each with distinct causes, detection methods, and remediation strategies. Understanding which category a quality issue belongs to is the prerequisite for choosing the right tool and fix.

Completeness failures are missing values in required fields — NULL where NOT NULL was expected, absent rows where rows were expected, or incomplete time series with gaps. Completeness monitoring tracks null rates and row counts against expected baselines. Completeness issues can be benign (optional fields left empty by users) or critical (a pipeline failure that stopped writing rows after a timestamp). Pandas profiling and the df.isnull().sum() idiom provide completeness snapshots; automated checks using Great Expectations’ expect_column_values_to_not_be_null alert when null rates exceed thresholds. Missing value imputation strategies — mean/median, forward fill, model-based imputation — are covered in our feature engineering guide.

Validity failures are values that are technically present but logically incorrect — ages of −5 or 500, dates in the future for historical events, negative transaction amounts where only positive are valid, categorical values not in the expected set. Validity checks encode domain knowledge as machine-executable rules. The key challenge is that validity failures are rarely obvious in aggregate statistics — a single invalid value among millions can silently corrupt a trained model if it happens to be systematically associated with a label. Our anomaly detection guide covers the unsupervised approaches (Isolation Forest, statistical outlier detection) used to identify invalid values when explicit rules cannot enumerate all valid states.

Consistency failures are contradictions between related fields — a transaction marked “completed” with a NULL completion timestamp, a customer with a creation date later than their first purchase date, a product with a sale price higher than its list price. Consistency checks require cross-field and cross-table validation that goes beyond single-column checks. The advanced SQL techniques — multi-table joins, CTEs with self-joins — are the most natural language for expressing consistency rules against relational data. dbt tests (relationships, accepted values, not-null) codify these rules in the transformation layer.

Timeliness failures are stale or delayed data — a feature expected to be updated daily that has not been refreshed in three days, a streaming table that has not received new records in the past hour. Timeliness is particularly critical for deployed ML models that consume features from a feature store: a stale feature can cause a model to make decisions based on outdated information. The Airflow pipeline monitoring and SLA alerts in data engineering fundamentals are the standard infrastructure for timeliness monitoring.

Data Contracts — Formalising Quality Agreements

Data contracts formalise the implicit agreements between data producers and consumers that, when violated, cause downstream failures. In a typical organisation without data contracts, the data engineering team makes a schema change that breaks an ML model, the ML team discovers the break when their model silently starts producing wrong predictions, and a debugging session lasting hours or days follows. With data contracts, the schema change is caught at the boundary — the contract check fails at the producer’s deployment step, before the change reaches production data consumers.

A data contract specifies: the schema (column names, types, nullable constraints); statistical guarantees (the null rate of column X is always below 5%, the mean of column Y is always between 100 and 200, the number of daily rows is always between 50,000 and 500,000); semantic guarantees (the primary key is always unique; the foreign key relationship to the customers table always holds); and freshness guarantees (the table is refreshed at least once per day). Contracts are stored as code (YAML or JSON) in version control, validated automatically in CI/CD pipelines, and enforced by data platform infrastructure. The data engineering fundamentals guide covers the pipeline infrastructure that data contracts plug into, and our MLOps guide covers how data quality gates are integrated into ML training pipelines.

The tools ecosystem for data contracts has matured rapidly. Great Expectations is the most widely adopted open-source framework: you define “expectations” (expect_column_values_to_not_be_null, expect_column_values_to_be_between, expect_table_row_count_to_be_between) in Python or YAML, and GE validates them against your data and generates HTML data quality reports. dbt tests (not_null, unique, accepted_values, relationships) are the natural contract layer for dbt-based transformation pipelines — they run after each dbt model and fail the pipeline if constraints are violated. The ETL pipelines guide covers dbt test integration in production pipelines.

Distribution Shift — The ML-Specific Data Quality Challenge

For ML models in production, the most insidious data quality problem is distribution shift: the statistical properties of the production data diverging from the training data over time. Unlike schema errors (which cause obvious failures), distribution shift causes models to degrade silently — predictions remain syntactically valid but semantically wrong. The model was trained on data from one population; production data increasingly represents a different population. The MLOps guide covers the full monitoring and retraining lifecycle; here we focus on the statistical detection methods.

Population Stability Index (PSI) is the most widely used metric for detecting feature distribution shift in credit scoring and financial ML. PSI = Σ (actual% − expected%) × ln(actual%/expected%) across bins of the feature distribution. PSI < 0.1 indicates no significant shift; 0.1–0.2 warrants monitoring; > 0.2 indicates significant shift requiring investigation. PSI is equivalent to a symmetrised KL divergence, the same divergence used in information-theoretic analysis in our Bayesian statistics guide. The probability distributions guide covers the distributional properties that PSI and KL divergence measure.

Two-sample statistical tests — the Kolmogorov-Smirnov test for continuous features, the chi-squared test for categorical features — provide formal hypothesis tests for distribution shift. These are the same tests covered in our hypothesis testing guide, applied in a monitoring context rather than an experimental one. The challenge in production monitoring is that with large sample sizes, any distributional difference will be statistically significant — PSI and effect size measures are more operationally useful than p-values for this reason. The model evaluation guide covers calibration drift — the specific form of distribution shift where the predicted probability no longer matches the empirical event rate — and its remediation through recalibration.

✦ SUMMARIZE THIS ARTICLE WITH AI

ChatGPT Perplexity Claude Grok

The data engineering pipeline infrastructure — Airflow, dbt, and data warehouse patterns — that data quality checks integrate with is covered in our ETL Pipelines guide and Data Engineering Fundamentals guide. Feature engineering — handling the missing values and outliers that data quality issues create — is in our Feature Engineering guide. MLOps model monitoring and retraining strategies for distribution shift are in our MLOps guide. Anomaly detection for identifying validity outliers in data is in our Anomaly Detection guide. The statistical tests used for distribution shift detection — KS test, chi-squared, KL divergence — are in our Hypothesis Testing and Bayesian Statistics guides.