Modern Data Architecture – Data Mesh, Lakehouse and Medallion Architecture

Data architecture — how data is stored, organised, moved, and made available for analysis and machine learning — has more impact on a data science organisation’s productivity than any individual tool or algorithm choice. A team working with a well-designed, reliable data platform ships models and insights faster, with fewer errors, and with greater confidence in their results. A team wrestling with a fragile, undocumented, poorly organised data environment spends most of its time on data plumbing rather than analysis. The architecture of data systems is therefore not just an engineering concern — it is a direct driver of data science output quality and velocity. This guide covers the evolution from data warehouse to data lake to lakehouse, the medallion architecture pattern, dbt as the transformation standard, the data mesh organisational paradigm, and the modern data stack in its 2026 form — including how each component connects to ML feature engineering and model training workflows.

From Data Warehouse to Lakehouse — The Storage Evolution

The traditional data warehouse (Teradata, IBM DB2, later Snowflake and BigQuery) stores structured, schema-on-write data in columnar format optimised for analytical queries — aggregations, filters, and joins over large tables. Data warehouses offer excellent query performance, ACID transactions, and schema enforcement, but at high cost per byte stored and with limited support for unstructured data (text, images, sensor streams) and the large, raw files that ML training requires.

The data lake (Hadoop HDFS, then S3/GCS/ADLS) swung to the opposite extreme: cheap, flat-file storage (Parquet, ORC, Avro) with no built-in schema enforcement, transactions, or indexing. Data lakes can store anything at very low cost — raw logs, images, JSON blobs, model artefacts — but querying them efficiently requires careful partitioning, and the absence of ACID transactions means concurrent writes can corrupt data and time-travel queries (querying the data as it was at a specific past timestamp) are difficult. Data lakes also suffer from “data swamps” — accumulating poorly documented, untested, and stale data that no one trusts.

The lakehouse combines the best of both: cheap object storage (S3, GCS, ADLS) as the physical storage layer, with a table format layer that adds ACID transactions, schema enforcement, partition pruning, and time-travel on top. The three competing open table formats — Delta Lake (Databricks), Apache Iceberg (Netflix, Apple), and Apache Hudi (Uber) — each implement this table format layer differently but achieve similar semantics. Delta Lake uses a transaction log (_delta_log) in JSON format; Iceberg uses manifest files and a metadata tree; Hudi uses a timeline-based log. All three support: insert, update, delete, and merge operations with ACID guarantees; schema evolution (adding, renaming, or dropping columns without rewriting data); time-travel queries (SELECT * FROM table AS OF TIMESTAMP ‘2026-01-01’); and partition pruning for efficient query execution. The connection to Spark and distributed compute is direct — all three formats are designed for Spark as the primary compute engine, though they also support DuckDB, Presto, and Trino for query execution on object storage.

Medallion Architecture — Bronze, Silver, Gold

The medallion architecture (also called multi-hop or layered architecture) organises data within a lakehouse into three quality tiers, each representing a different stage of processing and a different audience of consumers. This organisation solves a fundamental data lake problem: without explicit quality layers, analysts cannot tell whether a table contains raw, unvalidated data or a curated, business-ready dataset, and the incentive to maintain quality is diffuse.

Bronze (raw layer) is the ingestion layer: data lands here exactly as it arrives from source systems — raw API responses, database CDC (change data capture) events, file uploads — with no transformation or validation. Bronze tables are append-only and immutable: once data arrives, it is never changed or deleted (unless legally required). The purpose of Bronze is data provenance: any downstream data quality issue can be traced back to the original raw record. Airflow, Fivetran, and Airbyte are the standard tools for loading data into Bronze from operational databases, SaaS APIs, and streaming systems. Our data engineering fundamentals guide covers CDC patterns and streaming ingestion in detail.

Silver (validated and joined layer) applies data quality checks, type casting, deduplication, and joins to produce clean, validated, entity-resolved tables. This is where the data quality checks — null checks, range validation, consistency assertions — run and where schema conformance is enforced. dbt is the standard tool for Silver transformations: dbt models are SQL SELECT statements that define Silver tables as transformations of Bronze tables, with built-in tests (not_null, unique, accepted_values, relationships) that validate quality and run automatically. dbt’s modular, version-controlled, documented approach applies software engineering best practices — code review, CI/CD, dependency graphs — to what was previously an ad-hoc collection of SQL scripts. Our ETL pipelines guide covers dbt in depth.

Gold (business and ML-ready layer) contains aggregated, business-level metrics and ML-ready feature tables. Gold is what analysts query for dashboards and what ML engineers read for model training. In ML contexts, Gold tables are often managed by a feature store (Feast, Tecton, Hopsworks) that handles point-in-time correct feature retrieval — ensuring that features used in training reflect only information that was available at the historical prediction time, preventing data leakage. Point-in-time correct feature retrieval is the feature engineering equivalent of walk-forward time series cross-validation — both prevent future information from leaking into historical training examples.

Data Mesh — Decentralised Data Ownership

Data mesh (Dehghani, 2019) is an organisational and architectural paradigm, not a technology. Its core insight is that the centralised data team model — where a single platform team owns all data pipelines and all data products — does not scale. As organisations grow, the centralised team becomes a bottleneck: they have the technical skills to build pipelines but not the domain knowledge to build correct pipelines, while the domain teams have the knowledge but not the platform access. Data mesh distributes data ownership to the domain teams that produce and understand the data, treating data as a product with clear owners, SLAs, and consumers.

Data mesh has four principles: (1) Domain-oriented data ownership — each domain team owns their data products end-to-end, from ingestion through transformation to serving. (2) Data as a product — data products have owners, versioned schemas, documented contracts, quality SLAs, and are discoverable by consumers. This is the organisational embodiment of the data contract concept. (3) Self-serve data infrastructure — a central platform team provides infrastructure (lakehouse, compute, governance tools) as a managed platform, so domain teams can build data products without infrastructure expertise. (4) Federated computational governance — policies (access control, privacy, quality standards) are enforced consistently across the mesh via automated tooling, not by a central approval gate. The connection to federated learning is conceptual: both “federated” paradigms distribute ownership while maintaining global consistency, and the same privacy considerations — which data can be shared, with whom, under what governance — apply in both contexts. The data science project management guide covers how cross-functional collaboration between domain teams and the data platform team works in practice within a data mesh organisation.

✦ SUMMARIZE THIS ARTICLE WITH AI

ChatGPT Perplexity Claude Grok

The ETL pipelines, dbt, and Airflow that power Silver and Gold layer transformations are in our ETL Pipelines guide and Data Engineering Fundamentals guide. Data quality monitoring and data contracts that enforce Bronze-Silver-Gold quality standards are in our Data Quality guide. Big Data technologies — Spark, Kafka — that process data at lakehouse scale are in our Big Data Technologies guide. Cloud platforms — Databricks, Snowflake, BigQuery — that implement lakehouse architectures are in our Cloud for Data Scientists guide. Feature engineering and point-in-time correct feature stores that connect the Gold layer to ML training are in our Feature Engineering guide.