Federated Learning – Privacy-Preserving Distributed Machine Learning

Privacy is increasingly a hard constraint in machine learning, not a soft preference. GDPR, HIPAA, CCPA, and sector-specific regulations restrict or prohibit moving personal data across organisational or jurisdictional boundaries for model training. The traditional “aggregate everything in a data lake, train centrally” approach — which has defined data science practice for the past decade — is no longer viable for many of the most valuable ML applications: medical diagnosis across hospital networks, fraud detection across competing banks, personalised keyboard prediction on personal devices. Federated learning is the ML paradigm designed for this constraint: models are trained where the data lives, and only model updates, not raw data, travel across the network. This guide covers the federated learning framework, the FedAvg algorithm and its variants, the central technical challenge of data heterogeneity, communication efficiency, privacy guarantees through differential privacy, and the practical considerations for deploying federated learning in cross-silo and cross-device settings.

The Federated Learning Framework

Federated learning involves three actors: a central server (or orchestrator), a set of participating clients (devices, hospitals, banks, or any data-holding entity), and a shared model architecture that all parties agree on. The training loop iterates as follows: (1) the server broadcasts the current global model weights to a subset of clients; (2) each selected client runs several epochs of local training on its private data, starting from the received weights; (3) each client sends its updated weights (or the gradient update — the difference from the received weights) back to the server; (4) the server aggregates the received updates into a new global model and returns to step 1. Raw data never leaves any client — only model weights or gradients travel over the network.

This simple loop raises immediate questions. Why not just send gradients rather than weights? Can the server infer anything about the raw data from the gradients? How do we handle clients that drop out mid-round? How do we aggregate updates from clients with very different local dataset sizes? These engineering and privacy questions define the space of federated learning variants and motivate the technical methods covered in this guide. The connection to standard gradient-based optimisation is direct — federated learning is fundamentally distributed stochastic gradient descent, with the twist that each worker (client) runs multiple local steps before synchronising. The neural network architectures and regularisation techniques used in centralised learning transfer directly to federated settings; only the optimisation procedure changes.

FedAvg and the Data Heterogeneity Challenge

Federated Averaging (FedAvg) (McMahan et al., 2017) is the foundational algorithm. In each communication round, the server selects a random subset C of clients, sends them the current global model w_t, each client k runs E epochs of SGD on its local dataset D_k producing updated weights w_k^(t+1), and the server aggregates: w_(t+1) = Σ_k (|D_k|/|D|) × w_k^(t+1), weighting by local dataset size. FedAvg reduces communication rounds by 10–100× compared to one-gradient-per-step federated SGD, because each client does meaningful local training before synchronising.

FedAvg works well when client data is IID — each client’s data is drawn from the same distribution as the global dataset. In practice, federated data is severely non-IID: a hospital specialising in paediatrics has a very different patient distribution than a general hospital; a user’s personal keyboard data reflects their idiosyncratic writing style and vocabulary. Under severe non-IID data, FedAvg’s local updates pull in different, conflicting directions (client drift), causing the global model to oscillate or converge to a poor solution. FedProx addresses this by adding an L2 proximal term to each client’s local objective, penalising drift from the global model: L_k(w) + (μ/2)||w − w_t||². The proximal term is a form of L2 regularisation anchored to the server’s model rather than zero, and its connection to weight decay makes FedProx intuitive for practitioners familiar with standard regularisation. SCAFFOLD uses control variates to explicitly correct for client drift, achieving provably faster convergence under heterogeneity at the cost of double the communication per round.

Privacy Guarantees — Differential Privacy in Federated Learning

A critical misconception is that federated learning automatically provides strong privacy. Sharing gradients rather than raw data reduces privacy exposure, but does not eliminate it: gradient inversion attacks (Zhu et al., 2019) demonstrated that high-fidelity reconstructions of individual training images can be recovered from gradient updates, and property inference attacks can reveal statistical properties of a client’s local dataset from the global model. Strong privacy requires explicit privacy-preserving mechanisms on top of federated learning.

Differential privacy (DP) provides formal privacy guarantees for federated learning via DP-FedAvg: each client clips their gradient update to a maximum L2 norm C (preventing any single example from dominating the gradient), then adds calibrated Gaussian noise N(0, σ²I) to the clipped gradient before sending it to the server. The noise magnitude σ is calibrated to the desired privacy budget ε — the same differential privacy framework used for private synthetic data generation. With DP-FedAvg, the central server learns nothing about any individual client’s data beyond what would be learnable from a world in which that client did not participate — the formal guarantee of (ε, δ)-differential privacy. The tradeoff between privacy and utility (model accuracy) is fundamental: stronger privacy requires more noise, which degrades the model’s accuracy. Choosing the right privacy budget requires understanding the threat model and the domain’s regulatory requirements, a topic covered in our data science ethics guide.

Secure aggregation is a complementary cryptographic approach: clients encrypt their model updates such that the server can compute the aggregate without seeing any individual client’s update, using protocols based on secret sharing or homomorphic encryption. Secure aggregation prevents the server from performing gradient inversion attacks, even without differential privacy noise. The combination of secure aggregation (preventing server-side reconstruction) and differential privacy (preventing inference from the aggregate) provides defence-in-depth privacy protection. The cloud platform guide covers the managed federated learning services (Google Vertex AI FL, AWS SageMaker FL) that implement these privacy mechanisms as managed infrastructure, abstracting away the cryptographic and distributed systems complexity.

✦ SUMMARIZE THIS ARTICLE WITH AI

ChatGPT Perplexity Claude Grok

Differential privacy for synthetic data — the same privacy mechanism applied to federated gradients — is in our Synthetic Data Generation guide. The neural network architectures and optimisation algorithms that federated learning trains are in our Neural Network Architectures and Calculus for ML guides. Data science ethics and privacy regulation context — GDPR, HIPAA, fairness — is in our Data Science Ethics guide. MLOps for federated model management — tracking experiments, versioning models across clients — is in our MLOps guide. Cloud ML platforms that offer managed federated learning infrastructure are in our Cloud for Data Scientists guide.