Differential Privacy for Machine Learning – DP-SGD, Privacy Budgets and Formal Guarantees

Data privacy in machine learning has moved from a compliance checkbox to a technical discipline. The Cambridge Analytica scandal, GDPR enforcement actions, and high-profile re-identification attacks on “anonymised” datasets have demonstrated that naive de-identification — removing names and obvious identifiers — provides weak privacy guarantees against adversaries with access to auxiliary information. Differential privacy offers a rigorous alternative: a mathematical definition of privacy that quantifies the information leaked about any individual and provides composable, auditable guarantees that hold under realistic adversarial conditions. As regulatory pressure increases and data-sharing requirements grow — between hospitals, across federated datasets, in synthetic data releases — differential privacy is becoming a standard tool in the data science privacy toolkit. This guide covers the formal DP definition, the privacy-utility tradeoff, DP-SGD for private model training, local differential privacy, the Rényi DP accounting framework, and practical implementations with Opacus and Google’s DP library.

Formal Definition, Privacy Budget and the Gaussian Mechanism

The formal definition of differential privacy (Dwork et al., 2006) is built around the concept of adjacent datasets: D and D’ are adjacent if they differ in exactly one individual’s record (one row added, removed, or changed). A mechanism M is (ε, δ)-differentially private if for all adjacent D, D’ and all measurable output sets S: P[M(D) ∈ S] ≤ e^ε × P[M(D’) ∈ S] + δ. Intuitively, this means that no single individual’s data can change the distribution of outputs by more than a factor of e^ε (plus a δ probability of failure), making it impossible for an adversary observing M’s output to determine with high confidence whether any particular individual was in the dataset. The privacy budget ε (epsilon) is the primary privacy parameter: ε = 0 means perfect privacy (outputs are identical regardless of the data); ε = 1 is strong privacy; ε = 10 is weak privacy. δ is typically set to a value smaller than 1/n (where n is the dataset size), ensuring the failure event is negligible.

The Gaussian mechanism achieves (ε, δ)-DP for real-valued queries by adding Gaussian noise calibrated to the query’s sensitivity — the maximum change in the query’s output when a single individual’s record is changed. For a query f: D → ℝ^d with L2 sensitivity Δ₂f = max_{adjacent D,D’} ||f(D) − f(D’)||₂, the Gaussian mechanism M(D) = f(D) + N(0, σ²I) achieves (ε, δ)-DP with noise standard deviation σ = Δ₂f × √(2 ln(1.25/δ)) / ε. The Laplace mechanism achieves (ε, 0)-DP (pure DP, without the δ failure probability) using Laplace noise, but Gaussian noise is preferred in practice because it has better composition properties and is better suited to high-dimensional outputs (like gradients). The connection to our Bayesian statistics framework is instructive: the Gaussian mechanism is a form of data perturbation that can be interpreted as adding a prior distribution over the query output, and the privacy guarantee is a Bayesian posterior indistinguishability bound.

DP-SGD — Differentially Private Model Training

DP-SGD (Abadi et al., 2016) achieves differential privacy for neural network training by modifying the standard mini-batch SGD update. Standard SGD computes the mean gradient over a batch and updates parameters; DP-SGD adds two steps: (1) Per-sample gradient clipping: compute the gradient of the loss for each sample individually (not averaged), clip each gradient vector to have L2 norm at most C (the clipping threshold), then average the clipped gradients. This bounding of each sample’s gradient by C sets the sensitivity of the gradient computation — the maximum change any single sample can cause in the gradient estimate. (2) Gaussian noise addition: add isotropic Gaussian noise N(0, σ²C²I) to the average clipped gradient before the parameter update, where σ is the noise multiplier that controls the privacy-utility tradeoff. Larger σ gives stronger privacy (more noise) at the cost of slower convergence and worse final accuracy.

The privacy cost of DP-SGD across T training steps is tracked by privacy accountants — algorithms that compute the (ε, δ) guarantee for the composition of T Gaussian mechanisms. Moments accountant and Rényi DP (RDP) accounting are the standard methods: RDP tracks the privacy cost in terms of the Rényi divergence between the output distributions on adjacent datasets, which composes tightly across multiple applications. The Opacus library (PyTorch) implements DP-SGD with RDP accounting in a few lines:

from opacus import PrivacyEngine

privacy_engine = PrivacyEngine()
model, optimiser, data_loader = privacy_engine.make_private_with_epsilon(
    module=model, optimizer=optimiser, data_loader=data_loader,
    epochs=20, target_epsilon=1.0, target_delta=1e-5, max_grad_norm=1.2
)
# train as normal — Opacus handles clipping and noise automatically

The privacy-utility tradeoff is the central practical challenge of DP-SGD. On CIFAR-10 (50K training images), DP-SGD with ε=1 achieves approximately 70–75% accuracy compared to 93% for non-private training — a substantial cost. The tradeoff improves with dataset scale: on ImageNet (1.2M images), DP ResNet-50 with ε=10 achieves ~74% accuracy vs 76% without privacy. The most important levers for improving utility under DP-SGD: (1) Pre-training on a public dataset (no privacy cost) and fine-tuning with DP-SGD on the private dataset — the transfer learning approach dramatically reduces the accuracy cost of DP because the model starts near convergence; (2) LoRA-based DP fine-tuning — applying DP-SGD only to the small LoRA adapter parameters (fewer parameters = less noise needed for the same privacy budget); (3) Larger batch sizes — DP-SGD’s noise-to-signal ratio improves with batch size, so large-batch training with lower learning rate is preferred.

Local DP, Synthetic Data and Privacy in Practice

Central DP (the model above) trusts the data curator: individual records are sent to a trusted server, which applies the DP mechanism. Local DP removes this trust assumption by requiring each individual to apply the DP mechanism to their own data before sharing it. Randomised response for a binary attribute b: report the truth with probability e^ε/(e^ε + 1) and the opposite with probability 1/(e^ε + 1). The aggregator receives noisy reports and can estimate the true proportion via debiasing, but cannot determine any individual’s true value with certainty. Google’s RAPPOR and Apple’s PrivacyPreservingDataCollection use LDP to collect app usage statistics from billions of devices — the scale makes the aggregated statistics accurate despite the per-device noise. The connection to our federated learning guide is direct: federated learning is often combined with LDP or central DP — either the local gradient updates are LDP before aggregation, or the aggregated model update is DP before being broadcast.

DP synthetic data generation — training a DP generative model (DP-GAN, PrivBayes, DP-VAE) and releasing synthetic samples — is the most practical approach for data sharing and publication because it decouples privacy protection from analytical use: downstream analysts consume synthetic data using standard tools with no DP machinery. The synthetic dataset carries a formal privacy guarantee: any query answerable from the synthetic data leaks at most ε-DP information about any individual in the training data. PrivBayes (a DP Bayesian network) and DP-CTGAN are the standard tools for tabular data; DP-Diffusion (a DP-SGD-trained diffusion model) achieves the best quality for image data. The connection to our synthetic data generation guide is direct: DP synthetic data generation is the privacy-preserving subset of the broader synthetic data generation toolkit, with the additional formal guarantee that distinguishes it from non-private synthetic methods.

✦ SUMMARIZE THIS ARTICLE WITH AI

ChatGPT Perplexity Claude Grok

Federated learning — the distributed training paradigm that pairs naturally with differential privacy — is in our Federated Learning guide. Synthetic data generation — both private (DP) and non-private — is in our Synthetic Data Generation guide. Transfer learning and LoRA fine-tuning that reduce the privacy-utility cost of DP-SGD are in our Transfer Learning guide and LLM Fine-Tuning guide. Data science ethics and responsible AI principles that motivate privacy engineering are in our Data Science Ethics guide. Data quality and governance frameworks that complement DP in regulated industries are in our Data Quality guide.