Self-Supervised and Contrastive Learning – SimCLR, BYOL, DINO and Masked Autoencoders

The history of machine learning is largely a history of finding cheaper supervision signals. Fully supervised learning requires expensive human-annotated labels; transfer learning reuses representations learned on large labelled datasets; active learning selects the most informative examples to label. Self-supervised learning takes a different path entirely: it creates supervision from the data itself, requiring no human labels at all. By designing pretext tasks — masking and predicting tokens, contrasting augmented views, predicting rotations — SSL methods extract a rich training signal from the statistical structure of the data. The resulting representations transfer remarkably well to downstream tasks: a model pre-trained with SSL often matches or exceeds a model trained from scratch on the full supervised dataset, even when fine-tuned on only a fraction of the labels. This guide covers the major families of self-supervised learning — contrastive methods (SimCLR, MoCo, BYOL, DINO), masked autoencoders (MAE, BERT), and the unified understanding of what these methods actually learn — along with practical guidance for using SSL pre-trained models in data science workflows.

Contrastive Learning — SimCLR and the Framework

Contrastive learning defines the learning problem as: given a batch of N images, create two augmented views of each image (applying random crops, colour jitter, flips, and Gaussian blur independently), then train the encoder to produce representations where the two views of the same image are similar (high cosine similarity) and representations of different images are dissimilar (low cosine similarity). The loss function — InfoNCE (or NT-Xent) — is a softmax cross-entropy over the similarities within the batch: for each anchor view, it treats the corresponding augmented view as the positive and all 2(N−1) other views as negatives. Minimising this loss forces the encoder to find representations that are invariant to the applied augmentations — which are deliberately chosen to be augmentations that do not change semantic content (a crop and a colour-jittered crop of the same image are semantically identical), so invariance to these augmentations captures semantic similarity.

SimCLR (Chen et al., 2020) is the clean, minimal instantiation of this framework: a ResNet encoder, a small MLP projection head (which maps encoder outputs to the space where the contrastive loss is applied), and NT-Xent loss over a large batch. SimCLR’s key insight is that a non-linear projection head is essential — the contrastive loss shapes the projection head representations, not the encoder representations, so the encoder is free to maintain more information useful for downstream tasks. After pre-training, the projection head is discarded and only the encoder is fine-tuned. The connection to the ResNet and ViT architectures is direct — SimCLR uses these as its backbone encoder, and the quality of the pre-training scales with backbone size. SimCLR also demonstrated a scaling law for contrastive learning: larger batches (more negatives), longer training, and stronger augmentation all improve transfer performance, with no sign of saturation.

MoCo (He et al., 2020) addressed SimCLR’s requirement for very large batches (4096–8192 images for sufficient negatives) by maintaining a dynamic queue of negative examples from past batches and a momentum-updated key encoder. The momentum encoder updates slowly (exponential moving average of the query encoder), ensuring that the negative queue remains consistent with the current query encoder representations. MoCo achieves SimCLR-quality representations with a standard batch size of 256 — far more memory-efficient — using a queue of 65,536 negatives that spans many past batches. This engineering solution connects directly to the gradient-based optimisation framework: the query encoder receives gradients, while the key encoder is updated without gradients via momentum averaging.

BYOL and DINO — SSL Without Negative Pairs

BYOL (Bootstrap Your Own Latent, Grill et al., 2020) demonstrated the surprising result that contrastive learning does not require negative pairs to avoid representational collapse. BYOL uses two networks: an online network (updated by gradient descent) with a predictor head, and a target network (updated by exponential moving average of the online network, without gradients). The online network is trained to predict the target network’s representation of a different augmented view of the same image. The stop-gradient on the target network, combined with the predictor head asymmetry, prevents trivial solutions (both networks collapsing to a constant representation). Why this works without negatives is still an active theoretical question, but empirically BYOL matches SimCLR quality with a standard batch size of 256 and no negative sampling.

DINO (Self-DIstillation with NO labels, Caron et al., 2021) applies the BYOL-style self-distillation framework to Vision Transformers (ViT), with the additional insight that centering and sharpening the teacher’s output distribution prevent collapse more reliably than the predictor head. DINO’s most striking property is what its ViT attention maps capture without any supervision: the attention heads spontaneously learn to attend to semantically meaningful regions of images — object boundaries, body parts, text regions — producing segmentation-quality attention maps with no segmentation supervision. This emergent segmentation reflects the rich structure encoded in the DINO representations, which transfer to object detection, depth estimation, and image retrieval without task-specific fine-tuning. The connection to our Transformers and Attention guide is direct — DINO’s attention maps are the self-attention maps of a ViT, and understanding their structure requires understanding how multi-head self-attention works.

Masked Autoencoders and BERT-Style Pre-training

While contrastive methods dominate SSL for vision, the NLP field took a different path with BERT (Devlin et al., 2018): mask random tokens in the input and train the model to predict them from context. This masked language modelling (MLM) objective is a form of self-supervised learning that creates supervision from the text itself — no human labels needed. The resulting representations encode rich syntactic and semantic information that transfers effectively to downstream tasks (classification, NER, QA) with minimal fine-tuning. Our Transformers guide covers the BERT architecture and MLM objective in detail; our NLP pipeline guide covers how BERT embeddings are used in practice.

MAE (Masked Autoencoders, He et al., 2021) brings the BERT-style masking paradigm to vision: randomly mask 75% of image patches, encode only the visible patches with a ViT encoder, then decode the masked patches from the encoder output plus mask tokens using a lightweight decoder. The high masking ratio (75% vs BERT’s 15%) is key — it forces the model to learn a high-level semantic understanding of the image rather than just interpolating from adjacent pixels. After pre-training, the decoder is discarded and the ViT encoder is fine-tuned for downstream tasks. MAE achieves 86.9% top-1 on ImageNet with a ViT-Huge backbone, matching supervised pre-training performance while being significantly faster to train. The autoencoder architecture underlying MAE — encoder compresses to a latent space, decoder reconstructs — is covered in our dimensionality reduction guide, and the ViT backbone is covered in our neural network architectures guide.

For data scientists, the practical workflow is: (1) select a pre-trained SSL model appropriate for your domain (DINO or MAE for vision, BERT/RoBERTa for text, wav2vec 2.0 for audio); (2) extract representations from the SSL encoder for your task’s inputs; (3) fine-tune on your labelled data using the SSL representations as initialisation. This workflow is covered in depth in our transfer learning guide. The sample efficiency gains — achieving competitive accuracy with 1–10% of the labelled data — directly reduce the annotation cost that drives active learning strategies, and the two approaches are naturally combined: use SSL pre-training for feature quality, active learning for annotation selection.

✦ SUMMARIZE THIS ARTICLE WITH AI

ChatGPT Perplexity Claude Grok

The Transformer and ViT architectures used by DINO, MAE, and BERT are in our Transformers guide. Transfer learning and fine-tuning SSL pre-trained models for downstream tasks is in our Transfer Learning guide. Active learning combined with SSL representations for maximum annotation efficiency is in our Active Learning guide. Dimensionality reduction — PCA, autoencoders — as the conceptual predecessor to MAE’s encoder-decoder structure is in our Dimensionality Reduction guide. The word embeddings and BERT embeddings that SSL produces for NLP are covered in our Word Embeddings guide and NLP Pipeline guide.