📋 KEY INSIGHTS
- Machine learning for genomics applies sequence modelling, graph neural networks, and generative models to the fundamental challenge of decoding the genome: understanding how DNA sequence encodes the instructions for building and operating a living cell. The human genome contains approximately 3 billion base pairs, of which only ~2% codes for proteins (exons) — but the remaining 98% contains regulatory elements (promoters, enhancers, silencers, insulators) that control when and where each gene is expressed. Identifying these regulatory elements, predicting how genetic variants affect gene regulation, and linking regulatory variation to disease are the core problems of functional genomics, and ML is transforming all three. The connection between genomics and NLP is deep: DNA is a 4-letter alphabet (A, C, G, T) sequence, and the same Transformer architectures that model language also model genomes, with nucleotides as tokens.
- Enformer (Avsec et al., Nature Methods 2021) is the landmark sequence-to-function model: a Transformer trained to predict 5313 epigenomic tracks (ATAC-seq open chromatin, ChIP-seq histone marks, RNA-seq gene expression) simultaneously from 196,608 bp of DNA sequence centred on a genomic locus. Enformer’s multi-head attention over long DNA sequences captures enhancer-promoter interactions at distances of 100kb — far beyond the range of convolutional models like DeepSEA and Basset. The model achieves state-of-the-art on the Genomics Benchmarks for sequence-to-function prediction and enables variant effect prediction (compute the difference in predicted epigenomic tracks between reference and alternate allele at any SNP position) that correlates with GWAS fine-mapping results. The connection to our Transformers guide is direct: Enformer uses the standard multi-head attention mechanism described there, adapted for the genomics setting with convolutional stem for local context and a 2048-token context window over 128bp tiles.
- Nucleotide Transformer (NT, Dalla-Torre et al., 2023) and DNABERT-2 are BERT-style foundation models pre-trained on large collections of genomic sequences from thousands of species via masked nucleotide prediction. NT-v2 (500M parameters) achieves state-of-the-art on 18 of 18 genomics benchmark tasks after fine-tuning, outperforming task-specific models trained from scratch. The key advantage of genomic foundation models over task-specific models: transfer learning from the pre-trained sequence representations dramatically reduces the amount of labelled data needed for each downstream task (promoter prediction, splice site detection, variant effect prediction, non-coding RNA classification). The connection to our foundation models guide is direct: genomic foundation models follow the exact pre-train/fine-tune paradigm described there — the DNA sequence is the “text”, masked nucleotide prediction is the “language modelling” objective, and downstream genomics tasks are the “fine-tuning” tasks, with NT-v2 achieving the genomics equivalent of GPT-3’s few-shot learning capability.
- Single-cell RNA sequencing (scRNA-seq) measures gene expression in thousands of individual cells simultaneously, revealing cellular heterogeneity that bulk RNA-seq masks. The key computational challenges: (1) dropout (many genes have zero counts due to low capture efficiency — a gene expressed in a cell may not be detected); (2) batch effects (technical variation between experiments must be removed without removing biological signal); (3) dimensionality (20,000 genes per cell, millions of cells). scVI (Lopez et al., Nature Methods 2018) models scRNA-seq data with a variational autoencoder: the encoder maps the high-dimensional gene count vector to a low-dimensional latent space, the decoder generates gene counts from the latent code using a ZINB (Zero-Inflated Negative Binomial) distribution that explicitly models dropout and overdispersion. scVI learns batch-corrected latent representations that enable downstream analyses: clustering (UMAP + Leiden clustering to identify cell types), differential expression (testing which genes are differentially expressed between cell type clusters), and trajectory inference (pseudotime ordering of cells along differentiation trajectories). The connection to our dimensionality reduction guide is direct: scVI is a probabilistic non-linear dimensionality reduction method for count data — analogous to PCA for single-cell data but with explicit modelling of the data generating process (dropout, overdispersion) rather than assuming Gaussian noise.
- Variant effect prediction models predict the functional impact of genetic variants (SNPs, indels, structural variants) on gene regulation, splicing, and protein function. Sequence-based variant effect prediction uses Enformer (for regulatory variants) or ESM-2 (for coding variants, predicting amino acid change effects) to compute the reference vs. alternate allele difference in predicted functional output — a zero-shot variant effect score that correlates with GWAS fine-mapping p-values and clinical variant pathogenicity classifications (ClinVar). Splicing variant prediction (SpliceAI, Jaganathan et al., Cell 2019) uses a deep residual network trained on annotated splice sites to predict the probability that a sequence position is a donor, acceptor, or neither splice site — enabling detection of deep intronic variants that create cryptic splice sites causing rare genetic diseases. The connection to our drug discovery guide is direct: variant effect prediction is used in therapeutic target validation — identifying whether a gene’s product is causally linked to disease (Mendelian randomisation using genetic variants as instruments) is the key step in selecting drug targets with high genetic support, which are 2-4× more likely to succeed in clinical trials.
- ATAC-seq and chromatin accessibility modelling identifies regulatory elements (open chromatin regions accessible to transcription factors) across cell types and conditions. scATAC-seq measures chromatin accessibility at single-cell resolution, enabling cell-type-specific regulatory analysis. PeakVI (a VAE analogous to scVI for ATAC data) and ArchR are the standard computational tools for scATAC-seq analysis. Transcription factor (TF) motif scanning identifies which TFs are likely to bind each accessible chromatin region by matching the DNA sequence to a library of TF binding motifs (position weight matrices). ChromBPNet (Pampari et al., 2024) uses a residual dilated convolutional network to predict ATAC-seq signal from raw DNA sequence, enabling de novo discovery of TF binding motifs from sequence alone by computing attribution scores (DeepLIFT, integrated gradients from our SHAP and LIME guide) on the predicted chromatin accessibility.
The revolution in genomics data generation — driven by falling sequencing costs (from $3 billion for the first human genome in 2003 to under $200 today) — has produced a tsunami of sequence data that classical bioinformatics tools struggle to interpret. Every person’s genome differs from the reference at ~4-5 million positions, the vast majority in non-coding regulatory regions whose functional effects are poorly characterised. Understanding which of these variants are functionally consequential, in which cell types, and through which molecular mechanisms, is the central challenge of human genomics — and machine learning is becoming indispensable to addressing it. This guide covers the full genomics ML landscape: DNA sequence modelling with Enformer and Nucleotide Transformer, genomic foundation models, single-cell RNA-seq analysis with scVI, ATAC-seq chromatin accessibility modelling, splicing variant prediction, and connections to drug target validation.
DNA Sequence Models, Enformer and Genomic Foundation Models
DNA sequence modelling treats the genome as a language and applies sequence models to predict functional properties from nucleotide sequences. The progression from convolutional models to Transformers mirrors the NLP progression from CNNs to attention: DeepSEA (2015) used a 3-layer CNN over 1kb sequences to predict chromatin features; Basset (2016) extended to larger sequences and more targets; Basenji (2018) introduced dilated convolutions to extend the receptive field to 32kb while maintaining computational efficiency; and Enformer (2021) replaced the dilated conv backbone with a Transformer to capture interactions at 196kb, the scale at which most enhancer-promoter loops operate in mammalian genomes. The fundamental insight motivating Enformer’s long context: gene regulation requires communication between enhancers (which bind transcription factors) and promoters (which initiate transcription) through 3D chromatin looping — these interactions can span 100kb or more of linear DNA sequence, inaccessible to shorter-context convolutional models. Enformer’s architecture: a convolutional stem (6 conv layers with max pooling, reducing resolution from 1bp to 128bp tiles) followed by 11 Transformer blocks with multi-head attention and relative positional encodings, then separate prediction heads for each of 5313 output tracks. The relative positional encoding (using learnable bias terms that decay with distance, as in Transformer-XL) is essential for genomics: the absolute position of a sequence element in the genome is irrelevant; only relative distances between elements matter. The multi-head attention over 1536 tiles (196,608 bp / 128 bp per tile) produces a 1536×1536 attention matrix — tractable with FlashAttention (from our long-context LLMs guide) but would be impossible with naive O(n^2) attention at single-nucleotide resolution. Nucleotide Transformer (NT-v2, 500M parameters) pre-trains on 850 genomes from 850 species via masked token prediction: 15% of nucleotide tokens are masked, and the model is trained to predict the masked nucleotides from bidirectional context. This multi-species pre-training teaches the model evolutionary conservation: nucleotide positions conserved across species (functional elements under purifying selection) are easier to predict than rapidly evolving non-functional positions, and the model learns to use conservation as a signal for functionality. NT-v2 fine-tuned on 18 downstream tasks (promoter prediction, enhancer activity, splice site detection, TF binding, non-coding RNA classification, variant effect prediction) achieves state-of-the-art on all 18, demonstrating that pre-training on raw sequence provides transferable representations that capture regulatory grammar. The connection to our self-supervised learning guide is direct: masked nucleotide prediction is the genomics application of the BERT masked language modelling objective described there — the same mathematical framework (mask tokens, predict masked tokens from bidirectional context, minimise cross-entropy loss) applied to a 4-letter biological alphabet rather than a 30,000-token natural language vocabulary.
| Model | Architecture | Context | Pre-training | Key Task |
|---|---|---|---|---|
| Enformer | Conv stem + Transformer | 196kb | Supervised (ENCODE/GTEx) | Sequence-to-epigenome; variant effect |
| Nucleotide Transformer v2 | BERT (500M) | 12kb | Masked nucleotide (850 genomes) | 18 genomics fine-tune tasks |
| DNABERT-2 | BERT + BPE tokenisation | 512 BPE tokens | Masked token (multi-species) | TF binding, splice, promoter |
| Evo (Arc Institute) | SSM (Mamba, 7B) | 131kb | Next-token (OpenGenome) | Generative genomics; protein design |
| SpliceAI | Deep residual conv | 10kb | Supervised (annotated splicing) | Splice site; cryptic variant detection |
Single-Cell Genomics, scVI and Chromatin Accessibility
Single-cell genomics has transformed biology by revealing the cellular heterogeneity underlying complex tissues: a “liver cell” is not a single cell type but a mixture of hepatocytes, Kupffer cells, stellate cells, endothelial cells, and immune cells, each with distinct gene expression profiles, functions, and responses to drugs. scRNA-seq measures this heterogeneity at the transcriptome level, and scATAC-seq measures it at the epigenome level. The Human Cell Atlas project aims to map every cell type in the human body using scRNA-seq, creating a reference map analogous to the reference genome for cellular biology. scVI’s probabilistic model: each cell i has a latent representation z_i ∈ R^10 (learned by the encoder) and a batch indicator s_i (which experimental batch the cell came from). The decoder generates the observed gene count x_ig for gene g in cell i from the ZINB distribution: with probability π_ig the gene is a dropout (count = 0 regardless of expression); otherwise the count follows a negative binomial with mean μ_ig = l_i · f_g(z_i, s_i) (where l_i is the library size and f_g is the decoder network) and dispersion θ_g. Training with ELBO (Evidence Lower BOund) optimisation jointly learns the encoder, decoder, and dispersion parameters. The resulting latent space z_i is batch-corrected (the batch indicator is included in the decoder but not the encoder, forcing the latent space to be batch-invariant) and biologically meaningful (similar cell types cluster together). Cell type annotation uses the UMAP projection of z_i combined with marker gene expression (each known cell type has a set of marker genes with high expression) to assign cell type labels. scANVI (semi-supervised extension of scVI) incorporates known cell type labels for a subset of cells as supervision, learning cell type-conditional latent representations that enable transfer of labels from labelled reference datasets to unlabelled query datasets. The connection to our active learning guide is direct: deciding which cells to annotate manually in a large scRNA-seq dataset is a classic active learning problem — selecting cells with high latent space uncertainty (cells near cluster boundaries that are ambiguously assigned to two or more cell types) for expert annotation maximises the value of limited annotation effort, as described in that guide’s pool-based uncertainty sampling strategy. ChromBPNet and GRCh38-based accessibility models enable in silico saturation mutagenesis: systematically mutate every position in an accessible chromatin region and measure the predicted effect on accessibility, creating a high-resolution map of functionally important nucleotides. This in silico mutagenesis is used to identify TF binding sites within regulatory elements, design synthetic enhancers with desired cell-type-specificity, and predict the regulatory impact of rare non-coding variants in clinical sequencing. The connection to our SHAP and LIME guide is direct: DeepLIFT and integrated gradients for attribution to input nucleotides in ChromBPNet and Enformer are the genomics application of the neural network attribution methods described there — assigning credit to individual nucleotide positions for the model’s predicted output, the same way SHAP assigns credit to input features in tabular ML models.
✦ SUMMARIZE THIS ARTICLE WITH AI
Foundation models (Nucleotide Transformer and Evo as genomic foundation models following the pre-train/fine-tune paradigm) are in our Foundation Models guide. Self-supervised learning (masked nucleotide prediction as BERT applied to DNA) is in our Self-Supervised Learning guide. ML for drug discovery (variant effect prediction for target validation; Mendelian randomisation) is in our Drug Discovery guide. Dimensionality reduction (scVI as probabilistic non-linear DR for count data) is in our Dimensionality Reduction guide. SHAP and LIME (DeepLIFT and integrated gradients for nucleotide attribution) are in our SHAP & LIME guide. Protein structure prediction (protein language models pre-trained on sequence data, analogous to genomic foundation models) is in our Protein Structure guide. Long-context LLMs (FlashAttention enabling Enformer’s 1536-tile attention) are in our Long-Context LLMs guide.



