📋 KEY INSIGHTS
- Foundation models are large-scale models pre-trained on broad, diverse datasets using self-supervised learning, then fine-tuned or zero-shot adapted for downstream tasks. While the term is most associated with LLMs (GPT-4, Claude, Llama), the foundation model paradigm has been successfully extended to time series (Lag-Llama, TimesFM, MOIRAI), images and video (ViT, SAM, Stable Diffusion, Sora), structured tabular data (TabPFN, CTGAN), genomics (Nucleotide Transformer, DNABERT, ESM-2 for proteins), and multimodal inputs combining text, images, audio, and structured data.
- Time series foundation models pre-train on large collections of diverse time series data to learn universal time series representations transferable to new forecasting and classification tasks with zero-shot or few-shot adaptation. TimesFM (Google, 2024) pre-trains a decoder-only Transformer on 100B time points from Google Trends, Wikipedia traffic, synthetic data, and other public sources, achieving zero-shot forecasting accuracy competitive with task-specific models on standard benchmarks (ETTH1, ETTH2, Weather) without any fine-tuning on the target dataset. MOIRAI (Salesforce, 2024) uses a masked encoder architecture and trains on LOTSA — a 27B time point collection across 9 domains — with any-variate tokenisation that handles varying numbers of covariates without architecture changes.
- Vision foundation models pre-trained on web-scale images have fundamentally changed computer vision pipelines: ViT (Vision Transformer, Dosovitskiy et al., 2020) pre-trained with supervised ImageNet labels or self-supervised DINO/DINOv2 objectives provides transferable visual features that outperform domain-specific CNNs on most vision tasks when fine-tuned with limited domain data. SAM (Segment Anything Model, Meta AI, 2023) is the CV analogue of GPT: pre-trained on 1.1 billion masks from 11 million images using an interactive prompting paradigm, SAM segments any object in any image given a point, box, or text prompt — enabling zero-shot segmentation on domains (medical imaging, satellite imagery, manufacturing defects) far outside its training distribution.
- Protein language models (PLMs) treat amino acid sequences as “language” — each amino acid is a “word” and protein sequences are “sentences” — and pre-train large Transformer models on hundreds of millions of protein sequences from UniRef and UniProt using masked language modelling. ESM-2 (Meta AI, 2022) pre-trains models up to 15 billion parameters on 250M unique protein sequences, learning representations that encode structural, functional, and evolutionary information without any structural labels. ESM-2 embeddings enable state-of-the-art zero-shot prediction of protein structure, stability, and function, and ESMFold uses ESM-2 representations to predict protein 3D structure 60× faster than AlphaFold2 by eliminating the multiple sequence alignment step.
- Multimodal foundation models (GPT-4V, Gemini, Claude) accept and generate combinations of text, images, audio, and other modalities within a single model. The dominant training paradigm: pre-train a large language model backbone, pre-train a vision encoder (ViT or CLIP-style), then connect the two with a vision-language alignment module (Flamingo’s cross-attention layers, LLaVA’s linear projection, InstructBLIP’s Q-Former) and fine-tune on instruction-following multimodal data. This creates models capable of image captioning, visual question answering, document understanding, and cross-modal reasoning without task-specific architecture design. The connection to our multimodal AI guide is direct: that guide covers CLIP, LLaVA, and GPT-4V — the building blocks from which modern multimodal foundation models are assembled.
- Foundation model adaptation for domain-specific applications follows a hierarchy of increasing cost and performance: zero-shot prompting (no training required; works surprisingly well for tasks similar to pre-training data); few-shot in-context learning (providing 3-10 labelled examples in the context window); parameter-efficient fine-tuning with LoRA/QLoRA (updating 0.1-1% of parameters on domain-specific labelled data); full fine-tuning on large domain datasets (expensive but maximally powerful). The connection to our LLM evaluation guide is direct: evaluating whether a foundation model is achieving the expected capability on a domain task — using benchmark metrics, task-specific test sets, and LLM-as-judge — is essential before deploying any foundation model in production.
The term “foundation model” (Bommasani et al., Stanford HAI, 2021) captures a paradigm shift: rather than training task-specific models from scratch, the field has converged on pre-training large models on massive, diverse datasets and then adapting them to specific downstream tasks. The efficiency gains are dramatic — a 7B parameter language model pre-trained on 2 trillion tokens can be fine-tuned for clinical note summarisation on a few thousand examples, outperforming models trained on those examples from scratch by a large margin. This paradigm shift was initially confined to NLP but has now spread across data modalities: time series, images, genomic sequences, molecules, proteins, structured tabular data, and combinations thereof. This guide surveys the landscape of foundation models beyond LLMs, covering the architectures, pre-training strategies, adaptation methods, and practical considerations for each modality.
Time Series Foundation Models — TimesFM, MOIRAI and Lag-Llama
Time series forecasting has historically been a domain where task-specific models (ARIMA, Prophet, LSTM, Temporal Fusion Transformer) trained per-dataset outperformed general models. Time series foundation models challenge this assumption by pre-training on sufficiently large and diverse collections of time series that the learned representations generalise across domains. TimesFM (Google, 2024) uses a decoder-only Transformer architecture (similar to GPT-2) that takes a context window of past values and autoregressively generates future values one patch at a time (a patch = 32 consecutive time points). The model is pre-trained on a 100B time point corpus spanning Wikipedia page views, Google Trends, synthetic data from ARIMA/ETS models, and other public datasets. Zero-shot evaluation on standard forecasting benchmarks (ETTH1, ETTH2, ECL, Traffic, Weather) shows TimesFM matching or exceeding task-specific models including PatchTST and N-HiTS on most datasets — without seeing a single example from the target dataset. MOIRAI (Salesforce) differentiates by targeting universal time series modelling rather than just forecasting: its any-variate tokenisation allows the same model to handle univariate and multivariate inputs, while training on LOTSA (27B time points, 9 domains) gives broader generalisation than TimesFM. The connection to our time series deep learning guide is direct: TimesFM and MOIRAI build on PatchTST’s patch tokenisation and Transformer architecture, extending the supervised PatchTST to the self-supervised pre-training paradigm. The connection to our time series analysis guide is direct: time series foundation models are evaluated on the same benchmark datasets (ETT, ETTH, ECL) and against the same baselines (N-BEATS, TFT, PatchTST) as task-specific deep learning models.
| Foundation Model | Modality | Pre-training Scale | Zero-Shot Capable? | Adaptation Method |
|---|---|---|---|---|
| TimesFM (Google) | Time series | 100B time points | Yes (zero-shot forecasting) | Fine-tune on domain data |
| MOIRAI (Salesforce) | Time series | 27B points, 9 domains | Yes | Fine-tune; in-context learning |
| SAM (Meta AI) | Images (segmentation) | 1.1B masks, 11M images | Yes (prompt-based) | Prompt engineering; MedSAM fine-tune |
| ESM-2 (Meta AI) | Protein sequences | 250M proteins | Yes (zero-shot mutation effect) | Linear probe; LoRA fine-tune |
| Nucleotide Transformer | DNA sequences | 3,202 genomes | Yes | Fine-tune on regulatory tasks |
| TabPFN | Tabular (small datasets) | Synthetic prior data | Yes (<1000 rows, in-context) | Direct inference (no fine-tune) |
Vision Foundation Models, Protein Language Models and Multimodal Systems
SAM (Segment Anything Model) represents the vision foundation model paradigm most clearly: trained on a dataset of 1.1 billion masks collected via an interactive annotation pipeline where annotators use the model’s own predictions to annotate faster (a form of model-in-the-loop data collection), SAM generalises to segment any object given a minimal prompt. MedSAM (Ma et al., 2024) fine-tunes SAM on 1.5 million medical image-mask pairs across CT, MRI, ultrasound, and pathology modalities — achieving better medical segmentation accuracy than SAM zero-shot while retaining the prompt-based interface. The connection to our healthcare guide is direct: medical image segmentation is the highest-value computer vision task in clinical settings, and MedSAM dramatically lowers the data requirements for deploying segmentation models in new clinical domains. DINOv2 (Meta AI, 2023) produces the best general-purpose visual features of any self-supervised ViT model: pre-trained with a self-distillation objective on a curated 142M image dataset (LVD-142M), DINOv2 features achieve competitive linear probe accuracy on ImageNet without any fine-tuning, and state-of-the-art on dense prediction tasks (depth estimation, semantic segmentation) when used as a frozen backbone — essentially a universal visual feature extractor. The connection to our self-supervised learning guide is direct: DINOv2 uses the DINO (self-DIstillation with NO labels) self-supervised objective described there. Protein language models connect data science to molecular biology: ESM-2 embeddings of amino acid sequences encode the evolutionary constraints that determine protein structure and function, enabling machine learning models for protein engineering (predicting mutation effects on stability and binding affinity), drug discovery (predicting protein-ligand binding), and synthetic biology (designing novel proteins with desired functions). TabPFN (Prior-data Fitted Networks, Hollmann et al., 2023) is the most surprising foundation model: a Transformer pre-trained on 18M synthetic tabular classification datasets generated from a Bayesian prior over data-generating processes, TabPFN performs in-context learning at inference time — the entire training dataset is provided as context, and the model predicts labels for test points by pattern matching against the in-context examples — achieving state-of-the-art accuracy on small tabular datasets (<1000 rows) with <1 second inference time and no gradient updates.
✦ SUMMARIZE THIS ARTICLE WITH AI
ChatGPTTime series deep learning models that time series foundation models extend and replace are in our Time Series Deep Learning guide. Multimodal AI (CLIP, LLaVA, GPT-4V) as the building blocks of multimodal foundation models are in our Multimodal AI guide. LLM fine-tuning (LoRA/QLoRA) for domain adaptation of foundation models is in our LLM Fine-Tuning guide. LLM evaluation for assessing foundation model adaptation quality is in our LLM Evaluation guide. Self-supervised learning (DINO, contrastive objectives) used in vision foundation model pre-training is in our Self-Supervised Learning guide. Data science for healthcare (MedSAM, clinical foundation models) is in our Healthcare guide.
