Sunday, November 22, 2026
HomeData ScienceGraph Foundation Models – OFA, GraphGPT, Graph RAG and Molecular Pre-Training

Graph Foundation Models – OFA, GraphGPT, Graph RAG and Molecular Pre-Training

Table of Content

📋 KEY INSIGHTS

  • Graph foundation models (GFMs) aim to do for graphs what LLMs have done for text: pre-train a single large model on massive and diverse graph data, then fine-tune or prompt it for downstream graph tasks without task-specific training from scratch. The motivation is compelling — the graph learning community has historically trained a separate GNN for each task (node classification, link prediction, graph classification) on each dataset, losing the ability to transfer representations across tasks and datasets. But realising graph foundation models faces unique challenges absent from LLMs: graphs lack a natural token sequence (node ordering is arbitrary), graph structure varies wildly across domains (social networks vs. molecular graphs vs. knowledge graphs vs. citation networks), and the inductive bias needed for one graph domain may be counterproductive for another.
  • OFA (One for All, Liu et al., 2023) is the first graph foundation model that achieves strong performance across diverse graph datasets (node classification on ogbn-arxiv, Cora, ogbn-products; link prediction on ogbl-collab; graph classification on MUTAG, IMDB-B) using a single model without fine-tuning. The key innovation: encode both graph structure and node/edge text attributes into a unified representation using a text-attributed graph, then apply a frozen LLM (DeBERTa) to generate node embeddings from text features. The LLM’s pre-trained text representations provide transferable semantic features that capture domain-invariant structure (similar node labels in different domains map to nearby representations), enabling zero-shot and few-shot transfer across datasets. The connection to our foundation models guide is direct: OFA is the graph instantiation of the foundation model paradigm described there — pre-train a large model on diverse data, then transfer to downstream tasks without task-specific training, the same paradigm that enables GPT-4’s few-shot generalisation and CLIP’s zero-shot image classification.
  • GraphGPT (Tang et al., 2023) and InstructGraph (Wang et al., 2023) take a different approach: align a pre-trained GNN with an LLM to enable graph reasoning through natural language. GraphGPT uses a graph-text alignment pre-training stage (aligning GNN node embeddings with LLM token embeddings using a contrastive objective) followed by instruction fine-tuning on graph question-answering datasets (asking the model to describe graph properties, predict node labels, or identify graph patterns in natural language). The result: a model that can answer complex graph questions in natural language (“which nodes are most central?”, “what is the likely label of node 42 given its neighborhood structure?”) — a qualitative leap beyond the numeric predictions of classical GNNs. The connection to our multimodal AI guide is direct: graph-text alignment in GraphGPT is the graph-modality version of image-text alignment in CLIP described there — the graph encoder (GNN) plays the role of the vision encoder (ViT), the text encoder plays the same role, and the contrastive alignment objective (maximise cosine similarity between matched graph-text pairs, minimise between unmatched pairs) is identical to CLIP’s pre-training objective.
  • Graph RAG (Edge et al., 2024, Microsoft) uses a knowledge graph as the retrieval index for a retrieval-augmented generation (RAG) system, enabling graph-aware reasoning over large document corpora. The pipeline: extract entities and relationships from documents to build a knowledge graph (using LLM-based IE from our NLP applications guide), store the knowledge graph in a graph database (Neo4j, from our knowledge graphs guide), then at query time retrieve relevant subgraphs (entities within k hops of query entities) and use the retrieved subgraph structure as additional context for the LLM’s answer generation. Graph RAG outperforms flat vector-RAG (pure dense retrieval) on questions requiring multi-hop reasoning across multiple documents — the graph structure explicitly encodes the relationships between entities that flat vector retrieval must implicitly infer. The connection to our Advanced RAG guide is direct: Graph RAG is an advanced RAG variant described in that guide’s section on structured knowledge retrieval — the knowledge graph provides the structured index that enables path-based multi-hop retrieval, complementing the dense vector index used for semantic similarity retrieval.
  • Molecular foundation models represent the most mature and impactful application of graph foundation models: pre-training on massive molecular datasets (ZINC, ChEMBL, PubChem) enables transfer to downstream property prediction, reaction prediction, and generative molecular design tasks. MolBERT (Fabian et al., 2020) pre-trains on SMILES string representations using masked token prediction; Uni-Mol (Zhou et al., 2023) pre-trains on 3D molecular structures with 209 million data points, achieving state-of-the-art on molecular property prediction benchmarks; and ChemBERTa uses domain-specific tokenisation for chemical language modelling. The dual representation challenge — molecules can be represented as strings (SMILES), graphs (atoms as nodes, bonds as edges), or 3D point clouds — requires models that can either unify representations or ensemble across modalities. The connection to our drug discovery guide is direct: molecular foundation models are the enabler of the AI-accelerated drug discovery pipeline described there — a Uni-Mol fine-tuned on assay data can predict binding affinity, ADMET properties, and synthetic accessibility for novel molecules, replacing the task-specific CGCNN/MEGNET models with a single pre-trained model that transfers across targets and assay types.
  • Graph pre-training strategies mirror NLP pre-training: masked node prediction (mask some node features, train the model to predict them from graph context — the graph version of BERT’s masked language modelling), contrastive graph pre-training (SimGRACE, GraphCL, generate two augmented views of a graph, maximise agreement between views — the graph version of SimCLR contrastive learning), and graph-level generation (predict the next graph in a sequence of temporal graphs — the graph version of GPT’s next-token prediction). The optimal augmentation strategy for contrastive graph pre-training is domain-dependent: for molecular graphs, valid augmentations include atom masking, bond deletion, and subgraph removal; for social networks, edge perturbation and node feature dropout; for knowledge graphs, relation path masking. The connection to our self-supervised learning guide is direct: graph contrastive pre-training methods (GraphCL, SimGRACE, GCC) are the graph domain application of the contrastive self-supervised learning methods described there — the SimCLR framework (two augmented views, NT-Xent loss maximising agreement between positive pairs) applies to graph representations with graph-specific augmentations substituting for image-specific augmentations (cropping, colour jitter).

The graph learning community has long envied NLP’s foundation model paradigm: a single GPT pre-trained on the internet transfers to hundreds of downstream tasks, while graph learning trains a fresh GNN for each (task, dataset) pair from scratch. Graph foundation models aim to close this gap — OFA, GraphGPT, and molecular foundation models are early evidence that the paradigm is feasible. But the graph domain presents unique challenges: no canonical token order, extreme domain heterogeneity, and the fundamental question of what a “universal” graph representation even means across social networks, molecules, knowledge graphs, and biological networks. This guide covers the full GFM landscape: OFA’s text-attributed graph approach, GraphGPT instruction tuning, Graph RAG for knowledge-grounded generation, molecular foundation models, graph pre-training strategies, and the theoretical connections to NLP foundation models.

OFA, GraphGPT and Graph-LLM Alignment

The core challenge for graph foundation models is the lack of a universal tokenisation: for text, every sentence is a sequence of tokens from the same vocabulary, enabling a single Transformer to process all text. For graphs, different domains have incompatible node feature spaces (molecular graphs have atom type, charge, chirality; citation graphs have word bag embeddings; social graphs have demographic features), different graph sizes (from 10-node molecules to billion-node social networks), and different structural properties (molecular graphs are sparse and planar; knowledge graphs are dense and heterophilic). OFA’s solution: use natural language as the universal node feature space. Every node in every graph can be described by a text attribute (a paper’s title and abstract in citation graphs, an atom’s IUPAC descriptor in molecular graphs, a person’s profile description in social graphs), and the LLM’s text encoder maps all these to the same embedding space. This converts the graph tokenisation problem into a text tokenisation problem — the LLM handles the cross-domain feature alignment, and the GNN handles the structural aggregation over the aligned node features. The elegance of OFA’s approach is that it leverages the LLM’s pre-trained knowledge of chemical, biological, and social concepts without task-specific pre-training on graph data — the GNN only needs to learn structural aggregation patterns that are largely domain-invariant. The connection to our graph Transformer guide is direct: OFA uses a GPS (General, Powerful, Scalable) graph Transformer as the GNN backbone described there, with LLM-generated node features as input. The GPS architecture’s combination of MPNN (message passing) and Transformer (global attention) is essential for OFA’s cross-dataset transfer: the MPNN captures local structural patterns while the Transformer captures global graph-level context, together providing the rich graph representations needed for transfer. GraphGPT’s two-stage training (graph-text alignment, then instruction fine-tuning) mirrors the LLaVA training paradigm for vision-language models: first align the graph encoder (GNN) with the LLM’s embedding space using contrastive learning, then fine-tune the aligned model on graph instruction-following datasets. The key design decision: freeze the LLM backbone during alignment (preserving the LLM’s pre-trained language knowledge) and train only the GNN and a projection layer, then unfreeze and instruction-tune on graph QA data. This two-stage approach prevents the graph-text alignment objective from degrading the LLM’s language capabilities — the same design principle as LLaVA’s two-stage vision-language alignment. The connection to our LLM alignment guide is direct: GraphGPT’s instruction fine-tuning stage uses the same SFT (supervised fine-tuning on instruction-response pairs) described there as the first stage of the RLHF pipeline — the graph question-answering datasets provide the instruction-following demonstrations that align the graph-LLM’s output format and reasoning style with what users expect from a conversational graph analysis tool.

ModelApproachBackboneTransferKey Task
OFAText-attributed graph + LLM featuresGPS Transformer + DeBERTaZero/few-shot cross-datasetNode cls, link pred, graph cls
GraphGPTGNN-LLM alignment + instruction tuningGNN + LLaMAFew-shot with instructionsGraph reasoning in NL
Graph RAGKG retrieval + LLM generationLLM + Neo4j KGZero-shot (retrieval-based)Multi-hop QA; document RAG
Uni-Mol3D structure pre-training (masked atoms)SE(3) TransformerFine-tune on assaysMolecular property; docking
MolBERTSMILES masked token pre-trainingBERTFine-tune on assaysMolecular property prediction

Graph Pre-Training Strategies, Molecular Foundation Models and Graph RAG

The optimal graph pre-training strategy depends on the downstream task distribution: masked node prediction (a discriminative pre-training task) produces features well-suited for node-level prediction tasks; graph-level generation (pre-training the model to generate valid graphs autoregressively) produces features well-suited for molecular generation and property prediction; and contrastive pre-training produces features well-suited for tasks requiring graph-level similarity judgements (graph retrieval, graph classification). DGI (Deep Graph Infomax, Velickovic et al., 2019) pioneered contrastive graph pre-training: maximise mutual information between local node representations and a global graph summary, without data augmentation. GraphCL (You et al., 2020) introduced data augmentation-based graph contrastive learning (node dropping, edge perturbation, attribute masking, subgraph sampling as augmentations), achieving strong transfer to molecular property prediction benchmarks. The connection to our self-supervised learning guide is direct: GraphCL is the graph-domain application of the SimCLR framework described there — the two augmented views are two augmented graphs, the positive pair is the two views of the same graph, the negative pairs are views of different graphs, and the NT-Xent contrastive loss is identical to SimCLR’s. The only graph-specific element is the choice of augmentation: image augmentations (crop, flip, colour jitter) would be semantically meaningless for graphs; graph augmentations (node dropping, edge perturbation) are chosen to preserve the graph’s semantic meaning while creating views that differ in local structure. Molecular foundation models face the multi-modal alignment challenge: molecules can be represented as SMILES strings (capturing 2D topology via text), molecular graphs (atoms as nodes, bonds as edges), or 3D conformers (atomic coordinates in 3D space). Each modality captures different information: SMILES is easy to generate and tokenise; molecular graphs enable MPNN-based structural reasoning; 3D conformers enable physics-inspired features (inter-atomic distances, angles, dihedral angles). MolBERT operates on SMILES; Uni-Mol operates on 3D conformers; and 3D-MoLM (Liu et al., 2023) aligns all three modalities using cross-modal contrastive learning. The connection to our geometric deep learning guide is direct: Uni-Mol’s SE(3)-Transformer backbone is the equivariant Transformer architecture described there — the architecture respects 3D rotation and translation symmetry by design, ensuring that physically equivalent molecular conformers (the same molecule in different orientations) produce identical representations, the equivariance property essential for physically meaningful molecular property prediction. Graph RAG’s community detection step (grouping knowledge graph entities into thematic communities using Leiden or Louvain clustering, then generating community summaries with an LLM) addresses a key limitation of entity-based retrieval: some questions are best answered at the community level (“what are the main themes in this document corpus?”) rather than the entity level (“who is person X?”). The hierarchical community structure of the knowledge graph enables hierarchical retrieval: coarse community summaries for broad questions, fine-grained entity subgraphs for specific questions. The connection to our knowledge graphs guide is direct: Graph RAG builds on the knowledge graph construction and querying infrastructure described there — the entity extraction (NER+RE), knowledge graph storage (Neo4j), and graph traversal (Cypher queries for k-hop entity retrieval) are the same components described there, with the RAG integration adding the LLM generation layer on top of the retrieved subgraph context.

✦ SUMMARIZE THIS ARTICLE WITH AI

Foundation models (the pre-train/fine-tune paradigm that GFMs extend to graphs) are in our Foundation Models guide. Graph Transformers (GPS architecture used as GFM backbone in OFA) are in our Graph Transformers guide. GNNs (message passing and graph classification foundations for GFMs) are in our GNNs guide. Self-supervised learning (GraphCL contrastive graph pre-training as graph-domain SimCLR) is in our Self-Supervised Learning guide. ML for drug discovery (molecular foundation models for property prediction and drug design) is in our Drug Discovery guide. Geometric deep learning (SE(3)-Transformer for equivariant 3D molecular representations) is in our Geometric DL guide. Advanced RAG (Graph RAG as multi-hop retrieval with knowledge graph structure) is in our Advanced RAG guide. Knowledge graphs (Neo4j infrastructure for Graph RAG entity extraction and storage) are in our Knowledge Graphs guide.

Leave feedback about this

  • Rating

Durgesh Kekare
Durgesh Kekarehttps://www.dataexpertise.in
Durgesh Kekare is a data science educator and founder of DataExpertise.in. With expertise in Python, machine learning, and analytics, he helps 10,000+ learners break into data careers.

Latest Posts

List of Categories