📋 KEY INSIGHTS
- Graph Attention Networks (GAT, Veličković et al., 2018) extend Graph Convolutional Networks (GCN) by replacing fixed symmetric normalisation with learned attention weights: at each node, GAT computes attention coefficients e_{ij} = a(Wh_i || Wh_j) for each neighbour j (where W is a shared linear transform and a is a learnable attention function), normalises them with softmax across all neighbours, and computes the updated node embedding as an attention-weighted sum of transformed neighbour embeddings. This lets GAT assign different importance to different neighbours rather than treating all edges equally.
- GraphSAGE (Hamilton et al., 2017) addresses the scalability limitation of transductive GNNs (models that require the full graph at training time) by introducing inductive graph learning: instead of training separate embeddings for each node, GraphSAGE learns an aggregation function that computes embeddings from a node’s sampled neighbourhood, enabling generalisation to nodes and graphs not seen during training. This inductive capability is essential for production recommendation systems and fraud detection where new nodes (users, items, transactions) are added continuously.
- The Graph Isomorphism Network (GIN, Xu et al., 2019) establishes the theoretical upper bound on GNN expressive power: it proves that any GNN whose aggregation function is at least as powerful as the Weisfeiler-Lehman (WL) graph isomorphism test achieves the maximum discriminative power achievable by GNNs over local neighbourhoods. GIN uses a simple sum aggregation with an MLP (rather than mean or max used by GCN/GraphSAGE), provably matching WL discriminative power, and is the recommended starting point when distinguishing graph structures is the primary objective.
- Hierarchical graph pooling — learning to coarsen a graph by clustering nodes and building a multi-scale graph representation — is the graph equivalent of spatial pooling in CNNs. DiffPool (Ying et al., 2018) learns a soft cluster assignment matrix S via a GNN at each pooling layer, producing a coarsened graph where each “super-node” represents a learned cluster of original nodes. MinCutPool enforces orthogonality and partition balance via regularisation losses. Hierarchical pooling is essential for graph classification (predicting a property of the whole graph, not individual nodes) on molecules, protein structures, and social networks.
- Heterogeneous graphs — graphs with multiple node types and edge types — are the natural representation for relational data in recommendation systems (user nodes, item nodes, purchase edges, click edges), knowledge graphs (entity nodes of different types, relation edges), and bioinformatics (gene nodes, protein nodes, metabolite nodes, interaction edges). Heterogeneous GNNs (HAN, HetGNN, R-GCN, HGT) handle heterogeneity by learning separate transformation matrices for each node type and edge type, then aggregating across relation types via attention or mean pooling.
- The most important practical advance in scalable GNN training is mini-batch training with neighbourhood sampling: instead of computing embeddings for all nodes simultaneously (which requires the full adjacency matrix in GPU memory), GraphSAGE and its successors sample a fixed-size neighbourhood at each layer (e.g., sample 25 neighbours at layer 1, 10 at layer 2), enabling mini-batch training on graphs with billions of nodes and edges. FastGCN, ClusterGCN, and GraphSAINT are alternative sampling strategies that trade sampling bias for better scalability.
Graph Neural Networks have matured from research curiosity to production infrastructure: Pinterest’s recommendation system (PinSage), Google’s drug discovery platform, Alibaba’s fraud detection, and LinkedIn’s economic graph all rely on GNN architectures at industrial scale. Our earlier Graph Neural Networks guide covered the foundational GCN and message-passing framework. This guide goes deeper into the architectures and design patterns that make production GNNs accurate, scalable, and interpretable: Graph Attention Networks (GAT) and their variants (GATv2), GraphSAGE for inductive learning, Graph Isomorphism Networks for theoretical completeness, hierarchical graph pooling for graph-level prediction, heterogeneous GNNs for multi-relational data, and the scalable mini-batch training strategies that enable GNNs on graphs with billions of nodes.
Graph Attention Networks — Learning Neighbour Importance
GCN’s aggregation scheme — mean of symmetrically normalised neighbour embeddings — weights neighbours by node degree: high-degree neighbours contribute less than low-degree neighbours (because their contribution is divided by √(d_i × d_j)). This normalisation is motivated by spectral graph theory but is not learned from data and does not adapt to the task. GAT replaces this fixed weighting with learned attention: the attention coefficient α_{ij} between nodes i and j is computed as α_{ij} = softmax_j(LeakyReLU(a^T [Wh_i || Wh_j])), where W ∈ ℝ^{d’×d} is a shared linear transform, a ∈ ℝ^{2d’} is a learnable attention vector, and || denotes concatenation. The softmax normalises attention over all neighbours, ensuring Σ_j α_{ij} = 1. Multi-head attention — running K independent attention mechanisms and concatenating (or averaging) their outputs — is the standard GAT configuration and stabilises training by reducing variance in attention estimates. The connection to our Transformers guide is direct: GAT’s multi-head attention is exactly the same mechanism as Transformer self-attention, adapted to the irregular neighbourhood structure of graphs rather than the fixed sequence structure of text.
GATv2 (Brody et al., 2022) identifies a theoretical limitation of GAT: the original attention function e_{ij} = a^T [Wh_i || Wh_j] is a linear function of (h_i + Wh_j) after factoring, making it unable to distinguish some graph structures that require attention to depend non-linearly on both source and target node embeddings. GATv2 fixes this by applying the attention MLP after (rather than before) concatenation: e_{ij} = a^T LeakyReLU(W[h_i || h_j]), where W is applied to the concatenated pair. This seemingly minor change makes GATv2 strictly more expressive than GAT and consistently improves performance on node classification benchmarks (OGB-Arxiv, OGB-Proteins). For production node classification tasks, GATv2 is the recommended default over GCN or GAT.
| Architecture | Aggregation | Inductive? | Heterogeneous? | Best For |
|---|---|---|---|---|
| GCN | Mean (spectral normalisation) | No (transductive) | No | Baseline node classification |
| GAT / GATv2 | Learned attention-weighted sum | Partial | No | Node classification, link prediction |
| GraphSAGE | Sample + mean/max/LSTM | Yes (inductive) | No | Large graphs; new nodes at inference |
| GIN | Sum + MLP (WL-equivalent) | Yes | No | Graph classification; structural discrimination |
| DiffPool | Hierarchical soft clustering | Yes | No | Graph classification (molecules, proteins) |
| HGT / R-GCN | Type-specific transforms + attention | Yes | Yes | Knowledge graphs; multi-relational data |
GraphSAGE, GIN and Scalable Mini-Batch Training
GraphSAGE (Hamilton et al., 2017) introduces inductive GNN learning by replacing node-specific embedding tables with a generalisation function: the embedding of node v at layer k is computed by sampling a fixed-size neighbourhood N_S(v) ⊆ N(v) (sampling without replacement from all neighbours), aggregating the previous-layer embeddings of sampled neighbours (via mean, max pooling, or LSTM), and combining with the current embedding via a linear transform and activation. This neighbourhood sampling makes GraphSAGE directly applicable to mini-batch training: sample a target set of nodes, sample their k-hop neighbourhoods, compute embeddings bottom-up through the k layers, and update parameters using only the subgraph induced by the sampled nodes. The memory footprint per mini-batch grows as b × s^k (batch size × samples per layer × layers), which is tractable even for graphs with millions of nodes. PinSage (Ying et al., 2018) — Pinterest’s production recommendation system — applies GraphSAGE to a graph of 2 billion pins and boards, demonstrating that GraphSAGE-style inductive GNNs scale to industrial graph sizes. The connection to our recommendation systems guide is direct: PinSage is a GNN-based collaborative filtering system, and GNN-based recommenders have largely superseded matrix factorisation methods because they can incorporate rich node features and multi-hop graph structure.
GIN (Xu et al., 2019) proves that sum aggregation with a sufficient MLP is as powerful as the Weisfeiler-Lehman graph isomorphism test — the strongest discriminator for local graph structure achievable by any GNN. The GIN update rule: h_v^{(k)} = MLP^{(k)}((1 + ε^{(k)}) × h_v^{(k-1)} + Σ_{u ∈ N(v)} h_u^{(k-1)}), where ε is a learnable scalar. The key is sum aggregation (not mean or max): sum preserves the multiset of neighbour features (two nodes with different-sized neighbourhoods are distinguished), while mean and max do not. For graph classification (predicting whether a molecule is toxic, whether a social graph contains a specific structural motif), GIN with sum readout (Σ_v h_v^{(K)}) is the theoretically principled and practically effective baseline. The connection to our ensemble methods guide is instructive: GNN expressiveness theory predicts which structural features a single GNN can represent; to go beyond WL, higher-order GNNs (k-WL) or GNN ensembles are needed, analogous to how ensemble methods escape single-model limitations in tabular ML.
Heterogeneous GNNs and Production Deployment
Heterogeneous graphs arise naturally in recommendation, fraud detection, and knowledge representation: an e-commerce graph has User, Item, Category, Brand, and Seller node types, and Purchase, Click, View, IsA, BelongsTo edge types. Standard GNNs treat all nodes and edges as identical, ignoring type semantics. Relational GCN (R-GCN, Schlichtkrull et al., 2018) learns a separate transformation matrix W_r for each edge type r: h_v^{(k)} = σ(Σ_r Σ_{u ∈ N_r(v)} (1/|N_r(v)|) W_r^{(k)} h_u^{(k-1)} + W_0^{(k)} h_v^{(k-1)}). For large numbers of edge types, basis decomposition (W_r = Σ_b a_{rb} V_b) reduces parameter count. HGT (Heterogeneous Graph Transformer, Hu et al., 2020) applies Transformer-style attention with type-specific query, key, and value matrices, enabling attention weights that depend on both source node type, edge type, and target node type — the most expressive heterogeneous GNN and the current state of the art on knowledge graph benchmarks. For production deployment, both PyG (PyTorch Geometric) and DGL (Deep Graph Library) provide heterogeneous graph data structures and pre-built HGT/R-GCN implementations. The connection to our knowledge graphs guide is direct: heterogeneous GNNs are the primary ML method for link prediction on knowledge graphs (predicting missing entity-relation-entity triples in Neo4j-style graph databases).
✦ SUMMARIZE THIS ARTICLE WITH AI
ChatGPTGNN foundations (GCN, message passing, node/graph/edge prediction tasks) are in our Graph Neural Networks guide. Transformer self-attention that GAT extends to graphs is in our Transformers guide. GNN-based recommendation systems (PinSage, GNN collaborative filtering) are in our Recommendation Systems guide. Knowledge graphs and graph databases (Neo4j) that heterogeneous GNNs reason over are in our Knowledge Graphs guide. Spatial GNNs for geographic graph data connect to our Spatial Data Science guide. Molecular GNNs for drug discovery connect to our Healthcare Data Science guide.
