Tabular Deep Learning – TabNet, FT-Transformer and SAINT Explained

Tabular data — structured data in rows and columns, the primary data format in business, healthcare, finance, and most non-image/non-text domains — has long been the stronghold of gradient-boosted trees. XGBoost won dozens of Kaggle competitions. LightGBM powers production recommendation and fraud detection systems at scale. CatBoost handles categorical features without preprocessing. For most tabular problems, the default recommendation in 2026 remains: try LightGBM first. But the gap between tree models and deep learning on tabular data has been narrowing, and for specific problem types — large datasets, heterogeneous feature types, settings where pre-training on related data is possible — deep learning architectures specifically designed for tabular data now achieve competitive or superior performance. This guide covers the landscape of tabular deep learning: why standard MLPs underperform on tabular data, the three major architectures (TabNet, FT-Transformer, SAINT), the pre-training advantage that makes deep learning worth considering even when gradient boosting is a strong baseline, and the practical decision framework for choosing between them.

Why MLPs Underperform on Tabular Data — and What Tabular DL Fixes

Standard MLPs applied to tabular data fail for several interconnected reasons. First, tabular features have very different statistical properties: some are continuous, some are categorical, some are counts, some are sparse indicators. MLPs treat all features identically as continuous inputs — a one-hot encoded categorical column is not meaningfully different from a continuous feature to the network, and the appropriate representation of a categorical as an embedding is not learned by standard MLPs without careful design. Second, MLPs are permutation-invariant with respect to features only if explicitly designed to be, but the order of feature columns in a tabular dataset is arbitrary — no inductive bias helps the network discover which features to pay attention to. Third, tabular datasets are often small (thousands to tens of thousands of rows) relative to image datasets (millions of images), and deep learning’s advantage in representation learning only materialises with sufficient data. Tree-based methods like gradient boosting avoid these issues: they handle mixed feature types natively, discover feature interactions via tree splits, and are data-efficient because their inductive bias (axis-aligned splits) is well-matched to tabular data structure.

Tabular deep learning architectures address these failures directly. Embeddings for categorical features (learned lookup tables mapping category indices to dense vectors) are the key innovation for mixed-type data: a categorical with cardinality 1,000 gets a 32-dimensional embedding instead of a 1,000-dimensional one-hot vector, and the embedding captures semantic relationships between categories. Embeddings are used in all production tabular deep learning systems and are covered in our feature engineering guide. Attention mechanisms provide the feature selection inductive bias: by attending over feature tokens, the model learns which features are relevant for each prediction, analogous to how gradient boosting learns feature importance via information gain at each split. Self-supervised pre-training on unlabelled tabular data (masking features and predicting them, analogous to BERT — covered in our self-supervised learning guide) addresses the data efficiency problem by learning general feature representations before task-specific fine-tuning.

TabNet — Sequential Attention for Tabular Data

TabNet (Arik and Pfenning, 2021) processes tabular features through a series of sequential decision steps. At each step, a sparse soft-selection mechanism (the attentive transformer) selects which features to focus on using a sparsemax activation that produces attention weights summing to 1 with most weights near 0. The selected features are passed through a feature transformer (a shared + step-specific GLU network), producing a contribution to the final prediction. By aggregating contributions from multiple steps, TabNet builds a prediction from multiple feature subsets, each step able to focus on different features than the previous one — analogous to how a decision tree ensemble uses different features at different nodes.

TabNet’s attention masks are directly interpretable: the attention weight assigned to each feature at each step represents its contribution to the prediction, producing feature importance scores that can be plotted and explained to stakeholders. Unlike post-hoc interpretability methods like SHAP values, TabNet’s feature importance is intrinsic — it is computed by the forward pass, not by a separate attribution algorithm. This makes TabNet attractive for regulated domains (credit scoring, healthcare) where interpretability is a regulatory requirement, as discussed in our data science ethics guide. TabNet also supports self-supervised pre-training: a TabNet encoder can be pre-trained by masking random subsets of features and predicting them from the unmasked features, producing a pre-trained encoder that fine-tunes on the labelled task with far less labelled data — the same label efficiency benefit seen in vision and NLP self-supervised learning.

FT-Transformer and SAINT — Attention Across Features and Samples

FT-Transformer (Gorishniy et al., 2021) applies the standard Transformer encoder to tabular data by tokenising each feature: numerical features are multiplied by a learnable weight vector to produce a d-dimensional token; categorical features are mapped through an embedding table to a d-dimensional token. The resulting sequence of F feature tokens (F = number of features) is then processed by L layers of multi-head self-attention and feed-forward networks. The [CLS] token appended to the sequence aggregates information for the final prediction. This architecture allows every pair of features to interact via attention at every layer — capturing F×F pairwise interactions per layer, compared to gradient boosting’s explicit feature interaction discovery via tree splits. FT-Transformer achieves state-of-the-art or near-state-of-the-art performance on most large tabular benchmarks and is the recommended starting point for tabular deep learning in most new projects due to its simplicity and alignment with the well-understood Transformer architecture.

SAINT (Somepalli et al., 2021) extends FT-Transformer with two innovations. First, intersample attention: in addition to attending across features within each row (row-wise attention, as in FT-Transformer), SAINT attends across multiple rows for each feature (column-wise attention), allowing the model to compare the current example to other examples in the same batch and use those comparisons in its prediction. This is the tabular analogue of attending across different parts of an image in vision Transformers, and it captures patterns that are invisible to any row-wise model. Second, SAINT uses contrastive self-supervised pre-training: two augmented views of each row (created by randomly masking features and adding feature-level noise) are processed by the same SAINT encoder, and the contrastive loss pulls the representations of the two views of the same row together and pushes representations of different rows apart — the same SimCLR-style contrastive learning applied to tabular rows instead of images. The pre-trained SAINT encoder fine-tunes very effectively with small labelled datasets, making it competitive on low-data regimes where gradient boosting traditionally dominates. The regularisation perspective on why contrastive pre-training helps is informative: it is a form of unsupervised regularisation that constrains the learned representations to be robust to feature perturbations, analogous to Mixup or feature dropout as data augmentation strategies.

✦ SUMMARIZE THIS ARTICLE WITH AI

ChatGPT Perplexity Claude Grok

Gradient boosting — the primary competitor and baseline for tabular deep learning — is in our Gradient Boosting guide. Feature engineering and categorical embeddings that enable tabular deep learning are in our Feature Engineering guide. Self-supervised pre-training (contrastive learning, masked prediction) that gives tabular DL its label efficiency advantage is in our Self-Supervised Learning guide. The Transformer architecture underlying FT-Transformer and SAINT is in our Transformers guide. SHAP interpretability for comparing tabular DL models with gradient boosting is in our Model Interpretability guide. AutoML and hyperparameter search for tabular DL architectures are in our AutoML guide.