Multimodal AI – CLIP, LLaVA, GPT-4V and Vision-Language Models Explained

The boundary between computer vision and natural language processing has dissolved. Models that once required separate specialised architectures — an image classifier for cats, a captioning model, an OCR system for text in images — are now unified under a single paradigm: the vision-language model, which understands and generates both images and text within a shared semantic space. This unification is practically transformative: a single GPT-4V or Gemini call can read a chart, summarise a slide deck, extract a table from a PDF screenshot, or describe the contents of a medical image with no task-specific training. Understanding multimodal models — how they are architectured, trained, and applied — is increasingly essential for data scientists working with real-world document corpora and mixed-modality datasets. This guide covers CLIP and contrastive vision-language pre-training, zero-shot transfer, Large Vision-Language Models (LLaVA, GPT-4V, Gemini), multimodal RAG, and the practical applications for data science workflows.

CLIP — Contrastive Language-Image Pre-training

CLIP (Radford et al., OpenAI 2021) is trained on 400 million image-text pairs scraped from the web — a scale of supervision available only because the pairing is naturally present in web data (images with captions, alt text, or surrounding text). The training objective is contrastive: for a batch of N image-text pairs, the model maximises the cosine similarity between the N matching pairs while minimising the similarity between all N² − N non-matching pairs. The image encoder (a ResNet or ViT) maps images to d-dimensional embeddings; the text encoder (a masked self-attention Transformer) maps text descriptions to the same embedding space. After training, the two encoders produce representations where semantic similarity is reflected by geometric proximity in the shared space — “a photograph of a dog” and an actual photograph of a dog will have high cosine similarity, while “a drawing of a cat” and a photograph of a car will have low similarity.

Zero-shot classification exploits this alignment: to classify an image into K classes, construct K text prompts (“A photo of a {class_name}”), encode both the image and all K prompts, compute cosine similarities, and return the class with the highest similarity. On ImageNet, this zero-shot approach achieves 76.2% top-1 accuracy — matching supervised ResNet-50 without any ImageNet training. The key insight is that class descriptions serve as supervision at inference time, replacing the fixed classification head that standard supervised models require. This generalisation extends to novel classes, fine-grained distinctions, and domain-specific concepts that were not present in CLIP’s training distribution, as long as they can be described in natural language. The connection to our self-supervised learning guide is direct: CLIP’s contrastive pre-training is the multi-modal extension of the SimCLR/MoCo contrastive framework, replacing image-image pairs with image-text pairs. The embedding space that CLIP produces — shared between vision and language — is also the natural space for multi-modal vector database retrieval, where images and text queries can be compared by cosine similarity.

Large Vision-Language Models — LLaVA, InstructBLIP and GPT-4V

Large Vision-Language Models (LVLMs) extend pre-trained LLMs to understand visual inputs by connecting a frozen vision encoder to a frozen (or lightly fine-tuned) LLM via a trainable projection layer. LLaVA (Large Language and Vision Assistant, Liu et al., 2023) is the landmark open-source LVLM: it uses CLIP’s ViT-L/14 encoder to produce 256 image patch embeddings (each 1024-dimensional), projects them to the LLM’s token dimension via a linear layer, and prepends them to the text token sequence before passing everything to a Vicuna or Llama LLM. The LLM then generates a response conditioned on both the image patch tokens and the text instructions. This simple architecture — frozen vision encoder + linear projection + pre-trained LLM — achieves remarkable visual understanding because the CLIP vision encoder provides rich semantic image representations and the LLM provides powerful language generation, and the linear projection learns to bridge the two representation spaces.

LLaVA-1.5 and LLaVA-1.6 replaced the linear projection with a 2-layer MLP and used a higher-resolution vision encoder, dramatically improving performance on document understanding tasks (reading text in images, understanding charts and diagrams) where fine-grained visual detail matters. InstructBLIP (Dai et al., 2023) uses a Q-Former (Query Transformer) instead of a projection layer: a small Transformer that distils the most relevant information from the image patch embeddings into a fixed number of query tokens via cross-attention, reducing the visual token count from 256 to 32 and enabling more efficient LLM processing. The connection to the Transformer cross-attention mechanism is direct: the Q-Former’s cross-attention between learned query tokens and image patch tokens is exactly the attention mechanism covered in our Transformers guide, applied to bridge vision and language.

For data scientists, the most immediately practical LVLM capabilities are: (1) Chart and plot reading — LVLMs can extract numerical values, axis labels, and trends from matplotlib, seaborn, or business intelligence charts directly from screenshots, enabling automated insight extraction from visualisation-heavy reports; (2) Table extraction from PDFs — GPT-4V reliably extracts structured tables from PDF screenshots and returns JSON, replacing fragile heuristic PDF parsers; (3) Document classification — classifying document types (invoice, contract, medical form) from scanned images without OCR; (4) Data quality review — reviewing data visualisation outputs for anomalies or formatting issues as part of automated data pipeline quality checks.

Multimodal RAG and Embedding Unified Corpora

Standard text-only RAG fails on document corpora that contain embedded visualisations, charts, scanned pages, and mixed text-image content — which describes the majority of real-world enterprise document collections. Multimodal RAG extends the RAG architecture to handle heterogeneous content by embedding both text and visual content in a shared semantic space (using CLIP or a multimodal encoder) and retrieving the most relevant content regardless of modality. The pipeline: (1) extract text chunks and image regions from documents; (2) encode both with a multimodal encoder (CLIP, SigLIP) into a shared vector database; (3) at query time, encode the user’s text query with the same encoder and retrieve the top-k most similar text chunks and image regions; (4) pass the retrieved text and images together to a multimodal generator (GPT-4V, LLaVA) as context; (5) generate the answer conditioned on both text and visual context.

Late-fusion MM-RAG — where text and images are retrieved separately and combined at the generator stage — is the simplest architecture and works well when text and visual content are independently informative. Early-fusion MM-RAG — where a natively multimodal model encodes interleaved text-image sequences from the start — achieves better performance on questions that require jointly reasoning about text and images (e.g. “what does the caption say about the trend shown in this chart?”) but requires natively multimodal models that are more expensive to serve. The connection to our prompt engineering guide is direct: the prompt structure for MM-RAG — how to present retrieved images alongside retrieved text in the generator’s context window — follows the same few-shot and instruction structuring principles covered there, adapted to include image tokens.

✦ SUMMARIZE THIS ARTICLE WITH AI

ChatGPT Perplexity Claude Grok

The contrastive pre-training framework shared between CLIP and SimCLR/BYOL is in our Self-Supervised Learning guide. The Transformer architecture and cross-attention mechanism underlying LVLMs is in our Transformers guide. Vector databases for multimodal embedding storage and retrieval are in our Vector Databases guide. RAG and prompt engineering for multimodal generators are in our Prompt Engineering guide. LLM fine-tuning (LoRA/QLoRA) applied to LVLMs for domain-specific visual understanding is in our LLM Fine-Tuning guide. Computer vision fundamentals — CNNs, ViT, detection — are in our Computer Vision guide.