📋 KEY INSIGHTS
- Retrieval-Augmented Generation (RAG) addresses the two fundamental limitations of standalone LLMs: knowledge cutoff (LLMs cannot access information after their training cutoff) and hallucination on knowledge-intensive questions (LLMs generate plausible-sounding but factually incorrect responses when the answer requires specific facts not well-represented in their weights). The basic RAG pipeline — embed the query, retrieve the top-k most similar documents from a vector database, prepend them to the LLM context, generate — is straightforward to implement but leaves significant performance on the table. Advanced RAG techniques address the three main failure modes of naive RAG: retrieval failures (the relevant chunk is not in the top-k results), context assembly failures (the retrieved chunks are relevant but lack the context needed to answer the question), and generation failures (the LLM ignores or misinterprets the retrieved context). Understanding which failure mode dominates in a given application is the first step toward choosing the right advanced technique.
- Chunking strategy is the most consequential but least discussed RAG design decision: how text is split into retrievable units determines the granularity of retrieval and the information density of each context window position. Fixed-size chunking (split every 512 tokens, overlap 50 tokens) is the default but suboptimal: chunks split in the middle of sentences or ideas break semantic coherence, reducing embedding quality. Semantic chunking (split at sentence or paragraph boundaries, using an embedding model to merge adjacent chunks whose cosine similarity exceeds a threshold) produces chunks that contain complete semantic units. Hierarchical chunking (parent-child: index fine-grained child chunks for retrieval precision, but return the parent chunk as context for completeness) achieves both retrieval precision (the child chunk embedding closely matches the query) and context completeness (the parent chunk provides surrounding context). Small-to-big retrieval (retrieve small chunks, expand to surrounding paragraphs before passing to LLM) is the LangChain/LlamaIndex implementation of this parent-child pattern.
- Hybrid retrieval combines dense (embedding) search with sparse (BM25/TF-IDF keyword) search, addressing their complementary failure modes: dense retrieval handles semantic similarity (finding documents that mean the same thing with different words) but misses exact keyword matches for rare terms, proper nouns, and product codes; sparse retrieval excels at exact keyword matching but misses paraphrases and synonyms. Reciprocal Rank Fusion (RRF) is the standard score fusion method: for each document, compute its rank in the dense results list (rank_dense) and sparse results list (rank_sparse), and assign combined score = 1/(k + rank_dense) + 1/(k + rank_sparse) where k=60 is a smoothing constant. RRF consistently outperforms linear score fusion because it is robust to score scale differences between dense and sparse retrievers. The connection to our vector databases guide is direct: Qdrant, Weaviate, and Elasticsearch all support hybrid search natively by combining their HNSW dense index with an integrated BM25 sparse index, returning RRF-fused results in a single API call.
- HyDE (Hypothetical Document Embeddings, Gao et al., 2022) addresses the query-document embedding mismatch: short, keyword-style queries embed differently from the long, detailed documents they are trying to retrieve. HyDE generates a hypothetical answer to the query using the LLM (without grounding — the LLM generates a plausible but possibly hallucinated answer), embeds the hypothetical answer, and uses that embedding for retrieval. Since the hypothetical answer is in the same register as real documents (long, detailed, fluent prose), its embedding matches document embeddings better than the short query embedding does. HyDE consistently improves retrieval recall on knowledge-intensive QA benchmarks, at the cost of one additional LLM call per query.
- RAPTOR (Recursive Abstractive Processing for Tree-Organised Retrieval, Sarthi et al., 2024) addresses the limitation of chunk-level retrieval for questions requiring synthesis across multiple documents: standard RAG retrieves individual chunks and cannot surface high-level themes that only emerge from reading many chunks together. RAPTOR builds a tree of abstractions: cluster the original chunks (using UMAP + GMM soft clustering), generate LLM summaries of each cluster, embed the summaries, cluster the summaries, generate higher-level summaries, and repeat until a single root summary is produced. At query time, retrieve from all levels of the tree simultaneously — a single query may retrieve leaf-level chunks (for specific details) and high-level cluster summaries (for thematic context). RAPTOR achieves state-of-the-art on multi-document QA benchmarks (QuALITY, NarrativeQA) where the answer requires synthesising information from many parts of a long document.
- RAGAS (Retrieval Augmented Generation Assessment) is the standard framework for evaluating RAG pipelines, decomposing RAG quality into four independent metrics: (1) Faithfulness — does the generated answer contain only claims supported by the retrieved context (measuring hallucination rate)? (2) Answer Relevance — does the generated answer address the question asked? (3) Context Recall — does the retrieved context contain all information needed to answer the question? (4) Context Precision — what fraction of the retrieved context is actually relevant to the question? These four metrics diagnose different failure modes: low faithfulness indicates generation failures (LLM ignoring context); low context recall indicates retrieval failures (relevant documents not retrieved); low context precision indicates retrieval noise (irrelevant documents drowning out relevant ones). The connection to our LLM evaluation guide is direct: RAGAS uses an LLM-as-judge approach (using the LLM itself to evaluate faithfulness and relevance by checking claims against retrieved context) — the same approach as LLM-as-judge evaluation described there, applied to the specific RAG evaluation task.
The basic RAG pipeline is now commodity: embed query, search vector database, prepend top-k chunks, generate. Every major LLM framework (LangChain, LlamaIndex, Haystack) implements it in ten lines of code. The gap between a demo RAG system and a production RAG system that reliably answers hard questions is closed by advanced techniques: sophisticated chunking, hybrid retrieval, query rewriting, contextual retrieval, multi-hop reasoning, and rigorous evaluation. This guide covers the full advanced RAG landscape: chunking strategies (fixed, semantic, hierarchical, late chunking), hybrid BM25+dense retrieval with RRF fusion, HyDE query expansion, RAPTOR hierarchical summarisation, Contextual Retrieval with prefix injection, multi-hop RAG for complex reasoning, and RAGAS evaluation for systematic pipeline improvement.
Chunking, Hybrid Retrieval and Query Transformation
The chunking granularity tradeoff is fundamental: small chunks (128-256 tokens) produce high-quality embeddings (the embedding captures a focused semantic unit) and high retrieval precision (the retrieved chunk is tightly relevant to the query), but low context completeness (the chunk lacks surrounding context needed to interpret the retrieved fact). Large chunks (1024-2048 tokens) provide rich context but dilute the embedding signal (many different topics in one chunk) and retrieve more irrelevant content. Hierarchical chunking resolves this tradeoff by separating the retrieval unit from the context unit: index small child chunks (256 tokens, high embedding quality) for retrieval, but when a child chunk is retrieved, return the full parent chunk (1024 tokens, containing surrounding context) to the LLM. This parent-child pattern is implemented in LlamaIndex as ParentDocumentRetriever and in LangChain as ParentDocumentRetriever. Late chunking (Jina AI, 2024) takes a different approach: encode the full document with a long-context embedding model (preserving cross-chunk context via attention), then mean-pool over token embeddings within each chunk boundary to produce chunk embeddings. Late chunking embeddings are contextualised (each chunk embedding is informed by the full document context) rather than independent, achieving higher retrieval quality on long document benchmarks. Contextual Retrieval (Anthropic, 2024) uses an LLM to prepend a brief context description to each chunk before embedding: “This chunk is from the Q3 2024 earnings report of Acme Corp and discusses revenue breakdown by region.” This contextualisation dramatically reduces retrieval failures caused by chunks that are ambiguous without their document-level context (e.g., a chunk that says “revenue increased by 12%” is ambiguous without knowing which company and time period). Anthropic reported a 49% reduction in retrieval failures on internal benchmarks with contextual retrieval. Query rewriting addresses the mismatch between conversational query phrasing and document language: a user asking “how does it work?” in a multi-turn conversation needs the query rewritten to include the entity being discussed (“how does AlphaFold2 protein structure prediction work?”). Multi-query retrieval (generate k query variants, retrieve for each, deduplicate and merge results) increases recall for queries that can be expressed in multiple ways. Step-back prompting (generate a more abstract version of the query: “what is the general principle behind X?” before retrieving) retrieves higher-level conceptual context alongside specific factual content, improving performance on questions requiring both. The connection to our prompt engineering guide is direct: query rewriting, step-back prompting, and contextual chunk summarisation are all LLM prompting techniques — the prompt engineering patterns described there for chain-of-thought and instruction following are directly applied to the query transformation and context summarisation steps of advanced RAG pipelines.
| Technique | Addresses | Cost | Implementation | Recall Gain |
|---|---|---|---|---|
| Semantic chunking | Semantic coherence | Low | LangChain SemanticChunker | +5-10% |
| Hierarchical (parent-child) | Precision + completeness | Low | LlamaIndex ParentDocRetriever | +10-15% |
| Contextual Retrieval | Ambiguous chunks | Medium (LLM calls) | Anthropic / custom | +49% failure reduction |
| Hybrid BM25+dense (RRF) | Keyword + semantic gaps | Low | Qdrant / Elasticsearch | +8-20% |
| HyDE | Query-doc mismatch | Medium (1 LLM call) | LangChain HyDE | +10-20% on BEIR |
| RAPTOR tree | Multi-document synthesis | High (build-time LLM) | LlamaIndex RAPTOR | +SOTA on QuALITY |
Multi-Hop RAG, Agentic RAG and RAGAS Evaluation
Multi-hop RAG handles questions that require chaining multiple retrieval steps: “Who founded the company that acquired DeepMind?” requires first retrieving which company acquired DeepMind (Google), then retrieving who founded Google (Larry Page and Sergey Brin). Single-step RAG fails on such queries because no single chunk contains the full answer chain. IRCoT (Interleaved Retrieval with Chain-of-Thought, Trivedi et al., 2022) alternates retrieval and reasoning: at each CoT step, retrieve relevant documents conditioned on the partial reasoning chain so far, append them to the context, generate the next reasoning step, and repeat until the final answer is reached. This interleaving allows each retrieval step to be conditioned on the conclusions of previous steps, progressively narrowing in on the answer. The connection to our LLM reasoning guide is direct: IRCoT is an application of chain-of-thought reasoning to the retrieval-augmented setting — the interleaving of retrieval and reasoning steps is the RAG analog of the multi-step CoT inference described there. Agentic RAG (the RAG-as-tool pattern) gives the LLM agency over the retrieval process: instead of a fixed single-pass retrieval, the LLM can decide when to retrieve, what queries to issue, which results to follow up on, and when it has enough context to answer. Tools like retrieval, calculation, search, and code execution are provided to the LLM agent, which orchestrates them in a ReAct-style loop. The connection to our AI agents guide is direct: agentic RAG is a specific instance of the ReAct agent framework described there, where retrieval is one of the available tools and the agent reasons about when and how to use it. RAGAS evaluation enables systematic, data-driven RAG pipeline improvement: instrument your RAG pipeline to log (query, retrieved_contexts, generated_answer, ground_truth_answer) for a representative sample of questions, run RAGAS metrics, and use the per-metric breakdowns to diagnose which component to improve. Low context recall → improve retrieval (better chunking, hybrid search, query rewriting). Low faithfulness → improve generation (stronger system prompt to stay grounded, smaller LLM context to reduce distraction). Low context precision → improve retrieval filtering (re-ranking with a cross-encoder, MMR for diversity). The connection to our MLOps CI/CD guide is direct: RAGAS metrics should be tracked in your ML monitoring dashboard and regression-tested in CI/CD — a code change that reduces context recall from 0.85 to 0.75 should fail the CI pipeline, the same way a model accuracy regression fails a CI test in the standard MLOps workflow described there.
✦ SUMMARIZE THIS ARTICLE WITH AI
Vector databases (HNSW index, hybrid search, metadata filtering) are in our Vector Databases guide. LLM evaluation (RAGAS as an LLM-as-judge framework) is in our LLM Evaluation guide. Prompt engineering (query rewriting, step-back prompting, contextual chunk summarisation) is in our Prompt Engineering guide. AI agents (agentic RAG, ReAct retrieval loop) are in our AI Agents guide. LLM reasoning (IRCoT multi-hop chaining retrieval with CoT) is in our LLM Reasoning guide. Generative AI (RAG as the standard LLM grounding pattern) is in our Generative AI guide. MLOps CI/CD (RAGAS regression testing in deployment pipelines) is in our MLOps CI/CD guide.



