LangChain & RAG Systems – Build LLM Apps with Python 2026

Large Language Models are powerful, but they hallucinate and have a knowledge cutoff. RAG (Retrieval-Augmented Generation) fixes both problems by letting the model retrieve relevant facts from your own documents before answering. LangChain is the most popular Python framework for building RAG pipelines and LLM-powered applications. This guide builds a working PDF QnA system from scratch.

What Is RAG?

RAG combines a retrieval system with an LLM generator. When a user asks a question, the system first retrieves relevant text chunks from a document database, then passes those chunks plus the question to the LLM as context. The LLM answers based on the retrieved facts rather than relying solely on training memory. This means answers are grounded, up-to-date, and attributable to sources — solving the hallucination problem for domain-specific knowledge.

Installing LangChain

pip install langchain langchain-community langchain-openai
pip install chromadb pypdf sentence-transformers faiss-cpu

Loading and Splitting Documents

from langchain_community.document_loaders import PyPDFLoader, DirectoryLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

# Load a single PDF
loader = PyPDFLoader("annual_report.pdf")
pages  = loader.load()

# Or load all PDFs in a directory
loader = DirectoryLoader("./docs/", glob="**/*.pdf", loader_cls=PyPDFLoader)
docs   = loader.load()

# Split into chunks
splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,        # characters per chunk
    chunk_overlap=200,      # overlap to preserve context across chunks
    separators=["

", "
", " ", ""]
)
chunks = splitter.split_documents(docs)
print(f"Split into {len(chunks)} chunks")

Creating a Vector Store

from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import HuggingFaceEmbeddings

# Use a free local embedding model (no API key needed)
embeddings = HuggingFaceEmbeddings(
    model_name="sentence-transformers/all-MiniLM-L6-v2")

# Create and persist the vector store
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"
)

# Reload existing store
vectorstore = Chroma(
    persist_directory="./chroma_db",
    embedding_function=embeddings)

# Semantic search
results = vectorstore.similarity_search("What was the revenue in Q3?", k=4)
for doc in results:
    print(doc.page_content[:200])

Building the RAG Chain

from langchain_openai import ChatOpenAI
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

prompt_template = '''Use the following context to answer the question.
If you don't know the answer from the context, say "I don't have enough information."

Context:
{context}

Question: {question}

Answer:'''

PROMPT = PromptTemplate(
    template=prompt_template,
    input_variables=["context", "question"])

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 4}),
    chain_type_kwargs={"prompt": PROMPT},
    return_source_documents=True)

result = qa_chain.invoke({"query": "What was the total revenue for FY2025?"})
print("Answer:", result["result"])
print("Sources:", [d.metadata.get("source") for d in result["source_documents"]])

Using Open-Source LLMs (No API Key)

from langchain_community.llms import Ollama

# Run a local LLM with Ollama (download from ollama.ai)
# ollama pull llama3.1
llm = Ollama(model="llama3.1", temperature=0)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 3}))

result = qa_chain.invoke({"query": "Summarise the key risks mentioned."})
print(result["result"])

Conversational RAG with Memory

from langchain.chains import ConversationalRetrievalChain
from langchain.memory import ConversationBufferWindowMemory

memory = ConversationBufferWindowMemory(
    memory_key="chat_history",
    return_messages=True,
    k=5  # remember last 5 exchanges
)

conv_chain = ConversationalRetrievalChain.from_llm(
    llm=llm,
    retriever=vectorstore.as_retriever(search_kwargs={"k": 4}),
    memory=memory,
    verbose=False)

# Multi-turn conversation
q1 = conv_chain.invoke({"question": "What are the main products?"})
q2 = conv_chain.invoke({"question": "Which of those had the highest growth?"})
print(q2["answer"])

Deploying as a FastAPI App

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI(title="Document QnA API")

class Question(BaseModel):
    query: str

@app.post("/ask")
def ask(q: Question):
    result = qa_chain.invoke({"query": q.query})
    return {
        "answer":  result["result"],
        "sources": [d.metadata.get("page", "?") for d in result["source_documents"]]
    }

Conclusion

RAG with LangChain is the most practical way to add LLM capabilities to domain-specific knowledge bases in 2026. The pattern — load documents, split, embed, store in a vector DB, retrieve on query, generate with LLM — is standardized and production-ready. Start with Chroma and a free HuggingFace embedding model, then swap in OpenAI or a local Ollama model for the LLM layer. The entire pipeline is under 50 lines of code.