Prompt Engineering for Data Scientists – In-Context Learning, Chain-of-Thought and RAG

Large language models have become infrastructure in 2026 in the same way that databases became infrastructure in the 1990s: they are the substrate on which vast categories of data science applications are built. Text classification, information extraction, code generation, document summarisation, question answering over internal knowledge bases, data cleaning and enrichment — all of these are now routinely implemented by prompting a pre-trained LLM rather than training a task-specific model. The quality, reliability, and cost of these applications depend critically on prompt design. A poorly written prompt produces hallucinated, inconsistent, or off-format outputs that require expensive post-processing or manual review. A well-engineered prompt produces reliable, formatted, auditable outputs that flow directly into downstream pipelines. Prompt engineering is therefore not a soft skill or a “nice to have” — it is a core engineering discipline for building LLM-powered data science applications. This guide covers the major prompting techniques (zero-shot, few-shot, chain-of-thought, tree-of-thought), RAG for grounding, structured output generation, prompt optimisation, and security considerations for production deployment.

Zero-Shot, Few-Shot and In-Context Learning

Zero-shot prompting instructs the model to perform a task with no examples, relying entirely on the knowledge and capabilities encoded in the model’s pre-training. Zero-shot works well for tasks well-represented in the training data (summarisation, translation, general classification) and fails for tasks with unusual formats, domain-specific conventions, or fine-grained distinctions the model was not trained on. The quality of zero-shot prompting is determined primarily by how precisely the task is specified: “classify this text” produces worse results than “classify this product review as positive, negative, or neutral. Return only one of these three words and nothing else.”

Few-shot prompting (Brown et al., 2020) includes examples of the desired input-output mapping in the prompt, enabling the model to learn the task format and distribution from context. With 3–8 well-chosen examples, few-shot prompting achieves performance competitive with fine-tuning on many tasks, with zero additional training. Example selection matters: examples should be diverse (covering different input patterns), unambiguous (clear ground-truth output), and representative of the actual input distribution. For NLP classification tasks — sentiment analysis, intent detection, topic labelling — few-shot prompting with 5–8 labelled examples per class typically achieves 85–95% of fully fine-tuned performance on standard benchmarks. The connection to active learning is direct: actively selecting the most informative few-shot examples — rather than randomly sampling them — further improves in-context learning performance.

In-context learning (ICL) is the general phenomenon of LLMs performing new tasks from examples in the context window, without gradient updates. ICL exploits the model’s learned ability to identify and generalise patterns from demonstrations — a form of implicit Bayesian inference over the space of tasks consistent with the observed examples. The theoretical understanding of ICL is still developing: leading hypotheses include gradient descent in the residual stream (Akyürek et al., 2022) and implicit Bayesian meta-learning, covered in our Bayesian statistics guide for those interested in the theoretical grounding.

Chain-of-Thought and Structured Reasoning

Chain-of-thought (CoT) prompting (Wei et al., 2022) dramatically improves LLM performance on tasks that require multi-step reasoning — arithmetic, logical deduction, code debugging, complex data analysis — by instructing the model to produce intermediate reasoning steps before the final answer. The canonical CoT prompt adds “Let’s think step by step” or provides examples of step-by-step reasoning in the few-shot demonstrations. Why does externalising intermediate reasoning improve accuracy? Because the model’s output tokens at each step are conditioned on the preceding tokens: making the reasoning visible forces the model to commit to and build on intermediate conclusions rather than jumping directly to a final answer that is inconsistent with the actual calculation. Visible reasoning also makes errors easier to identify and debug — a benefit that extends to communicating LLM-assisted analysis to stakeholders who need to audit the reasoning chain.

Self-consistency (Wang et al., 2022) extends CoT by generating multiple independent CoT reasoning chains (typically 10–40) and returning the most common final answer via majority vote. Because each chain samples different intermediate steps, different chains make different errors — majority voting over many chains is more reliable than any single chain. Self-consistency improves accuracy on arithmetic and logical reasoning by 5–15 percentage points over single CoT at the cost of proportionally more inference compute. The connection to ensemble methods is exact: self-consistency is an ensemble of LLM reasoning chains, analogous to how random forests aggregate many decision trees. The Monte Carlo sampling perspective is also relevant: sampling many reasoning chains and taking the modal answer is equivalent to Monte Carlo estimation of the most likely answer under the model’s distribution.

RAG — Retrieval-Augmented Generation for Knowledge-Grounded Applications

The two fundamental limitations of pure LLM generation are knowledge cutoff (the model knows nothing about events or documents created after training) and hallucination (the model generates plausible-sounding but factually incorrect information, confident and fluent). Retrieval-Augmented Generation (RAG) addresses both by retrieving relevant documents from an external knowledge base at inference time and including them in the prompt context, grounding the model’s generation in verified, current information.

The standard RAG pipeline: (1) encode the user’s query using an embedding model (e.g. Sentence-BERT — covered in our word embeddings guide); (2) retrieve the top-k most similar document chunks from a vector database (Qdrant, Pinecone, pgvector) using approximate nearest-neighbour search; (3) construct a prompt that includes the retrieved chunks as context: “Answer the question based on the following documents: [chunks]. Question: [query]”; (4) generate the answer using the LLM conditioned on the retrieved context. The quality of the final answer depends critically on retrieval quality — if the relevant chunks are not retrieved, the generator cannot produce a correct answer regardless of its capability. Evaluating retrieval recall@k separately from generation quality is the correct evaluation methodology, as covered in our generative AI guide. The knowledge graph-augmented RAG approach — combining vector retrieval with structured graph traversal — further improves retrieval quality for relational, multi-hop queries.

Advanced RAG architectures address the limitations of naive RAG: HyDE (Hypothetical Document Embeddings) generates a hypothetical document answering the query and embeds that for retrieval, rather than embedding the raw query — improving retrieval for questions whose surface form differs from the document vocabulary. Re-ranking applies a cross-encoder model (computationally expensive but highly accurate) to re-score the top-k retrieved chunks, improving precision before passing to the generator. Self-RAG (Asai et al., 2023) trains the LLM to decide when to retrieve (not all queries need retrieval) and to evaluate the relevance of retrieved passages, reducing unnecessary retrieval costs and hallucination from irrelevant retrieved content. These approaches connect directly to the NLP pipeline techniques for document processing and the evaluation frameworks for retrieval system quality.

✦ SUMMARIZE THIS ARTICLE WITH AI

ChatGPT Perplexity Claude Grok

The generative AI and RAG pipeline that prompt engineering drives is in our Generative AI guide. Word embeddings and vector search underlying RAG retrieval are in our Word Embeddings guide. Knowledge graphs for structured RAG are in our Knowledge Graphs guide. RLHF — the fine-tuning approach that aligns LLMs to follow instructions well — is in our RLHF guide. Model compression for efficient LLM inference — quantisation and distillation — is in our Model Compression guide. NLP applications that prompt-engineered LLMs power — classification, NER, summarisation — are in our NLP Applications guide.