26 — Applications

RAG#

Retrieval-Augmented GenerationPractice 1 min read
◆ The PatternGrounding LLM answers in your own documents

RAG reduces hallucination by retrieving relevant documents before generation. The pipeline: Chunk documents → Embed chunks → Store in vector DB → At query time: Embed query → Retrieve top-K chunks → Generate answer with chunks as context.

Pipeline: Query → Embed → Search → Rerank → Augment Prompt → Generate
Each stage has design choices: chunk size, overlap, embedding model, retriever, reranker, prompt template.
Chunk size: 256–512 tokens with 10–20% overlap
Too small → lost context. Too large → diluted relevance. Semantic chunking (split at paragraph breaks) works better than fixed-size.

Common pitfalls: (1) Chunking destroys context — tables, lists split mid-content. (2) Embedding model mismatch — query embeddings must match document embeddings. (3) Top-K too small — misses relevant but lower-ranked chunks. (4) No reranking — embedding similarity ≠ answer relevance.

Reranking is often one of the most effective improvements to a RAG system: a cross-encoder re-scores the top 20–100 retrieved passages more accurately than the first-stage search.
Interactive — RAG pipeline flow

Python — RAG with LangChain#

Pattern bridge: Retrieval-Augmented Generation grounds the model in external knowledge. Cosine similarity retrieves relevant passages. In statistics, Bayesian updating brings prior evidence to new questions. Add tools and a planning loop and retrieval becomes one step of an agent.
Worked example

A fact that spans 100 tokens, in a document cut into 512-token chunks. With no overlap, 19.3% of such facts are split across two chunks, so no single retrieved chunk contains them. With 64 tokens of overlap, 7.8%; with 128 — more than the fact’s length — none. Chunking decides what can be retrieved before any embedding model is involved.

Where it misleads
  • Retrieval failures look like model failures: if the right passage is not retrieved, the answer cannot be grounded (Barnett et al. 2024 list seven such failure points).
  • Retrieved text placed in the middle of a long prompt is used less (Liu et al. 2024).
  • RAG reduces hallucination but does not remove it; the model can still ignore or misread the sources.
In code
rng = np.random.default_rng(4)
def split_rate(chunk, overlap, span=100, doc=100_000, trials=20_000):
    step = chunk - overlap
    starts = rng.integers(0, doc - span, trials)          # where the fact begins
    k = starts // step                                    # the last chunk to start before the fact
    ok = starts + span <= k * step + chunk                # does it reach the fact's end?
    return 1 - ok.mean()
rates = {(c, o): round(split_rate(c, o), 3) for c, o in [(512, 0), (512, 64), (512, 128)]}
Python, in your browser — edit the code and run it again

Assumes import numpy as np. Configurations named after real models use their published shapes; other numbers are made up or simulated, as the comments say.

The same idea with a library
from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.chains import RetrievalQA

# 1. Chunk documents
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(documents)

# 2. Embed & store
vectorstore = FAISS.from_documents(chunks, OpenAIEmbeddings())

# 3. Retrieve & generate
qa = RetrievalQA.from_chain_type(
    llm=ChatOpenAI(model="gpt-4o", temperature=0),
    retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
)
answer = qa.invoke("What is the refund policy?")
Sources
  • P. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, NeurIPS, 2020
  • S. Barnett et al., “Seven Failure Points When Engineering a Retrieval Augmented Generation System”, arXiv:2401.05856, 2024
  • N. F. Liu et al., “Lost in the Middle: How Language Models Use Long Contexts”, Transactions of the ACL 12, 2024

Last reviewed October 2026 · How this site checks its content · Report a mistake

Check yourself

1. A RAG system answers wrongly although the right document is in the corpus. What should you check first?

Show answer

C. Measure retrieval recall separately from generation. If the passage never reached the model, no prompt will fix it.

2. What goes wrong when chunks are too large?

Show answer

B. Chunk size trades context for precision; test a few sizes on real questions.

3. Does RAG stop hallucination?

Show answer

A. Evaluate faithfulness (is every claim supported by the retrieved passages?) as well as answer accuracy.

Linked from 1
← Previous
Prompt Engineering
Open in the full reader, with the topic sidebar