LLM 103: Retrieval-Augmented Generation in Practice

In LLM 102, we introduced Retrieval-Augmented Generation (RAG) as a way to connect a language model to external knowledge. This post goes one layer deeper. We will look at how a RAG system actually works: embeddings, chunking, vector search, reranking, prompt assembly, citations, and evaluation.

The basic idea is simple: when the model needs knowledge, do not ask it to rely only on its parameters. Retrieve relevant evidence, place that evidence into the context window, and ask the model to answer from that evidence.

1. Why RAG Exists

LLMs store some knowledge in their parameters, but that knowledge can be incomplete, stale, private, or difficult to inspect. RAG adds a second memory: an external collection of documents that can be searched at inference time.

This distinction matters:

For research, RAG is useful when the model needs to answer from papers, reports, datasets, lab notes, interview transcripts, policy documents, or project-specific files. Instead of asking, “What do you know about this?”, we ask, “Given these retrieved sources, what answer is supported?”

2. The RAG Pipeline

A RAG system has two sides: indexing and answering.

Indexing happens before the user asks a question. Documents are loaded, split into chunks, embedded, and stored. Answering happens at query time. The user question is embedded, similar chunks are retrieved, the best evidence is selected, and the LLM writes an answer.

The design choices in the middle matter more than people expect. A weak chunking strategy can bury evidence. A weak retriever can miss it. A weak prompt can cause the model to ignore it. RAG is a system problem, not just a vector database problem.

Here is a minimal pseudocode sketch of the full loop — from question to grounded answer:

# Minimal RAG pipeline (pseudocode)
from embedding_model import embed
from vector_store import search
from llm import generate

def answer(question, corpus_index):
    # 1. Embed the question
    query_vec = embed(question)
    
    # 2. Retrieve top-k similar chunks
    chunks = search(corpus_index, query_vec, top_k=5)
    
    # 3. Assemble prompt with evidence
    context = "\n".join(f"[S{i+1}] {c.text}" for i, c in enumerate(chunks))
    prompt = f"""Use only the evidence below. Cite sources using [S1], [S2], etc.
    If evidence is insufficient, say what is missing.
    
    Evidence:\n{context}\n\nQuestion: {question}"""
    
    # 4. Generate grounded answer
    return generate(prompt)

3. Embeddings: Turning Text into Searchable Vectors

An embedding model converts text into a vector. Similar meanings should land near each other in vector space. This lets us search semantically, not just by exact keyword overlap.

For example, a question like:

might retrieve chunks containing terms like:

This is the strength of dense retrieval: it can connect related concepts even when the words differ. The weakness is that semantic similarity is not the same as factual usefulness. A passage can be topically similar but still not answer the question.

4. Chunking: The Quiet Part That Breaks Everything

Chunking is the process of splitting documents into pieces. It sounds mundane, but it is one of the most important RAG decisions.

Chunks that are too small may lose context. Chunks that are too large may bury the answer and waste the context window. Chunks that ignore document structure may separate a table from its explanation, a figure from its caption, or a claim from its citation.

There is no universal best chunk size. A legal document, a research paper, a spreadsheet, and a meeting transcript need different handling. Good chunking preserves meaning, source location, and metadata.

Useful metadata often includes:

• document title;
• author or source;
• date;
• section heading;
• page number;
• file path;
• table or figure identifier;
• access permissions.

5. Retrieval: Sparse, Dense, and Hybrid Search

RAG systems often combine several retrieval methods.

Sparse retrieval uses exact or near-exact term matching. BM25 is the classic example. It works well when the query uses distinctive terms, names, citations, acronyms, or technical phrases.

Dense retrieval uses embedding similarity. It works well when the query and document use different words for similar ideas.

Hybrid retrieval combines both. In practice, hybrid search is often a strong default because keyword matching and semantic matching fail in different ways.

After retrieval, many systems use approximate nearest-neighbor indexes to make vector search fast at scale. That is where algorithms and libraries such as HNSW and FAISS enter the picture.

6. Reranking: Retrieval Is Not the Final Answer

Initial retrieval is usually broad. It tries to avoid missing relevant evidence. Reranking is more careful. It takes the candidate passages and scores which ones are actually useful for the specific question.

The difference is subtle but important:

Reranking can be done with cross-encoders, late-interaction models, LLM judges, or task-specific scoring rules. The tradeoff is speed versus quality. A reranker is slower than first-stage retrieval, but it can sharply improve what enters the prompt.

7. Prompt Assembly and Citations

After retrieval and reranking, the system must assemble the prompt. This is where many RAG systems quietly succeed or fail.

A practical prompt should separate the user’s question from the retrieved evidence and make the model’s obligations clear:

Citations should point to the retrieved source, not just decorate the answer. A useful citation lets the reader inspect the document, page, section, or chunk that supports the claim.

Good RAG answers tend to:

• answer the question directly;
• cite each factual claim;
• distinguish evidence from inference;
• avoid using retrieved text that is irrelevant;
• say when the sources do not support an answer.

8. Evaluation: Measuring the Whole System

RAG evaluation is not one number. You need to evaluate retrieval and generation separately, then evaluate the complete workflow.

For retrieval, ask:

• Did the system retrieve a source that contains the answer?
• Was the source ranked high enough to fit into the context window?
• Did retrieval work for paraphrased queries?
• Did it handle acronyms, dates, tables, and domain terms?

For generation, ask:

• Is the answer faithful to the retrieved evidence?
• Are citations attached to the right claims?
• Does the model admit when evidence is missing?
• Is the answer useful for the intended audience?

A small hand-built evaluation set is often more valuable than a big vague one. For a research website or lab assistant, you might create 30-50 representative questions with known supporting sources and expected answer traits.

9. Common Failure Modes

RAG reduces some LLM risks, but it creates its own stack of failure modes.

The last one is easy to underestimate. RAG systems need access control at retrieval time, not only at the user interface. If a user cannot access a document, the retriever should not place that document into the prompt.

10. A Practical RAG Checklist

When building a RAG system, I would start with this checklist:

1. Define the corpus. What documents are allowed sources?
2. Preserve metadata. Can every chunk point back to a document, page, section, or table?
3. Chunk by meaning. Do not split blindly if document structure is available.
4. Use hybrid retrieval. Combine keyword and semantic search when domain terms matter.
5. Rerank before generation. Put the strongest evidence into the limited context window.
6. Require grounded answers. Tell the model to cite and to admit missing evidence.
7. Evaluate with real questions. Test retrieval recall, citation quality, and answer usefulness.
8. Monitor drift. Re-index when documents change and retest when models change.

In the next post, we can move from RAG to LLM agents: planning, tool use, memory, guardrails, and why agent workflows are powerful but easy to overbuild.