What Is RAG

Learning objectives

• Explain the core mental model behind What Is RAG
• Apply What Is RAG within RAG Mental Model
• Identify important boundaries, trade-offs, and failure modes
• Produce concrete evidence from the practice exercise

Related: RAG Why It Exists | RAG Core Concepts | RAG Architecture Overview | RAG Index

A Working Definition

Retrieval-Augmented Generation (RAG) is a technique that enhances LLM outputs by retrieving relevant documents from an external knowledge base and injecting them into the model's context at generation time.

A precise one-line definition:

RAG is a pattern that grounds LLM responses in specific, retrievable facts rather than parametric memory.

That definition has three operative words:

Grounds. The model's answer is tethered to source documents. If the document is wrong, the answer is wrong — but it is traceable.

Specific. RAG answers questions about your data: your wiki, your emails, your product docs. Not the internet at large.

Retrievable. The source documents are fetched on demand, not baked into the model weights. This means the knowledge base can be updated without retraining.

RAG vs Fine-Tuning

RAG and fine-tuning are complementary. RAG handles facts; fine-tuning handles style and format.

RAG vs Long Context

Modern models support 128k+ tokens. Why not just stuff the whole corpus into the prompt?

• Cost: Long prompts are expensive
• Noise: Irrelevant context degrades answer quality
• Latency: Longer prompts = slower generation
• Attribution: You still need to know which parts of the corpus matter

RAG is selective retrieval, not brute-force context stuffing.

The Basic Flow

Query → Embed → Retrieve (top-k docs) → Prompt (query + docs) → Generate → Answer

Engineering analogy: RAG is like an open-book exam. The model is the student; the retriever is the index at the back of the textbook.

What RAG Is Not

• Not a database. The retriever is a search layer; the vector store is the database.
• Not a guarantee of correctness. Garbage in, garbage out. If retrieved docs are wrong, the answer is wrong.
• Not magic. Every step is deterministic: chunking, embedding, similarity search, prompt construction.
• Not only for text. RAG works for images, audio, and structured data with multi-modal embeddings.

A One-Sentence Mental Model to Carry Forward

RAG is a two-stage pipeline: first, find the most relevant documents; second, ask the model to answer using only those documents.

Everything else — chunking, reranking, hybrid search, agentic retrieval — is a refinement of those two stages.

See RAG Architecture Overview for the deep dive.

Related

• RAG Why It Exists — the limits of parametric knowledge
• RAG Core Concepts — the vocabulary of retrieval and generation
• RAG Architecture Overview — the system block diagram
• RAG Index — full topic map

Practice lab

Implement the smallest measurable RAG experiment for What Is RAG. Keep the corpus and query set fixed, change one variable, and compare retrieval evidence and answer quality before and after.

Review questions

1. What problem does What Is RAG solve, and what assumptions does it rely on?
2. Which boundary or failure case is easiest to miss, and how would you expose it?
3. What alternative design would you consider, and what trade-off would change the decision?
4. What artifact, trace, test, or metric proves that your implementation is correct?

Completion evidence

• A working artifact, annotated trace, or reproducible experiment
• At least one normal case and one deliberately failing or boundary case
• A concise explanation of the design choice and its trade-offs
• Saved output showing how correctness was evaluated