RAG Architecture Overview

Advanced RAG Engineering · 1/13

Module 1: Failure-Oriented Architecture

RAG Architecture Overview

Learning objectives

  • Explain the core mental model behind RAG Architecture Overview
  • Apply RAG Architecture Overview within Failure-Oriented Architecture
  • Identify important boundaries, trade-offs, and failure modes
  • Produce concrete evidence from the practice exercise

Related: RAG Core Concepts | RAG Retrieval Strategies | RAG Generation and Prompting | RAG Index

This note is the block diagram. Every other note in the vault is a zoom-in on one of the boxes here.

The Big Picture

┌──────────────────────────────────────────────────────────────────────┐
│                         INGESTION PIPELINE                           │
│                                                                      │
│   Raw Data → Load → Parse → Chunk → Embed → Index → Vector Store     │
│      │         │      │      │       │       │          │            │
│      ▼         ▼      ▼      ▼       ▼       ▼          ▼            │
│   ┌────────┐ ┌────┐ ┌────┐ ┌────┐ ┌────┐ ┌────┐  ┌──────────┐      │
│   │  PDF   │ │Web │ │JSON│ │Text│ │Open│ │FAISS│ │ Chroma   │      │
│   │  File  │ │Page│ │ API│ │File│ │AI  │ │     │ │ Pinecone │      │
│   └────────┘ └────┘ └────┘ └────┘ └────┘ └────┘  └──────────┘      │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌──────────────────────────────────────────────────────────────────────┐
│                         QUERY PIPELINE                               │
│                                                                      │
│   Query → Embed → Retrieve (top-k) → Rerank → Prompt → LLM → Answer  │
│      │       │          │              │         │       │            │
│      ▼       ▼          ▼              ▼         ▼       ▼            │
│   ┌────┐  ┌────┐   ┌─────────┐   ┌────────┐ ┌────┐  ┌────┐          │
│   │User│  │Open│   │ Vector  │   │Cross-  │ │RAG │  │GPT │          │
│   │    │  │AI  │   │ Store   │   │Encoder│ │Prompt│ │4   │          │
│   └────┘  └────┘   └─────────┘   └────────┘ └────┘  └────┘          │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘

Read left-to-right. The ingestion pipeline prepares the knowledge base. The query pipeline answers questions using that base.

The Two Pipelines

Ingestion Pipeline

Runs offline or on a schedule. Responsible for:

  • Loading raw documents
  • Parsing into text
  • Chunking into pieces
  • Embedding each chunk
  • Indexing into the vector store

Engineering analogy: ETL. Extract, transform, load — but for embeddings.

See RAG Document Loading and Parsing, RAG Chunking Strategies, RAG Embeddings and Models, RAG Vector Stores.

Query Pipeline

Runs online, per user request. Responsible for:

  • Embedding the query
  • Retrieving candidate chunks
  • Reranking for precision
  • Prompting the LLM with context
  • Generating the answer

Engineering analogy: a search engine results page (SERP) plus a summarization layer.

See RAG Retrieval Strategies, RAG Reranking, RAG Generation and Prompting.

Why This Decomposition

PipelineChanges when...Fails because...
IngestionNew documents arriveParser errors, bad chunks, wrong embedding model
QueryUser asks a questionWrong retrieval, irrelevant context, hallucination

Separating them lets you:

  • Re-index without touching query code
  • A/B test retrieval strategies independently
  • Scale ingestion and query separately

What RAG Doesn't Guarantee

  • Correctness. If the corpus has errors, RAG propagates them.
  • Completeness. If retrieval misses the relevant chunk, the model cannot invent the fact.
  • Freshness. Ingestion lag means recent documents may not be indexed yet.

Related

  • RAG Core Concepts — vocabulary used in the diagram
  • RAG Retrieval Strategies — the retrieval layer
  • RAG Generation and Prompting — the generation layer
  • RAG Evaluation Metrics — measuring pipeline quality
  • RAG Index — full topic map

Practice lab

Implement the smallest measurable RAG experiment for RAG Architecture Overview. Keep the corpus and query set fixed, change one variable, and compare retrieval evidence and answer quality before and after. Add an operational constraint such as concurrency, recovery, security, latency, or cost, and defend the resulting design trade-off.

Review questions

  1. What problem does RAG Architecture Overview solve, and what assumptions does it rely on?
  2. Which boundary or failure case is easiest to miss, and how would you expose it?
  3. What alternative design would you consider, and what trade-off would change the decision?
  4. What artifact, trace, test, or metric proves that your implementation is correct?

Completion evidence

  • A working artifact, annotated trace, or reproducible experiment
  • At least one normal case and one deliberately failing or boundary case
  • A concise explanation of the design choice and its trade-offs
  • Saved output showing how correctness was evaluated