The RAG Pipeline

TL;DR: RAG anchors a model in outside knowledge: rather than leaning on what the model memorized, you retrieve the most relevant chunks from a knowledge base at query time and drop them into the prompt, so answers come from real, current sources. It is the usual remedy for hallucination and stale knowledge, and refreshing it means updating the index, not retraining. The stages: ingest and chunk documents, embed and index them, retrieve (and typically rerank) the top chunks for a query, then generate an answer grounded in them with citations.

Why RAG exists

Because a model's knowledge is fixed at training time and sits fuzzily inside its weights, it cannot cite sources, drifts out of date, and hallucinates when a fact is missing. RAG separates knowledge from the model: hold that knowledge in an external store and retrieve what is relevant for each query. Answers then become grounded (traceable to sources), current (you update the store, not the model), and scoped (only the right chunks reach the prompt, which is cheaper and more accurate than a giant context, see context window).

The two phases

Indexing (offline):

1. Ingest documents and parse them (handling PDFs, tables, layout).
2. Chunk them into passages (see chunking), the granularity that balances retrieval precision and context.
3. Embed each chunk with an embedding model and store the vectors in a vector index.

Querying (online): 4. Embed the query and retrieve the nearest chunks (often combined with keyword search, see hybrid search). 5. Rerank the candidates with a more precise model and keep the top few (see reranking). 6. Generate the answer from those chunks, with citations so it is verifiable.

Quality is bounded by retrieval

A crucial fact: RAG answers are only as good as what you retrieve. If the relevant chunk is not retrieved, no amount of generation quality recovers it, so retrieval recall is the ceiling for the whole system (see retrieval evaluation). Most RAG failures are retrieval failures: bad chunking, a domain-mismatched embedding model, or missing reranking. Evaluate retrieval and generation separately so you know which to fix.

Why interviewers probe this

RAG is the most common production LLM architecture, so designing one is the modal applied-AI design round. A strong answer walks the pipeline (chunk, embed, index, retrieve, rerank, generate with citations), states that it grounds the model to fix hallucination and staleness, and stresses that retrieval recall is the ceiling so most failures are retrieval failures. That framing, plus evaluating retrieval and generation separately, signals you can build and debug a real RAG system.

Common misconceptions

• "RAG fine-tunes the model on documents." It retrieves documents at inference; the model is unchanged and the knowledge is external.
• "RAG eliminates hallucination." It greatly reduces it but the model can still misuse context; you need citations and faithfulness checks.
• "Better generation fixes a bad RAG." If retrieval misses the chunk, generation cannot recover; retrieval recall is the ceiling.
• "RAG and long context are the same." RAG retrieves the few relevant chunks; long context stuffs everything (often worse and pricier).

Key takeaways

• RAG grounds the model by retrieving relevant chunks at query time and generating from them with citations.
• It fixes hallucination and stale knowledge and updates by changing the index, no retraining.
• The pipeline is chunk, embed, index, retrieve, rerank, generate.
• Retrieval recall is the ceiling, so evaluate retrieval and generation separately; most failures are retrieval failures.