How Do You Design a RAG System?

A RAG system has five parts, and a system design interview expects you to name all five. Ingestion pulls in documents. Chunking splits them into pieces. Embedding turns each piece into a vector and stores it in an index. Retrieval finds the pieces that match a user question. Generation passes those pieces to a language model, which writes the final answer.

RAG stands for retrieval augmented generation. It exists because a language model only knows what it saw during training. RAG lets the model answer using your private or current documents, without retraining it.

Most candidates describe the working case and stop. The score comes from the four decisions below.

Quick Overview

Ingestion and Chunking

Documents arrive from a store like S3, a wiki, or a database. Run a batch job for the first load, then a change feed for updates. Say how you handle deletes, because a deleted document that stays in the index leaks private data.

Chunking splits each document into pieces small enough to retrieve. Pick a size in tokens, such as 200 to 500, and add a small overlap so a sentence is not cut in half. Chunk on structure, like headings or paragraphs, rather than on a fixed character count.

Embedding and the Vector Index

An embedding is a list of numbers that represents the meaning of a chunk. Similar text produces vectors that sit close together.

Store these vectors in a vector database, which is an index built for nearest neighbor search. Name the index type and its trade-off. HNSW is fast to query but uses more memory. IVF uses less memory but needs tuning.

Say what happens when the embedding model changes. The answer is a full re-index, and it is expensive, so version the index.

Retrieval and Re-ranking

Pure vector search misses exact terms like product codes and error numbers. Hybrid search fixes this. It runs keyword search and vector search together, then merges the two result lists.

Then re-rank. Retrieve the top 50 chunks cheaply, then score those 50 with a slower and more accurate model, and keep the best 5. This two-stage shape is the same retrieval and ranking split used in recommendation systems.

Generation and Grounding

Send the top chunks to the language model with the user question. Keep the context small, because a larger context costs more and can lower accuracy.

Grounding means the answer must come from the retrieved text. Ask the model to cite the chunk it used, and to say it does not know when nothing matches. Both behaviors can be tested automatically, so say that you would test them.

Latency, Cost, and Evaluation

Give a latency budget and split it. Retrieval usually takes 50 to 100 milliseconds. Generation takes most of the rest.

Cache aggressively. Cache embeddings for repeated queries, and cache full answers for common questions. Generation is the expensive part of the bill, so every cache hit is a direct saving.

Evaluation is the part most candidates forget. Measure retrieval quality and answer quality separately. Recall at 5 tells you whether the right chunk was found. A graded answer set tells you whether the model used it.

How to Prepare

• Build the five-part outline into a habit. Ingestion, chunking, indexing, retrieval, generation, in that order, every time.
• Learn the retrieval layer properly. Start with what a vector database is and what RAG is and why it matters.
• Practice full AI designs. Grokking the AI System Design Interview covers RAG, search, and ranking systems end to end.
• Know the model layer. Embeddings, context limits, and inference cost all come up. Grokking Modern AI Fundamentals covers them.
• Have an eval answer ready. Interviewers ask how you would know the system got worse. Recall at 5 plus a graded answer set is a strong reply.