Top 10 RAG Interview Questions Answered

RAG or Retrieval Augmented Generation is
one of the most important topics in AI
interviews today. Every company building
AI products uses RAG. Here are the top 10
questions you will face and exactly how
to answer them.

1. What is RAG and why is it used?

RAG combines information retrieval with
LLM generation. Instead of relying on
training data alone, RAG fetches relevant
documents at query time and provides them
as context to the LLM.

Why it matters:

• Reduces hallucination significantly

• Keeps information current without retraining

• Allows LLMs to reference specific documents

• Much cheaper than fine-tuning for knowledge

2. Explain the RAG pipeline end to end.

A complete RAG pipeline has 6 steps:

1. Document ingestion - load PDFs, docs, web pages

2. Chunking - split into 256-512 token pieces

3. Embedding - convert chunks to vectors

4. Storage - save vectors in vector database

5. Retrieval - embed query, find similar chunks

6. Generation - pass chunks as context to LLM

3. What chunking strategy do you recommend?

This depends on the content type.

For general documents use recursive
character splitting with 512 tokens and
50 token overlap. This is the LangChain
default and works well for most cases.

For long technical documents use
semantic chunking which splits on meaning
boundaries rather than character count.

For Q&A datasets use parent-child chunking
where small chunks are retrieved but the
parent chunk is passed to the LLM for
more context.

4. What is the difference between dense

and sparse retrieval?

Dense retrieval uses semantic vector
similarity. It understands meaning so
similar meaning queries find relevant
results even with different words.

Sparse retrieval uses BM25 keyword
matching. It is exact match based and
works better for specific terms, names
and technical jargon.

Hybrid search combines both and gives
the best results in production. Use
alpha of 0.7 dense and 0.3 sparse as
a starting point.

5. How do you improve RAG retrieval quality?

Five techniques that actually work in
production:

1. Query expansion - generate multiple
   versions of the query before retrieval

2. HyDE - generate a hypothetical answer
   and embed that instead of the raw query

3. Reranking - use Cohere Rerank or
   FlashRank to reorder retrieved chunks

4. Hybrid search - combine dense and
   sparse retrieval

5. Parent-child chunks - retrieve small
   chunks but pass larger context to LLM

6. What embedding model do you recommend?

For production with budget use
text-embedding-3-small from OpenAI.
It is fast, cheap and very good quality.

For open source use BAAI/bge-large-en-v1.5.
It is the best open source English
embedding model.

For multilingual use Cohere
embed-multilingual-v3.

Always benchmark on your specific data
before choosing. General benchmarks do
not always match your use case.

7. What is reranking and when should you use it?

Reranking is a second pass scoring of
retrieved documents using a more powerful
cross-encoder model. Vector search returns
semantically similar documents but not
always the most relevant ones.

A cross-encoder like Cohere Rerank reads
both the query and each document together
and gives a relevance score. This is much
more accurate than vector similarity alone.

Always use reranking when answer quality
matters more than speed. It adds 100-200ms
latency but significantly improves answers.

8. How do you evaluate a RAG system?

Use the RAGAS framework with these metrics:

Faithfulness - is the answer grounded in
the retrieved context? Score 0 to 1.

Answer Relevance - does the answer
actually address the question? Score 0 to 1.

Context Recall - did retrieval find all
the relevant information? Score 0 to 1.

Context Precision - are the retrieved
chunks actually relevant? Score 0 to 1.

A production RAG system should target
above 0.8 on all four metrics.

9. What is the lost in the middle problem?

LLMs pay more attention to content at
the start and end of their context window.
Information in the middle gets ignored.

This matters for RAG because if you
retrieve 10 chunks the most relevant
ones should be first and last not buried
in the middle.

Solution: after reranking place the
highest scored chunk first, second highest
last and fill the middle with lower
scored chunks.

10. When would you use fine-tuning

instead of RAG?

Use RAG when:

• Knowledge changes frequently

• You need to cite sources

• You have a large document collection

• Budget is limited

Use fine-tuning when:

• You need specific output format or style

• Domain vocabulary is very specialized

• Latency requirements are very strict

• Knowledge is stable and does not change

In most production cases start with RAG.
Add fine-tuning only if RAG cannot achieve
the required quality.

Conclusion

RAG is essential knowledge for any AI
engineering role. Master these 10 concepts
and you will be prepared for most RAG
interview questions.

Practice building a RAG system from
scratch using LangChain and Qdrant.
Hands-on experience is what separates
good candidates from great ones.

Download our free RAG Complete Guide
cheat sheet at amanailab.com/resourcesfor a quick reference during interview prep.