Go to Source
When a RAG system gets it wrong, the model is usually innocent. It answered faithfully from the wrong context. The bug is upstream, in a retrieval layer nobody bothered to measure.
The bot cited a real document and was still wrong
A support assistant I was helping debug confidently answered a billing question, and it cited a real internal doc by name. The answer was wrong. The citation was real. That combination is what makes you crazy.
Everyone in the room said the same word: hallucination. The fix queue filled up fast — tighten the prompt, add “only answer from the provided context,” drop the temperature, maybe swap to a bigger model. All reasonable-sounding. All aimed at the generator.
Then I did the boring thing. I logged the actual chunks that got retrieved for that query and read them. The passage that contained the correct answer was not in the context window. It wasn’t rank 3 or rank 8. It wasn’t retrieved at all. The doc the model cited was retrieved — a different, adjacent section of it — and the model did exactly what you’d want: it summarized the context it was handed. The context was just wrong.
The model didn’t lie. Retrieval lied. The model faithfully repeated the lie.
A RAG generator answers the question “given this context, what’s the response?” If the context is wrong, a perfectly honest model gives you a perfectly wrong answer with a real citation stapled to it.
You’re debugging the half of the system you can see
Here’s why this misdiagnosis is the default, every time. The generation step is loud and legible. You can read the prompt, read the output, eyeball the diff between them. Retrieval is a vector similarity search that returns chunk IDs. Nobody reads chunk IDs.
So when something breaks, all your attention flows to the part you can see. You rewrite the prompt. The answer changes. You declare progress. But you changed the phrasing of an answer that was built on the wrong context — you never changed the context. The next query with the same retrieval gap fails the same way, and you’re back to blaming the model.
I’ve watched teams burn weeks of prompt iteration on what was, underneath, a chunking bug. Months on what was an embedding-model mismatch. The generation layer became a worry stone they kept rubbing because it was the only surface they could feel.
The uncomfortable rule: you cannot fix what you don’t measure, and almost nobody measures retrieval.
The five ways retrieval lies, none fixed by a prompt
These are well-known RAG failure modes, and in my experience they show up in roughly this order of frequency.
1. Naive fixed-size chunking. You split documents every 512 tokens with no respect for structure. So a number lands in one chunk and the heading that says what it means lands in another. A table gets severed from its column headers. The fact survives; its context doesn’t. Retrieval returns the chunk, the model sees “$4,200” with no idea it’s a refund cap, and you get a confident wrong number.
2. Embedding mismatch. You’re using a generic, off-the-shelf embedding model on a domain full of jargon, internal acronyms, and product names it never saw in training. To that model, your domain’s most-relevant document and a vaguely-worded distractor look about equally far from the query. The right doc doesn’t rank because the embedding space doesn’t understand your vocabulary.
3. No reranker. Vector search top-k is noisy by nature. The right passage is often in your top 20 but sitting at rank 18, below five plausible-looking distractors. You only feed the top 5 to the model. The answer was retrievable and you threw it away.
4. Retrieving too much. The over-correction. “Recall is low, so let’s stuff top-20 into the context.” Now the right passage is buried among nineteen distractors, and distractors measurably lower answer accuracy — the relevant fact gets lost in the middle of a long context where models attend to it least. More context is not more signal. It’s more noise with the signal hidden in it.
5. No retrieval eval. The one that causes the other four to persist. You have an eval for the final answer, maybe an LLM-as-judge score. You have nothing that asks the prior question: did retrieval even surface the document that contains the answer? Without that, every retrieval regression is invisible until a user hits it.
Measure retrieval as its own component, or stay blind
The fix isn’t a clever trick. It’s treating retrieval like the engineered subsystem it is and putting a number on it. Two numbers, mostly.
Recall@k — of the queries in my labeled set, for what fraction did the gold document show up in the top k retrieved? This is the one that catches “the answer was never in the context.” If recall@5 is 0.6, then 40% of your answers are being generated from context that doesn’t contain the answer, and no prompt on earth fixes that.
MRR (mean reciprocal rank) — when the right doc is retrieved, how high does it rank? This is what tells you whether you need a reranker. High recall but low MRR means the answer is in your top-20 but buried, and a cross-encoder reranker is your highest-leverage fix.
You build a labeled set of (query, gold_doc_id) pairs — a few hundred is enough to start — and you run it as a gate. Here’s the harness, small enough to drop into CI:
def eval_retrieval(labeled_set, retrieve, k=5, min_recall=0.85):
hits, rr = 0, 0.0for query, gold_id in labeled_set:
ranked = retrieve(query, top_k=k) # -> [doc_id, ...]if gold_id in ranked:
hits += 1
rr += 1.0 / (ranked.index(gold_id) + 1) # reciprocal rank
recall_at_k = hits / len(labeled_set)
mrr = rr / len(labeled_set)
print(f"recall@{k}={recall_at_k:.3f} MRR={mrr:.3f}")
assert recall_at_k >= min_recall, (
f"Retrieval regressed: recall@{k}={recall_at_k:.3f} < {min_recall}"
)
return recall_at_k, mrr
The assert is the entire point. It fails the build when retrieval gets worse, before the answer-level eval ever runs.
The regression the answer eval slept through
Let me make this concrete, because this is the moment the argument earns its keep.
In one pipeline I worked on, someone changed the chunking config — bumped chunk size and dropped the overlap to save on embedding cost. Sensible-looking change. The end-to-end answer eval barely moved: a point or two on an LLM-judge score, well inside the noise we’d learned to ignore. It shipped.
The retrieval harness above told a different story. Recall@5 fell from 0.91 to 0.74 on the exact same labeled set. The new chunk boundaries were splitting facts from their headers, so for a chunk of queries the gold passage simply stopped being retrievable. The answer eval didn’t catch it because for most queries the model still produced a fluent, plausible answer — just an increasingly wrong one for the affected slice, and “fluent and wrong” is precisely what an answer-level judge is worst at flagging.
The retrieval gate caught it on the commit. Recall is a far sharper instrument here than answer quality, because it measures the thing that actually broke instead of a downstream symptom smeared across the generator’s fluency.
That ordering is the whole discipline: gate on retrieval first, look at generation second. If recall@k is in the floor, you have a retrieval bug, and every minute spent on the prompt is a minute spent debugging the wrong half of the system. I’ve spent enough of my LLM-orchestration work watching pipelines fail at the seams between components to trust the seam I can measure over the one I can only feel.
Takeaways
- “It hallucinated” is a diagnosis you have to earn. Before you touch the prompt, log the retrieved chunks and check whether the answer was even in the context. Usually it wasn’t.
- The bug is almost always upstream. Chunking, embeddings, ranking, and context size are all retrieval problems, and a better prompt fixes exactly none of them.
- Recall@k tells you if the answer was reachable; MRR tells you if you need a reranker. Two cheap numbers that point straight at the actual fix.
- Gate the build on retrieval, not just the answer. An answer-level eval will sleep through a retrieval regression that recall@k catches on the commit.
- You can’t fix what you don’t measure — and almost nobody measures retrieval. That’s not a footnote. It’s why the generator keeps taking the blame.
Next time your RAG bot says something confidently wrong, don’t open the prompt file. Open the retrieval log. The model is probably telling the truth about a context that was lying to it.
