RAG Is Not a Search Problem. It's a Retrieval Contract.

For: A product or engineering lead at a 50–200-person SaaS company who has been asked to add an 'AI chat over our data' feature and is three days into reading about RAG — understands the acronym, has seen the vector-database pitch, but cannot figure out why their prototype gives confidently wrong answers half the time

If your RAG prototype confidently returns wrong answers, the LLM is almost certainly not the problem. Your retrieval layer is handing it the wrong chunks with high similarity scores, and the model is doing exactly what you asked: summarizing the garbage you gave it. RAG is not a search problem you solve by dropping in a vector database. It is a contract between retrieval and generation, and the contract is usually never explicitly defined.

This is the piece I wish someone had emailed me on day three of the last RAG project. Skip the tutorials that treat the pipeline as one blob. Read this instead.

The problem RAG actually solves

LLMs know a lot of general things and none of your specific things. They do not know your product docs, your customer support tickets, your internal policies, or what your CFO said in the last board deck. Fine-tuning teaches the model new patterns but is a poor fit for facts that change weekly. You do not want your invoicing policy baked into model weights.

Retrieval-augmented generation solves this by keeping the model general and injecting the relevant facts at query time. At runtime: user asks a question, you fetch the most relevant snippets from your data, you stuff them into the prompt, the model answers grounded in those snippets. That is it. The elegance of the idea is why every tutorial makes it look like a weekend project.

The analogy: RAG is an open-book exam with a bad librarian

Think of the LLM as a smart intern taking an open-book exam. The intern is fluent, reads fast, and will write a confident answer to anything. The retrieval layer is the librarian who hands the intern the relevant pages before each question.

If the librarian hands over the right three pages, the intern nails it. If the librarian hands over three pages that look relevant — same keywords, similar topic — but are actually from last year's deprecated policy, the intern will still write a confident, well-structured answer. It will just be wrong. The intern has no way to know the pages are wrong. Neither does your prompt.

That is what "confidently hallucinated" looks like in production. The model is not lying. The librarian is.

How RAG actually works: four contracts, not one pipeline

Every RAG system is four handoffs. Each one is a contract that can silently break.

1. Chunking contract: document → chunks

You take your source docs and split them into pieces small enough to embed and retrieve. Naive chunking (every 500 tokens, hard cut) will slice a sentence in half, separate a table from its header, or split a policy from the exception clause underneath. The chunk that gets retrieved is now missing the context that makes it correct.

2. Embedding contract: chunks → vectors

You run each chunk through an embedding model (OpenAI's text-embedding-3, Cohere, a local BGE model). The model produces a vector that approximates semantic meaning. Approximates. Two chunks about "refund policy" — one for enterprise, one for free tier — will land close together in vector space. Cosine similarity does not know which one applies to the user asking.

3. Retrieval contract: query → top-K chunks

User query gets embedded, you fetch the top K nearest chunks (usually K=3 to 10). This is where most failures happen. The top-K result is a ranking, not a relevance guarantee. A chunk with 0.89 cosine similarity to the query might be topically adjacent and factually irrelevant. There is no threshold below which the system says "I do not know." It always returns something.

4. Generation contract: chunks + query → answer

You pass the chunks and the query to the LLM with a prompt like "Answer using only the context below." The model complies. If the context is wrong, the answer is wrong. If two chunks contradict, the model picks one, usually the first.

A minimal worked example

Imagine a SaaS help center. A user asks: "Can I export my data after canceling?"

Your docs contain two chunks:

• Chunk A (from 2022 policy, still indexed): "After cancellation, data is retained for 90 days and can be exported via the admin panel."
• Chunk B (from 2024 policy, the current one): "Upon cancellation, data is purged within 7 days. Export must be initiated before cancellation."

Both chunks are semantically almost identical to the query. Cosine similarity might rank Chunk A at 0.91 and Chunk B at 0.88. The retriever hands the LLM Chunk A. The LLM answers: "Yes, you have 90 days to export your data after canceling."

The model did nothing wrong. The prompt did nothing wrong. The retrieval contract had no mechanism for recency, no metadata filter for "active policy," and no confidence floor. That is the failure. And no amount of prompt engineering fixes it.

The gotchas, ranked by how much time they will cost you

1. You have no evaluation set. If you cannot measure retrieval precision on a held-out set of 50–200 real questions with known correct chunks, you are debugging blind. Build this before you tune anything else.
2. Chunking is not "solved." Fixed-size chunking is a starting point, not a strategy. For structured docs, chunk by section. For chat logs, chunk by conversation. For code, chunk by function. Preserve headings and metadata in the chunk itself so the model gets context.
3. Cosine similarity is not relevance. Add a reranker (Cohere Rerank, BGE reranker, or a cross-encoder) as a second stage. Retrieve top 50, rerank to top 5. This one change fixes more hallucinations than any prompt tweak.
4. No metadata filters. If your data has tenant IDs, dates, product versions, or access levels, filter before vector search, not after. A vector DB that cannot do pre-filtering (or does it badly) will leak wrong-tenant data at high similarity scores.
5. No "I do not know" path. Set a similarity threshold below which the system refuses to answer. Users tolerate "I could not find that" far better than a confident wrong answer.
6. Stale index. Docs change; embeddings do not update themselves. Build a re-indexing pipeline on day one, not month six.
7. Query and document asymmetry. User queries are short and often ambiguous. Documents are long and structured. Consider query rewriting (HyDE, multi-query) so the retrieval side sees something closer to what it was trained on.

RAG vs fine-tuning: when to use which

This gets asked in every planning meeting. Short version:

• Use RAG when the answer depends on facts that change, are user-specific, are too many to fit in a prompt, or need to be cited. Almost every "chat with our data" use case is RAG.
• Use fine-tuning when you need the model to adopt a style, format, or reasoning pattern — not to remember facts. Fine-tuning a model on your docs to answer questions is almost always the wrong call. It is expensive, hard to update, and still hallucinates.
• Use both when you have a domain (legal, medical, financial) where the model needs to reason in a specific way and ground answers in current documents. Fine-tune for the reasoning pattern, retrieve for the facts.

When RAG is a bad fit

Honest tradeoffs: RAG is bad at questions that require aggregation across your entire dataset ("how many tickets mentioned billing last quarter?" — that is a SQL query, not a retrieval problem). It is bad at multi-hop reasoning where the answer requires chaining facts across documents. It struggles when the source data is contradictory and there is no ground truth. And it adds real latency — two model calls plus a vector search per query.

If your users are asking analytical questions, you probably want text-to-SQL or a semantic layer, not RAG. If they are asking factual lookups grounded in documents, RAG is the right tool — provided you treat it as four contracts, not one pipeline.

The mindset shift

Stop thinking of RAG as "vector search plus an LLM." Start thinking of it as a chain where each link has a defined input, output, and failure mode. When answers are wrong, do not tweak the prompt first. Log the retrieved chunks. Read them. Ask: could a human answer the question correctly given only these chunks? If no, the retrieval contract failed. If yes, then look at the prompt.

That single diagnostic habit — separating retrieval failures from generation failures — is what moves a RAG project from "cool demo" to something you can actually put in front of customers. Teams building production AI features (including our own AI studio work) spend most of their engineering time on retrieval quality, not on prompts. That is not a coincidence.

Frequently Asked Questions

Why does my RAG chatbot hallucinate even when the answer is in my docs?

Almost always because the retrieval layer is returning the wrong chunks with high similarity scores. The LLM does not see your full document set — it only sees the top-K chunks the retriever hands it. If those chunks are topically similar but factually wrong (old versions, wrong tenant, missing context), the model will confidently answer from bad input. Log the retrieved chunks before blaming the model.

Do I need a vector database, or can I use Postgres?

For under a few million chunks, Postgres with the pgvector extension is usually enough and lets you keep metadata filtering, transactions, and vector search in one place. Dedicated vector DBs (Pinecone, Weaviate, Qdrant) become worth it at higher scale or when you need advanced features like hybrid search or distributed indexing. Start with pgvector unless you already know why you cannot.

Should I fine-tune a model instead of using RAG?

Not for factual recall. Fine-tuning is for teaching a model a style, format, or reasoning pattern — not for cramming your knowledge base into weights. RAG is faster to update, cheaper to run, and easier to debug when answers are wrong. Consider fine-tuning only when your problem is "the model does not reason the way our domain requires," not "the model does not know our facts."

How do I know if my retrieval is actually working?

Build an evaluation set: 50–200 real user questions paired with the chunks that should be retrieved to answer them. Measure recall@K (did the right chunk make it into the top K?) and precision. Without this, you are guessing. This is unglamorous work and it is the single highest-leverage thing you can do.

How long does it take to build a production-grade RAG system?

It depends heavily on data quality, volume, domain complexity, and what "production-grade" means for your risk tolerance. A prototype is a weekend. Something you would put in front of paying customers with acceptable accuracy is a different order of magnitude. For a scoped estimate against your specific data and use case, talk to CodeNicely for a personalized assessment.