The Moment I Stopped Blaming the Model

I’ve lost count of how many times I’ve heard (and said) the same line: “my RAG works perfectly in testing, but in production it starts giving weird answers.” The natural reaction is to swap LLMs, try a more expensive model, tweak the prompt again. I’ve done that. It didn’t fix anything.

The problem is almost never the model. It’s the retrieval pipeline — the boring, silent part nobody wants to debug because it “already looks done.”

After banging my head against this enough times, I landed on a 4-step flow that fixes practically every RAG case that returns generic, incomplete, or flat-out wrong answers. Let me walk you through exactly where each step fits and why.

Step 1: Query Cleaning and Clarification (Query Rewriting)

Half of a RAG’s bad answers aren’t even the retrieval’s fault — they’re the fault of the question that arrived at it. If the user sends something vague or context-free, the search retrieves low-quality chunks, because it’s searching for the wrong thing.

How I fix it:

  • Use conversation history. I add an intermediate step where a lightweight LLM rewrites the user’s question using the chat’s context. If someone asks “why did the bird eat that?”, this step turns it into something like “why does the woodpecker feed on tree larvae, according to article X?” — now that’s an actual searchable query.
  • Set up clarifying questions. When the system flags a query as too vague, it asks for a quick context confirmation before hitting the vector store. It costs one extra turn of conversation, but it prevents confidently serving garbage.

Step 2: Hybrid Search (Vectors + BM25)

This was the trap I got stuck in the most early on: relying solely on vector (semantic) search.

Vector search is great at capturing concepts and semantic meaning. The problem is it fails miserably at exact terms — error codes like ERR_404_NET, serial numbers, product SKUs, rare proper nouns. Vectors understand “meaning”; they don’t understand “exact string.”

How I fix it:

I implement hybrid search, combining vector search with a traditional keyword search algorithm like BM25. Keyword search guarantees you catch the exact term; vector search guarantees you understand the context around it. Each covers the other’s weakness.

Step 3: Metadata Filtering

While your knowledge base is small, searching everything works well enough. The problem shows up when it grows to thousands of documents — at that point, searching the entire database just adds noise and tanks precision.

How I fix it:

At chunking time, I enrich each document chunk with metadata tags:

  • Publication date or year (year: 2024)
  • Document type (category: refund_policy)
  • Access level or department

When someone asks “what’s the refund policy for subscriptions in 2024?”, the system applies a metadata filter before vector search, restricting the pool exclusively to 2024 documents. This eliminates the risk of retrieving an old, outdated policy — which, honestly, is one of the worst kinds of mistakes a corporate system can make.

Step 4: Reranking

Hybrid search returns a broad — and sometimes messy — list of candidate chunks. This is where the Reranker comes in.

How I fix it:

The Reranker is a specialized scoring model that reads the user’s question against each chunk retrieved by hybrid search, re-evaluates each one’s actual relevance, and reorders the list. The goal is simple: make sure the most precise chunks land at the top of the context that gets sent to the LLM — because, at the end of the day, models tend to weigh what’s at the beginning of the context more heavily.

The Ideal Pipeline: Order of Operations

For all of this to work together, execution order matters. This is the sequence I follow:

StepPipeline actionMain function
1. Query CleaningRewriting and clarifying the questionTurn the user’s question into a precise search
2. Hybrid RetrievalVector search + BM25Capture both semantic meaning and exact terms
3. Metadata FilteringFiltering by tags and categoriesNarrow search scope to relevant documents
4. RerankingReordering by relevance scorePut the best chunks at the top of the LLM’s context

What I Actually Think

Every time someone asks me “which model should I use for my RAG,” I feel like the question is one step ahead of itself. The model matters, but it’s the last piece of the machine — not the first.

What I’ve learned, the hard way, is that improving a RAG isn’t about buying bigger intelligence. It’s about building layered retrieval engineering, where each step fixes the structural failure of the step before it. A bad query becomes a bad search. A bad search becomes bad context. Bad context becomes a bad answer — no matter how expensive the model at the end of the chain is.

I’m Left With This Question

If your RAG works great in testing and fails in production, I’d bet the problem sits in one of these four steps — well before the LLM ever enters the picture.

Which of these four layers is missing from your pipeline today? Let me know:

Query rewriting, hybrid search, metadata filtering, reranking. In that order, your RAG stops delivering generic answers and starts operating with enterprise-grade precision.


Read Also