My RAG Isn't Working Well: How to Fix Your Application in 4 Practical Steps
The Moment I Stopped Blaming the Model
I’ve lost count of how many times I’ve heard (and said) the same line: “my RAG works perfectly in testing, but in production it starts giving weird answers.” The natural reaction is to swap LLMs, try a more expensive model, tweak the prompt again. I’ve done that. It didn’t fix anything.
The problem is almost never the model. It’s the retrieval pipeline — the boring, silent part nobody wants to debug because it “already looks done.”
After banging my head against this enough times, I landed on a 4-step flow that fixes practically every RAG case that returns generic, incomplete, or flat-out wrong answers. Let me walk you through exactly where each step fits and why.
Step 1: Query Cleaning and Clarification (Query Rewriting)
Half of a RAG’s bad answers aren’t even the retrieval’s fault — they’re the fault of the question that arrived at it. If the user sends something vague or context-free, the search retrieves low-quality chunks, because it’s searching for the wrong thing.
How I fix it:
- Use conversation history. I add an intermediate step where a lightweight LLM rewrites the user’s question using the chat’s context. If someone asks “why did the bird eat that?”, this step turns it into something like “why does the woodpecker feed on tree larvae, according to article X?” — now that’s an actual searchable query.
- Set up clarifying questions. When the system flags a query as too vague, it asks for a quick context confirmation before hitting the vector store. It costs one extra turn of conversation, but it prevents confidently serving garbage.
Step 2: Hybrid Search (Vectors + BM25)
This was the trap I got stuck in the most early on: relying solely on vector (semantic) search.
Vector search is great at capturing concepts and semantic meaning. The problem is it fails miserably at exact terms — error codes like ERR_404_NET, serial numbers, product SKUs, rare proper nouns. Vectors understand “meaning”; they don’t understand “exact string.”
How I fix it:
I implement hybrid search, combining vector search with a traditional keyword search algorithm like BM25. Keyword search guarantees you catch the exact term; vector search guarantees you understand the context around it. Each covers the other’s weakness.
Step 3: Metadata Filtering
While your knowledge base is small, searching everything works well enough. The problem shows up when it grows to thousands of documents — at that point, searching the entire database just adds noise and tanks precision.
How I fix it:
At chunking time, I enrich each document chunk with metadata tags:
- Publication date or year (
year: 2024) - Document type (
category: refund_policy) - Access level or department
When someone asks “what’s the refund policy for subscriptions in 2024?”, the system applies a metadata filter before vector search, restricting the pool exclusively to 2024 documents. This eliminates the risk of retrieving an old, outdated policy — which, honestly, is one of the worst kinds of mistakes a corporate system can make.
Step 4: Reranking
Hybrid search returns a broad — and sometimes messy — list of candidate chunks. This is where the Reranker comes in.
How I fix it:
The Reranker is a specialized scoring model that reads the user’s question against each chunk retrieved by hybrid search, re-evaluates each one’s actual relevance, and reorders the list. The goal is simple: make sure the most precise chunks land at the top of the context that gets sent to the LLM — because, at the end of the day, models tend to weigh what’s at the beginning of the context more heavily.
The Ideal Pipeline: Order of Operations
For all of this to work together, execution order matters. This is the sequence I follow:
| Step | Pipeline action | Main function |
|---|---|---|
| 1. Query Cleaning | Rewriting and clarifying the question | Turn the user’s question into a precise search |
| 2. Hybrid Retrieval | Vector search + BM25 | Capture both semantic meaning and exact terms |
| 3. Metadata Filtering | Filtering by tags and categories | Narrow search scope to relevant documents |
| 4. Reranking | Reordering by relevance score | Put the best chunks at the top of the LLM’s context |
What I Actually Think
Every time someone asks me “which model should I use for my RAG,” I feel like the question is one step ahead of itself. The model matters, but it’s the last piece of the machine — not the first.
What I’ve learned, the hard way, is that improving a RAG isn’t about buying bigger intelligence. It’s about building layered retrieval engineering, where each step fixes the structural failure of the step before it. A bad query becomes a bad search. A bad search becomes bad context. Bad context becomes a bad answer — no matter how expensive the model at the end of the chain is.
I’m Left With This Question
If your RAG works great in testing and fails in production, I’d bet the problem sits in one of these four steps — well before the LLM ever enters the picture.
Which of these four layers is missing from your pipeline today? Let me know:
- Email: fodra@fodra.com.br
- LinkedIn: linkedin.com/in/mauriciofodra
Query rewriting, hybrid search, metadata filtering, reranking. In that order, your RAG stops delivering generic answers and starts operating with enterprise-grade precision.
Read Also
- Piece by Piece: How Chunking Strategies Define Your RAG’s Success — Before the search, there’s chunking. If the retrieval base is already cut poorly, none of these 4 steps will fix it on their own.
- Fine-Tuning vs. RAG: The Definitive Guide for Your AI Strategy in 2026 — Before optimizing your RAG, it’s worth confirming RAG was even the right choice for your use case.
- RAG without Vectors: The End of Document ‘Chunking’ in AI? — If retrieval still bothers you even after these 4 layers, this is an alternative approach worth knowing about.