Three years into shipping AI for real businesses, we still see the same pattern: a beautiful RAG demo, a promising pilot, then a quiet collapse when the questions get multi-part. The system retrieves something. It sounds confident. It is wrong in a way that is hard to catch without evals.
This post is the long-form playbook we give engineering partners when they ask for “agentic RAG”, not a slide about embeddings, but the loop, the failure modes, and the decision of when a simpler pipeline is still the right call. The structure is intentionally close to the deep production guides you see on Dev.to: problem, architecture, components, code sketch, failure modes, costs, and a clear “when not to.”
Key idea: Agentic RAG is not “RAG with an agent costume.” It is a controlled loop that routes, retrieves, grades, retries, generates, and verifies, with budgets so the loop cannot run forever.
What traditional RAG gets wrong
Standard RAG is linear: embed the query, pull top-k chunks, stuff them into a prompt, generate. That works for narrow FAQ-shaped questions. It breaks when the user needs two documents, a comparison, a recency check, or language that does not match how the corpus was written.
The fixed-pipeline assumption
Linear RAG assumes one retrieval step is enough for every question. Ask: “Compare our cancellation rules for personal vs commercial plans, and which has the shorter waiting period?” You need at least two sections, a shared definition of “waiting period,” and a synthesis the source docs never wrote. Top-k similarity will not reliably do that, and it will not retry when the context is thin.
- Multi-hop questions that need facts from two or more documents
- Recency-sensitive answers when the index lags the business
- Numeric comparisons buried in tables or appendices
- Paraphrased user language that weakens vector similarity
What agentic RAG actually does
Agentic RAG turns the pipeline into a graph. An agent decides whether to retrieve at all, what to retrieve, whether the evidence is good enough, whether to reformulate and try again, and whether the final answer is grounded. The model is no longer a single shot, it is a supervisor over retrieval quality.
Five components we ship
- Router, classify the query: retrieve, answer directly, or decline
- Retriever, hybrid search (dense + keyword) with metadata filters
- Grader, reject weak context before generation burns tokens
- Generator, answer only from graded evidence
- Grounding check, verify claims against retrieved chunks; escalate if not
Each node has one job. That sounds pedantic until you debug a production incident at 2am and need to know whether the router, the retriever, or the grader lied to you.
A minimal graph sketch
In practice we implement this as a state machine (LangGraph or a thin custom orchestrator). Shared state carries the query, reformulations, docs, answer, retry count, and flags. Conditional edges encode “sufficient / insufficient / hallucinated.”
from typing import TypedDict, List
from langgraph.graph import StateGraph, END
class RagState(TypedDict):
query: str
docs: List[str]
answer: str
retries: int
grounded: bool
def build_graph():
g = StateGraph(RagState)
g.add_node("router", router_node)
g.add_node("retrieve", retrieve_node)
g.add_node("grade", grade_node)
g.add_node("generate", generate_node)
g.add_node("ground", ground_node)
g.set_entry_point("router")
g.add_conditional_edges("router", route, {
"retrieve": "retrieve",
"direct": "generate",
"decline": END,
})
g.add_edge("retrieve", "grade")
g.add_conditional_edges("grade", grade_ok, {
"ok": "generate",
"retry": "retrieve", # reformulate query first
})
g.add_edge("generate", "ground")
g.add_conditional_edges("ground", is_grounded, {
"yes": END,
"no": "retrieve",
})
return g.compile()Hard rule: cap retries (usually 2 to 3) and total token budget per request. Infinite “agentic” loops are how you turn a $0.03 query into a $3 outage.
Four failure modes that kill first deployments
1. Infinite or near-infinite loops
Graders that always say “insufficient” and routers that always say “retrieve again” will melt your bill. Fix with retry caps, exponential backoff on reformulation, and a terminal “I don’t know, escalate” path.
2. Graders that never reject
If the grader rubber-stamps every chunk, you built traditional RAG with extra latency. Calibrate the grader on a labelled set of bad retrieves. Track reject rate in production, a healthy system rejects often enough to matter.
3. Context overflow
Stuffing more chunks “just in case” quietly truncates the important ones. Prefer fewer graded chunks, structure-aware chunking, and citations that point to sources instead of dumping entire PDFs into the window.
4. Latency spirals
Five frontier-model calls per query feels clever until p95 is eight seconds. Tier models: small/fast for router and grader, larger only for generate and grounding. Most of the accuracy win comes from better retrieval and grading, not from upgrading every node to the biggest model.
Cost and when NOT to build this
In our deployments, simple lookups stay cheap; multi-hop agentic paths cost more and should be reserved for questions that need them. If 90% of traffic is “What is your refund window?”, keep a linear RAG path for that cohort and route only complex queries into the loop.
- Stay linear when questions are single-hop and the corpus is small and clean
- Add grading + retry when multi-hop and comparison queries show up in support logs
- Add full agentic orchestration when tools, permissions, and multi-system actions enter the picture
- Never start with the most complex graph, start with evals that prove the simple path fails
What we measure before calling it production
- Retrieval recall@k on a labelled question set (not just answer vibes)
- Grader reject rate and false-accept rate
- Grounding pass rate and human escalation rate
- p50 / p95 latency and cost per successful answer
- Prompt and index versions, so provider model updates do not silently drift quality
Demos hide all of this. Production does not. If you cannot name your failure modes and your budgets, you do not have agentic RAG, you have a more expensive chatbot.
The Digiflux bias
We would rather ship a graded linear pipeline with honest escalation than an unbounded agent graph that looks impressive in a demo. When the business needs multi-hop reasoning, we build the loop, with caps, logs, and a Phase 2 plan. That is the same philosophy behind our Discover → Design → Deploy → Iterate process: prove the pain, then earn the complexity.
If you are stuck behind a Logic Wall of “RAG that almost works,” bring one real multi-hop question and your corpus shape. We will tell you whether you need agentic retrieval, or a cleaner index and a smaller system.