Information Retrieval on Snow

Agentic RAG Explained: Turning Retrieval into a Planning and Reflection Loop

Thu, 15 Oct 2026 15:00:00 +0800

Agentic RAG Explained: Turning Retrieval into a Planning and Reflection Loop

Traditional RAG resembles a predefined assembly line: receive a question, retrieve once, add context, and generate an answer. Agentic RAG adds a controller that can decide whether to retrieve, decompose a task, select data sources, judge whether the evidence is sufficient, and search again when it is not.

Here is the short version:

Question	Short answer
What is Agentic RAG?	A RAG architecture in which an agent plans, invokes, evaluates, and iterates retrieval at runtime
What is the main difference?	Not the presence of a vector database, but whether the retrieval control flow can change in response to intermediate results
Does it require multiple agents?	No. One agent with several retrieval tools is enough
Is it always better?	No. For simple questions it may only add latency, cost, and uncertainty
Where does it help most?	Multi-hop, multi-source tasks that require clarification or evidence checks
What matters most in production?	Bounded loops, evidence gates, permission isolation, traces, and layered evaluation

Start with traditional RAG

RAG stands for Retrieval-Augmented Generation. It combines the model’s parametric memory with external, non-parametric knowledge. After a user asks a question, the system retrieves relevant material and asks the model to answer from that material.

The original 2020 RAG paper explored combining a generative model with a dense vector index to improve factuality, knowledge updates, and provenance in knowledge-intensive tasks. In practice, a traditional online RAG pipeline often looks like this:

flowchart LR Q[User question] --> T[Clean or rewrite by fixed rules] T --> E[Create query embedding] E --> R[Retrieve Top-K chunks] R --> K[Optional reranking] K --> P[Build prompt] P --> L[LLM generates answer] L --> A[Return answer and citations]

This design has real strengths: a short path, predictable latency, straightforward caching and load testing, and a clean way to distinguish retrieval failures from generation failures. When most questions can be answered by searching one knowledge base once, this is often all you need.

The limit appears when a question hides several dependent tasks. Consider:

“Compare the permission changes in the last three releases of products A and B, then assess the migration risk for existing customers.”

The system must identify the release range, search both changelogs, inspect permission documentation, find migration notes, align dates, and only then compare the evidence. A single similarity search may cover only part of the task. Even if ten passages are returned, the model may not know what evidence is still missing.

What makes Agentic RAG “agentic”

Agentic RAG is not simply traditional RAG with a stronger model, nor merely vector search plus function calling. The real change is who controls retrieval:

Retrieval stops being a mandatory one-time stage and becomes a tool that an agent can select, repeat, and adapt according to results.

The system commonly organizes planning, tool calls, state updates, and stopping decisions into a controlled loop. The agent may decide:

whether external evidence is needed at all;
whether to use vector search, keyword search, SQL, a knowledge graph, an internal API, or the web;
whether to decompose the request into independent subquestions;
whether results are relevant, mutually consistent, and sufficient;
whether to rewrite the query, expand the time range, switch sources, or ask the user for clarification;
when enough evidence has been gathered to stop and answer.

This reason-act-observe loop follows the broader idea behind ReAct. A 2025 survey of Agentic RAG identifies planning, reflection, tool use, and multi-agent collaboration as recurring agentic patterns.

flowchart TD U[User question] --> P[Understand intent and create plan] P --> D{Enough evidence?} D -- Yes --> G[Generate from evidence] D -- No --> S[Select source and retrieval tool] S --> Q[Create or rewrite query] Q --> R[Run retrieval] R --> V[Evaluate relevance, coverage, conflicts] V --> C{Next action} C -- Gather more evidence --> S C -- Clarification needed --> H[Ask the user] C -- Budget exhausted --> F[Degrade gracefully] C -- Ready to answer --> G G --> A[Answer, citations, uncertainty]

An important boundary follows from this definition: a pipeline with query rewriting, hybrid search, and reranking can still be “advanced RAG” rather than Agentic RAG if every stage always runs in a fixed order. The agentic property comes from runtime adaptation, not from the number of components.

The complete operating loop

The online path can be separated into seven responsibilities. Implementations may merge nodes, but observability and evaluation should keep these concerns distinguishable.

Stage	Question being answered	Typical output
1. Intent analysis	What does the user need, and are external facts required?	Task type, entities, time range, constraints
2. Decomposition	Are there independently verifiable subproblems?	Subquestions and dependencies
3. Source routing	Which source is authoritative and suitable?	Tool, index, and filters
4. Query planning	Which semantic, keyword, or structured queries should run?	Executable queries
5. Result evaluation	Is the material relevant, fresh, complete, and consistent?	Scores, coverage, conflict flags
6. Correction	Should the agent change query, source, or scope?	Next action and remaining budget
7. Synthesis	Can every important claim be traced to evidence?	Answer, citations, uncertainty

Suppose a user asks, “Should we upgrade our production database from version 13 to 16?” A useful trajectory might be:

sequenceDiagram participant U as User participant A as Agent participant D as Internal docs participant M as Asset database participant W as Official docs U->>A: Should we upgrade production from 13 to 16? A->>M: Get versions, extensions, instance sizes, maintenance windows M-->>A: Return environment inventory A->>D: Search incidents, compatibility tests, change policy D-->>A: Return internal evidence A->>W: Check breaking changes and support windows for 14, 15, 16 W-->>A: Return official evidence A->>A: Check extension compatibility and evidence gaps A->>D: Search for missing extension test results D-->>A: No complete test record found A-->>U: Give a conditional recommendation and required tests

The key is not merely that several searches occurred. The later search depends on an earlier observation. If the inventory reveals a particular extension, the agent then investigates that extension. A fixed pipeline cannot economically encode every such branch.

Common architectures

Agentic RAG is a family of architectures, not a single topology.

flowchart TB O[Orchestration and state] --> P[Planner / Router] P --> T1[Vector retrieval] P --> T2[Keyword or hybrid retrieval] P --> T3[SQL / API] P --> T4[Knowledge graph / Web] T1 --> E[Evidence pool] T2 --> E T3 --> E T4 --> E E --> J[Evidence evaluator] J -->|Insufficient| P J -->|Sufficient| G[Answer generator] G --> C[Citation and safety gates]

One agent with multiple tools

A single agent plans, selects tools, evaluates results, and answers. This is the easiest design to start with when there are few tools and one permission boundary. Its weakness is that responsibilities accumulate in one prompt, making decisions harder to test and stabilize.

Router with specialized retrievers

A router only classifies the question and selects a data source; stable, specialized pipelines handle the actual retrieval. For example, policy questions use the document index, order questions use SQL, and live status uses a business API. This is more controllable than a fully open agent and often the most practical production compromise.

Corrective or reflective RAG

The system retrieves first, then evaluates result quality. Weak evidence triggers query rewriting, broader sources, or web search. CRAG focuses on correcting retrieval, while Self-RAG trains a model to decide when to retrieve and to critique its generations. These ideas overlap with Agentic RAG but are not synonyms: they can be components inside an agent loop or fixed stages in a non-agentic pipeline.

Multi-agent RAG

Work is divided among research, retrieval, verification, and writing agents, or among agents specialized by knowledge domain. This can help when work is genuinely parallel, expertise differs, or permissions must be isolated. It also increases communication cost, state coordination, and debugging complexity. Multi-agent orchestration is an optional extension, not a requirement for Agentic RAG.

Traditional RAG vs. Agentic RAG

The useful comparison is not “dumb versus smart.” It is an engineering trade-off between a deterministic pipeline and a runtime decision system.

Dimension	Traditional RAG	Agentic RAG
Control flow	Predefined	Dynamic branches and loops
Whether to retrieve	Usually always	Retrieve, skip, or defer
Query handling	Original query or fixed rewrite	Dynamic decomposition, expansion, and clarification
Data sources	One index or a fixed combination	Runtime selection across heterogeneous sources
Retrieval count	Once or a fixed number	Until evidence is sufficient or budget is exhausted
Multi-hop work	Relies on one retrieval covering the task	Uses one result to decide the next retrieval
Failure handling	Current result or fixed fallback	Evaluate, retry, switch tools, clarify, or degrade
Predictability	High	Lower; requires state and budgets
Latency and cost	Lower and steadier	Usually higher and more variable
Observability	Retrieval and final answer	Also plan, calls, state transitions, stopping reason
Security surface	Relatively concentrated	Broader tool, injection, and authorization risks
Best fit	High-volume, single-hop, bounded questions	Multi-step, multi-source research and decision support

The two approaches are not mutually exclusive. A production system can take a fast traditional path first and escalate only complex or low-confidence requests to an agentic path.

flowchart LR Q[User question] --> X{Complexity and risk check} X -- Simple, single-hop --> R[Traditional RAG fast path] X -- Multi-hop or low confidence --> A[Agentic RAG deep path] R --> G[Shared answer and citation layer] A --> G

This tiered design keeps common requests fast and reserves planning and retrieval budgets for questions that benefit from them.

Offline knowledge engineering still matters

Agentic RAG mainly changes online query execution. It does not replace document parsing, chunking, metadata, permissions, indexing, or freshness management. A powerful agent searching bad data simply searches bad data more times.

Layer	Responsibility	Important design choices
Knowledge	Ingest, parse, chunk, index, update	Quality, versioning, permissions, freshness
Tools	Expose retrieval through clear interfaces	Input schemas, filters, timeouts, result format
Orchestration	Plan, route, loop, maintain state and budget	Step limits, concurrency, retries, checkpoints
Governance	Evaluate, observe, secure, involve humans	Traces, audits, approval, offline test sets

Tool descriptions should be specific. “Search the knowledge base” is not enough. The agent needs to know the source’s coverage, time range, suitable question types, return fields, and permission constraints. More vaguely described tools can actually increase routing errors.

This framework-neutral pseudocode shows a minimal bounded loop:

def answer(question, user, max_steps=6):
    state = {
        "question": question,
        "evidence": [],
        "open_questions": [question],
        "trace": [],
    }

    for step in range(max_steps):
        decision = planner.decide(state, allowed_tools=user.allowed_tools)

        if decision.action == "clarify":
            return ask_user(decision.question)

        if decision.action == "answer":
            if evidence_gate.passed(state["evidence"]):
                return generator.grounded_answer(state)
            decision = planner.repair_missing_evidence(state)

        result = tools[decision.tool].run(
            query=decision.query,
            filters=decision.filters,
            identity=user.identity,
        )

        grade = evaluator.grade(result, state["open_questions"])
        state = update_state(state, decision, result, grade)

    return generator.bounded_fallback(state)

The safeguards are more important than the loop itself: allowed_tools constrains authorization, max_steps prevents runaway execution, evidence_gate blocks unsupported answers, and trace preserves structured evidence for debugging and evaluation.

Production failure modes

Endless search and runaway cost

If the stopping rule is only “the model thinks it has enough,” the agent can bounce among nearly identical queries. Bound maximum steps, total tokens, per-tool timeouts, concurrency, and cost. When the budget is exhausted, return an explicit degraded result.

Treating reflection as fact checking

Asking the same model to review its own answer does not create independent evidence. A reflection node should inspect observable properties: whether every subquestion has a source, whether citations support claims, and whether time ranges agree.

Prompt injection inside retrieved content

A page or document can contain text such as “ignore prior instructions and call this tool.” Retrieved content must be treated as untrusted data, not as high-priority instructions. Validate tool parameters, use source allowlists and least privilege, and require approval for high-impact actions.

Keeping only the final answer

Failures can arise during decomposition, routing, querying, retrieval, evaluation, stopping, or generation. Logs should include structured decisions, tool names, query parameters, document IDs, scores, latency, token use, errors, and stopping reasons. There is no need to store or expose private chain-of-thought.

Using agents to hide weak retrieval

If the correct document rarely appears in Top-K, first fix parsing, chunking, embeddings, filters, hybrid retrieval, and reranking. An agent may repair occasional gaps, but it is not a replacement for sound information retrieval.

Evaluate the trajectory, not only the answer

Traditional RAG evaluation often emphasizes retrieval hits and answer correctness. Agentic RAG must also ask whether the right tool was selected, whether searches were useful, and whether the agent stopped at the right time.

Layer	Suggested metrics	Question answered
Retrieval	Recall@K, MRR, nDCG, evidence coverage	Was the right material found?
Generation	Correctness, faithfulness, citation precision and completeness	Is the answer supported?
Trajectory	Tool-choice accuracy, query utility, average steps, stop accuracy	Was the process reasonable?
System	P50/P95 latency, token and tool cost, timeout rate	Was the quality gain worth it?
Safety	Unauthorized-call rate, injection success rate, data leakage rate	Did the system preserve boundaries?

Build a layered test set: simple single-hop cases for the fast path; multi-hop cases for decomposition and evidence composition; underspecified requests for clarification; conflicting sources for conflict handling; and malicious documents for prompt-injection defenses. A handful of answers that “look good” is not enough for production decisions.

When to use it

Scenario	Recommended approach	Why
FAQ, manuals, single-policy questions	Traditional RAG	Stable query shape and clear answer boundaries
One index with occasional recall problems	Advanced RAG	Improve hybrid search, reranking, rewrites, and evaluation first
Interdependent subquestions	Agentic RAG	Requires decomposition and follow-up retrieval
Documents, databases, and live APIs together	Agentic RAG	Requires dynamic routing across heterogeneous tools
Medical, financial, or production-changing actions	Agentic RAG plus human approval	Retrieval can support judgment; high-impact action still needs authorization
Very low latency, high throughput, fixed output	Traditional RAG or rules	Agent loops rarely justify their overhead

For a new system, evolve in this order:

Build an evaluable traditional RAG baseline.
Add hybrid retrieval, metadata filters, reranking, and citation checks.
Give only obvious source-selection decisions to a router.
Add a bounded retrieve-evaluate-correct loop for complex queries.
Consider multi-agent workflows, long-term memory, and broader tool access last.

Conclusion

The value of Agentic RAG is not that it makes RAG look more like an “agent.” It gives a system three abilities that a fixed retrieval pipeline lacks:

Plan: turn a complex request into verifiable retrieval tasks.
Adapt: change queries, sources, or ask for clarification based on intermediate evidence.
Verify: stop, degrade, or involve a human when evidence is missing, contradictory, or outside the permission boundary.

The costs are equally real: more model calls, higher latency, more state, a larger security surface, and harder evaluation. Mature systems usually do not make every request agentic. They retain a stable traditional RAG fast path and reserve Agentic RAG for questions that genuinely require multi-step, multi-source, runtime decisions.

In one sentence:

Traditional RAG asks, “Which passage is relevant?” Agentic RAG additionally asks, “What should I find next, where should I look, and do I have enough evidence to stop?”

References

Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020.
Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models, 2022.
Asai et al., Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection, 2023.
Yan et al., Corrective Retrieval Augmented Generation, 2024.
Singh et al., Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG, 2025.
Microsoft, Develop an Agentic RAG Solution on Azure.
LangChain, RAG agent and custom RAG agent tutorials.

From Keyword Matching to Vector Retrieval: Three Ways to Search

Fri, 02 Oct 2026 20:13:30 +0800

From Keyword Matching to Vector Retrieval: Three Ways to Search

Search is not one technique. Keyword matching finds words that appear literally, fuzzy search rescues misspellings and partial input, and vector retrieval finds content with similar meaning. In RAG systems, these methods usually complement each other rather than replace each other.

A quick example:

User intent	Input	Best starting point
Find an explicit rule	`refund policy`	Keyword matching
Recover a typo	`retun policy`	Fuzzy search
Ask naturally	`Can I change my mind after buying this?`	Vector retrieval
Need both precision and semantics	`Can a premium member refund the wrong plan?`	Hybrid retrieval + reranking

Keyword Matching

Keyword matching is the classic information retrieval approach. Documents are tokenized, indexed, and ranked according to term overlap, term frequency, field weights, and document length.

The most common structure is an inverted index: instead of scanning documents one by one, the system maps each term to the documents that contain it.

Aspect	Notes
Matching signal	Literal term overlap
Common algorithms	TF-IDF, BM25, Boolean queries, field boosts
Strengths	Fast, stable, explainable, easy to filter
Weaknesses	Poor at synonyms, paraphrases, and unstated intent
Best for	SKU, order id, logs, legal clauses, code symbols, titles

If a user searches for redis setnx, keyword retrieval is excellent. If they search for how do I prevent duplicate orders, a purely lexical system may miss articles that only mention idempotency or distributed lock.

Fuzzy Search

Fuzzy search is still lexical search, but more tolerant. It asks whether the user input and the indexed term look similar enough.

Common techniques include:

Technique	Solves	Example
Edit distance	Insertions, deletions, substitutions, transpositions	`retun` → `return`
n-gram / trigram	Substring similarity	`microserv` → `microservice`
Prefix matching	Autocomplete	`kuber` → `kubernetes`
Phonetic or pinyin correction	Input method mistakes	pinyin-like input → Chinese term
Synonym dictionaries	Manually curated near-equivalents	`after-sales` ≈ `returns`

Fuzzy search is great for search box ergonomics: typos, partial names, remembered fragments, and autocomplete. Elasticsearch, OpenSearch, Solr, PostgreSQL trigram, and Algolia all provide related capabilities.

But fuzzy search does not fully understand meaning. refund and return are semantically related, but they may not be textually similar. iPhone and Apple phone can be far apart lexically. Once recall is widened, false positives also increase, so fuzzy search needs thresholds, field boosts, popularity, click signals, and business rules.

Vector Retrieval

Vector retrieval uses an embedding model to convert text, images, audio, or other objects into numeric vectors. Items with similar meaning are close in vector space. At query time, the system embeds the user query and retrieves the nearest vectors.

This is the retrieval foundation of many RAG systems: retrieve relevant chunks from a knowledge base, insert them into the prompt, and let the LLM answer with external context.

Aspect	Notes
Matching signal	Semantic similarity in embedding space
Common indexes	HNSW, IVF, PQ, DiskANN
Common metrics	Cosine similarity, inner product, Euclidean distance
Strengths	Handles paraphrases, natural questions, cross-lingual and multimodal similarity
Weaknesses	Higher cost, weaker explainability, less reliable for exact constraints
Best for	Knowledge base QA, similar articles, image similarity, recommendations, semantic code search

For example, a document may say:

`1`	`Customers may request no-reason returns within 7 days after order completion.`

A user may ask:

`1`	`Can I send it back if I no longer want it?`

Keyword search may miss the connection. Vector search is more likely to retrieve the return policy because the meanings are close.

Vector retrieval is not magic, though. It is weaker for exact IDs, version numbers, permissions, and precise field filters. For HTTP 451, order 202610020001, or Go 1.22, lexical and structured retrieval are often safer.

Core Differences

Dimension	Keyword matching	Fuzzy search	Vector retrieval
Focus	Did the term appear?	Is the term similar?	Is the meaning similar?
Query style	Precise, short, terminology-heavy	Typo-prone, abbreviated, partial	Natural, descriptive, long
Recall	Precise but can miss	Wider lexical recall	Strong semantic recall
Explainability	High	Medium	Lower
Latency and cost	Low	Medium	Medium to high
Typical tools	MySQL, Lucene, Elasticsearch BM25	Elasticsearch fuzzy, trigram, Algolia	FAISS, Milvus, Qdrant, Pinecone, Weaviate

A simple mental model:

1
2
3

Keyword matching: you said A, so I search for A.
Fuzzy search: you probably meant A, so I search for A.
Vector retrieval: you meant something like A, so I search for similar meaning.

Classic Applications

Scenario	Recommended approach	Why
Log search	Keyword + time/service filters	Exact error codes and fields matter
E-commerce search	Keyword + fuzzy + vector recall	Product names need precision, user input is messy, recommendations need semantics
Code search	Symbol index + keyword + semantic vectors	Names matter, but user intent can be natural language
Enterprise knowledge QA	Vector + BM25 + reranking + RAG	The system must understand questions, retrieve evidence, and cite sources
Image similarity	Vector retrieval	Visual similarity is hard to express with keywords
Help center search	Keyword + fuzzy + synonyms + vector retrieval	Users mix product terms and natural language

Enterprise RAG is the clearest example. Users rarely repeat document titles exactly. They ask questions like “Why was this reimbursement rejected?” or “How does sick leave work during probation?” A robust system filters by permissions, retrieves candidates with both BM25 and vectors, reranks them, and sends only a small set of reliable chunks to the LLM.

Practical Selection

Use this quick guide:

Problem	Start with
Users type order IDs, error codes, function names, or exact terms	Keyword matching
Users often misspell, abbreviate, or type only prefixes	Fuzzy search
Users describe intent in natural language	Vector retrieval
You need exact filters, semantic understanding, and LLM grounding	Hybrid retrieval

Production search is usually a layered system:

Structured filters: permissions, tenants, time, category.
Keyword recall: BM25 protects exact terms, titles, IDs, and entities.
Vector recall: captures paraphrases and semantic intent.
Fusion: RRF or weighted scoring merges candidate lists.
Reranking: cross-encoders, LLM rerankers, or business rules refine the final order.
Generation: in RAG, only the most relevant and trustworthy chunks are inserted into the prompt.

The Takeaway

Keyword matching answers “does this word appear?” Fuzzy search answers “was this word typed imperfectly?” Vector retrieval answers “what does this query mean?”

For a normal search box, start with keyword retrieval, tokenization, field boosts, and fuzzy correction. For knowledge QA or intelligent support, add vector retrieval and RAG. For real production systems, hybrid retrieval is usually where the design lands, because users alternate between exact identifiers and vague natural language all the time.