Gen AI

Agentic RAG

Instead of a rigid pipeline where retrieval happens once, Agentic RAG gives an LLM a set of tools and allows it to autonomously decide what to search, when to search, and when it has enough information to stop.

In a standard pipeline, data flows in one direction. In an agentic system, the LLM loops, calling tools, reviewing the results, and deciding its next move autonomously.

Why Does This Exist?

Every major advancement in RAG—from Query Decomposition to Self-RAG to Corrective RAG—was an attempt to make a rigid, linear pipeline more flexible. But ultimately, they are still hardcoded pipelines. If the system was coded to do a web search when the vector search fails, it will always do exactly that, regardless of nuance.

Agentic RAG represents a complete paradigm shift. Instead of hardcoding a sequence of operations (Search $\rightarrow$ Retrieve $\rightarrow$ Generate), you simply provide an advanced LLM (like GPT-4) with a toolbox. The toolbox contains a Vector Search tool, a SQL Search tool, a Web Search tool, and a Calculator.

You give the LLM a goal, and you let the LLM autonomously decide how to use the tools to achieve that goal. It can search, read the results, realize it searched for the wrong thing, search again, combine that with a SQL query, and finally output an answer. The "pipeline" is entirely dynamic, driven by the reasoning capabilities of the model.

Think of It Like This

The assembly line vs. the detective

Standard RAG is an assembly line. A question enters the factory. Station 1 (Vector DB) attaches 5 documents to it. Station 2 (LLM) reads whatever is attached and outputs an answer. It happens the exact same way, in the exact same order, every single time. It is highly efficient but completely inflexible.

Agentic RAG is a private detective. A client gives the detective a question. The detective might start by checking the archives (Vector DB). They read a file, find a name, and realize they need to check financial records (SQL DB). They don't find what they need there, so they call a contact (Web Search). The detective loops, thinks, and investigates until they are satisfied they have the full answer.

How It Actually Works

Agentic RAG is typically implemented using the ReAct (Reason + Act) framework. This framework forces the LLM to think aloud before it takes an action.

The ReAct Loop

Thought: The LLM analyzes the user's request and thinks about the next logical step. (e.g., "I need to find the company's revenue for 2023.")
Action: The LLM selects a specific tool from its toolbox and provides the arguments for that tool. (e.g., Tool: vector_search, Input: "2023 revenue report")
Observation: The orchestrator (the Python code running the agent) intercepts this request, actually executes the vector search, and hands the retrieved documents back to the LLM as an "Observation."
Loop: The LLM reads the observation. It might formulate a new thought, take a new action, or realize it has enough information to finish.
Final Answer: When the LLM decides it has satisfied the user's request, it stops looping and outputs the final response.

The Toolbox (Routing)

In Agentic RAG, "Retrieval" is just one of many tools. This solves the classic RAG routing problem. If a user asks, "How many PTO days do I have?", standard RAG will search the PDF employee handbook and fail (because your specific balance isn't in the handbook). An Agent will look at its tools, see hr_database_api, and query that instead of the vector database.

Show Me the Code

Building an agent from scratch requires a parsing loop, but modern frameworks like LangChain or LlamaIndex make it incredibly easy to define tools and instantiate an agent.

1from langchain.agents import initialize_agent, Tool, AgentType2from langchain.chat_models import ChatOpenAI3
4# 1. Define our Mock Tools5def vector_search(query):6    print(f"\n[Agent called Vector DB Tool with: '{query}']")7    return "The Q3 marketing campaign focused on Gen-Z influencers."8
9def sql_database(query):10    print(f"\n[Agent called SQL Tool with: '{query}']")11    return "Q3 Revenue: $15M. Q4 Revenue: $12M."12
13# 2. Wrap the functions in Tool objects with descriptions.14# The LLM uses these descriptions to decide WHICH tool to use.15tools = [16    Tool(17        name="VectorSearch",18        func=vector_search,19        description="Useful for finding qualitative information, strategies, and text documents."20    ),21    Tool(22        name="SQLDatabase",23        func=sql_database,24        description="Useful for finding quantitative data, exact revenue numbers, and user counts."25    )26]27
28# 3. Initialize the Agent29llm = ChatOpenAI(model="gpt-4o", temperature=0)30agent = initialize_agent(31    tools, llm, agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION, verbose=True32)33
34# 4. Ask a complex question35question = "Why did revenue drop in Q4, and what was the revenue difference from Q3?"36agent.run(question)37
38# --- Agent's Internal Thought Process (Simplified) ---39# THOUGHT: I need to find the exact revenue numbers for Q3 and Q4 to calculate the difference.40# ACTION: SQLDatabase("Q3 and Q4 revenue")41# OBSERVATION: "Q3 Revenue: $15M. Q4 Revenue: $12M."42# THOUGHT: The difference is a $3M drop. Now I need to find the qualitative reason why it dropped.43# ACTION: VectorSearch("reasons for Q4 revenue drop compared to Q3 marketing")44# OBSERVATION: "The Q3 marketing campaign focused on Gen-Z influencers." (Plus other docs...)45# FINAL ANSWER: Revenue dropped by $3M in Q4. This may be related to the shift away from the successful Gen-Z influencer campaign in Q3.

Watch Out For

Unpredictability and Infinite Loops

Because you are handing the control flow over to an LLM, the system is no longer deterministic. The LLM might get confused by the output of a tool, try the exact same tool again, get confused again, and enter an infinite loop, consuming massive amounts of API credits in seconds. You must always implement a max_iterations failsafe in your agent loops.

Latency is terrible

A standard RAG pipeline takes 1-3 seconds. An Agentic RAG pipeline might take 15-30 seconds because it is making multiple sequential calls to a heavy reasoning LLM (like GPT-4), waiting for tools to execute in between each call. Agentic systems are brilliant for asynchronous background research tasks, but they are often too slow for snappy customer-facing chatbots.

The Quick Version

Standard RAG pipelines are hardcoded and inflexible.
Agentic RAG replaces the rigid pipeline with an autonomous agent powered by a reasoning LLM.
The agent is given a toolbox containing various retrieval methods (Vector DB, SQL, Web Search).
Using the ReAct framework, the agent actively thinks, uses tools, reads the results, and decides its next steps dynamically until the question is answered.