Gen AI

RAG Fusion

Instead of trusting a single search query, RAG Fusion generates multiple variations of the query, searches the database for all of them, and mathematically fuses the results together.

Why Does This Exist?

When a user submits a query to a Retrieval-Augmented Generation (RAG) system, they are pulling the slot machine lever exactly once. If they phrased the question poorly, or if they used a synonym that doesn't align well with the embedded documents, the retrieval will fail, and the LLM will hallucinate.

Query Rewriting attempts to fix this by intercepting the user's query and having an LLM rewrite it into a "better" query. But what if the LLM's single rewritten query is also slightly off?

RAG Fusion acknowledges that we cannot perfectly predict the single best search query. Instead of generating one rewritten query, RAG Fusion uses an LLM to generate multiple variations of the user's query from different angles. It executes all of these queries in parallel and then mathematically merges the results using an algorithm called Reciprocal Rank Fusion (RRF).

This ensemble approach dramatically increases the robustness of the retrieval pipeline, ensuring that a single poorly-phrased query doesn't ruin the entire RAG process.

Think of It Like This

Asking three different doctors

Imagine you have a mysterious pain in your shoulder, and you ask a doctor, "Why does my arm hurt when I lift it?" The doctor might focus on the word "arm" and retrieve information about a bicep tear. (A single query missing the mark).

Instead of asking one doctor, RAG Fusion asks an LLM to rephrase your question for three different specialists:

"What causes rotator cuff pain during elevation?" (Orthopedic query)
"How do pinched nerves in the neck manifest in arm pain?" (Neurological query)
"Common sports injuries related to overhead lifting." (Physical therapy query)

You send these three distinct queries into the medical database. The database returns books for each query. If a specific book on "Shoulder Impingement Syndrome" appears in the top results for all three queries, that book is given the highest score. You have fused multiple perspectives to find the absolute best answer.

How It Actually Works

The RAG Fusion pipeline consists of three steps:

1. Multi-Query Generation

The user's original query is sent to a fast LLM (like GPT-4o-mini). The prompt instructs the LLM to generate $N$ (usually 3 to 5) different variations of the query, attempting to capture different intents, synonyms, and abstraction levels.

User: "Impact of climate change on business" LLM Variations:

"Financial risks associated with global warming"
"Corporate sustainability adaptations to climate shifts"
"Economic consequences of extreme weather events on supply chains"

2. Parallel Retrieval

The system executes a standard vector search for the original user query plus all the generated variations. If you generate 3 variations and retrieve the top 5 documents for each, you will get 20 total documents (some of which will be duplicates).

3. Reciprocal Rank Fusion (RRF)

You cannot just sum up the cosine similarity scores from the different searches, because vector spaces can be heavily skewed. Instead, RAG Fusion uses Reciprocal Rank Fusion (RRF), a scoring algorithm that completely ignores the raw similarity scores and relies only on the rank of the document.

For every document retrieved, its RRF score is calculated as: $\text{RRF Score} = \sum_{q \in Q} \frac{1}{k + \text{Rank}_q(D)}$

$Q$ is the set of queries.
$\text{Rank}_q(D)$ is the rank (1st, 2nd, 3rd) of Document $D$ for query $q$ .
$k$ is a smoothing constant (usually set to 60).

If a document is ranked #1 for Query A, #3 for Query B, and didn't appear for Query C, its score is: $(1 / 61) + (1 / 63) + 0 = 0.0322$

Documents that consistently appear near the top across multiple query variations will mathematically bubble to the top of the final fused list. This final list of highly robust documents is then fed into the LLM for generation.

Show Me the Code

This is a clean, conceptual implementation of the Reciprocal Rank Fusion algorithm in Python.

1def reciprocal_rank_fusion(search_results_dict, k=60):2    """3    Fuses multiple ranked lists using RRF.4    5    search_results_dict: A dictionary where keys are queries and values are 6                         ordered lists of document IDs (from rank 1 to N).7    """8    fused_scores = {}9
10    # Iterate through the results of each query variation11    for query, doc_list in search_results_dict.items():12        13        # Iterate through the documents, noting their rank (1-indexed)14        for rank, doc_id in enumerate(doc_list, start=1):15            16            # Initialize the doc score if we haven't seen it yet17            if doc_id not in fused_scores:18                fused_scores[doc_id] = 0.019                20            # Add the reciprocal rank score21            fused_scores[doc_id] += 1.0 / (rank + k)22
23    # Sort documents by their final fused score in descending order24    reranked_docs = sorted(fused_scores.items(), key=lambda x: x[1], reverse=True)25    return reranked_docs26
27# --- Example ---28# We ran 3 queries. Doc_A appears in all 3. Doc_B is #1 in the first query, but doesn't appear again.29results = {30    "Original":  ["Doc_B", "Doc_A", "Doc_C"],31    "Variant_1": ["Doc_A", "Doc_D", "Doc_E"],32    "Variant_2": ["Doc_F", "Doc_A", "Doc_G"]33}34
35fused_results = reciprocal_rank_fusion(results)36
37print("Final RRF Rankings:")38for doc_id, score in fused_results:39    print(f"{doc_id}: {score:.5f}")40
41# -> Final RRF Rankings:42# -> Doc_A: 0.04838  <-- Bubbled to the top because it appeared consistently!43# -> Doc_B: 0.01639  <-- Dropped because it was a one-hit wonder.44# -> Doc_F: 0.0163945# -> Doc_D: 0.0161346# -> Doc_C: 0.0158747# -> Doc_E: 0.0158748# -> Doc_G: 0.01587

Watch Out For

Retrieval Latency Spike

Because RAG Fusion requires generating multiple queries and executing a vector search for every single one, it is inherently slower than standard RAG. If your vector database takes 100ms to execute a search, generating 5 query variations will take 500ms (unless you meticulously parallelize the database calls). Always execute the vector searches concurrently via threading or asyncio.

The Quick Version

Standard RAG relies on a single query, which is a single point of failure if the vocabulary is slightly off.
RAG Fusion uses an LLM to generate multiple semantic variations of the user's query.
It executes a vector search for every variation independently.
It merges the resulting document lists using Reciprocal Rank Fusion (RRF).
RRF ignores vector similarity scores and ranks documents based on how consistently they appear at the top of the different search variations.