Dense Passage Retrieval (DPR)

The Problem

Open-domain question answering systems rely on a retriever to find relevant text passages before an answering model reads them. Traditionally, these retrievers used sparse lexical methods like TF-IDF or BM25, which match exact keywords. However, lexical search fails when the query uses synonyms or paraphrases not present in the document. Neural models could capture semantic meaning, but using a heavy cross-encoder to compare a query against millions of documents was computationally impossible.

The Idea

The authors proposed Dense Passage Retrieval (DPR), using a dual-encoder (bi-encoder) architecture. They trained two separate BERT networks: a question encoder and a passage encoder. By embedding all passages into a dense vector space offline, the system could process a query in real-time by embedding the question and performing a fast Maximum Inner Product Search (MIPS) using tools like FAISS to find the closest passage vectors.

How It Works

DPR relies on two independent BERT encoders:

Passage Encoder: Maps any text passage to a d-dimensional vector.

Question Encoder: Maps the user's question to a d-dimensional vector.

The relevance score is simply the dot product between the two vectors.

The critical innovation was the training scheme. To train the embedding space effectively, they used contrastive learning with "in-batch negatives." For every question, they provided one positive passage (containing the answer) and multiple negative passages. Instead of randomly sampling negatives, they used "BM25 negatives" (passages that match keywords but don't answer the question) to force the model to learn semantic distinctions, and efficiently reused positive passages from other queries in the same batch as additional negatives.

Why It Mattered

DPR conclusively demonstrated that dense neural retrieval could outperform highly tuned BM25 baselines in open-domain QA. It established the dual-encoder architecture as the standard for scalable semantic search, balancing high accuracy with the efficiency needed to search millions of documents in milliseconds.

What Came After

DPR became a core component of RAG architectures (including the original RAG paper). The paradigm shifted towards even better embedding models (like OpenAI's text-embedding-ada-002 or open-source equivalents like bge and jina). Later approaches, like ColBERT, introduced "late interaction" to bridge the accuracy gap between bi-encoders and computationally expensive cross-encoders.