Retrieval-Augmented Generation (RAG)

Large Language Models (LLMs) are trained on massive amounts of text, but their parametric memory is static. Once training finishes, they cannot learn new facts without being retrained or fine-tuned. Furthermore, if you ask an LLM for specific or obscure information, it often "hallucinates" a plausible-sounding but incorrect answer because it lacks the actual knowledge.

Retrieval-Augmented Generation (RAG) solves this by separating memory from reasoning. Instead of forcing the model to memorize every fact, RAG provides the model with a search engine.

How RAG works

1. Embed the query: When you ask a question, it is first converted into a dense vector (an embedding).
2. Vector search: We search a vector database containing document chunks (also embedded) to find the nearest neighbors to your query in the vector space. High cosine similarity means the text is semantically related.
3. Context injection: The most relevant document chunks are retrieved and prepended to your original query, forming an "augmented prompt".
4. Generation: The LLM reads the context and your query, and generates an answer strictly based on the provided text.

By explicitly providing the context, the LLM acts more like a reading comprehension engine than a memorization engine.

The retrieval half of this lab (embedding the query, computing cosine similarity against the knowledge base, and filtering by threshold) runs the real mathematics a retrieval system uses. The generation step is a stand-in: rather than calling an actual language model, it matches the top retrieved document against a short list of prewritten sentences. What is not modeled here is the LLM's generation call itself, only the search that decides what context it would be allowed to read.

Experiment

Try typing different queries in the interactive lab. Watch how the query's location in vector space changes, pulling in different nearest neighbors. If you set the similarity threshold too high, the system will fail to retrieve anything, leaving the model with no context to answer the question. Try the opposite too: drag the threshold all the way down and watch an unrelated document get pulled into the context alongside the real answer, precision traded for recall.