What is RAG? Retrieval-augmented generation, without the jargon

Ask a general AI model about your company's refund policy and it will give you a confident, well written answer. It will also probably be wrong, because it has never seen your refund policy. Retrieval-augmented generation, or RAG, is the standard fix: before the model answers, look up the relevant documents and hand them over.

That is the whole idea. The rest of this guide explains how it works under the bonnet, when RAG is the right tool and when it is not, why it still matters now that models can read a million tokens at once, and how to build one, with or without code.

RAG in one paragraph

Think of an exam. A model without RAG is sitting it closed-book: it can only use what it memorised during training, and if it does not know, it may bluff. A model with RAG is sitting it open-book. A search step finds the most relevant pages from your own material and puts them in front of the model along with the question, and the model writes its answer from those pages. The name describes the three steps exactly: retrieve the relevant text, augment the prompt with it, then generate the answer.

Why models need it

A large language model learns from a huge snapshot of text and then stops learning. That leaves three gaps that RAG fills.

• It does not know your stuff. Your contracts, product manuals, support tickets, HR policies and meeting notes were never in its training data, and you would not want them to be.
• It goes out of date. Every model has a training cut-off. Prices change, policies change, products launch. Retraining a model every time a document changes is not practical. Updating a search index is.
• It cannot show its working. A model answering from memory cannot tell you where an answer came from. A RAG system can point to the exact paragraph it used, so a person can check it. In regulated work that is often the difference between usable and unusable.

RAG also reduces made-up answers, because the model is told to answer from the text in front of it and to say so when that text does not contain the answer. It does not eliminate them, which is covered below.

How RAG works, step by step

A RAG system has two halves: a preparation stage you run whenever your documents change, and a question stage that runs every time someone asks something.

Preparation: building the index

1. Collect the documents. PDFs, web pages, wiki articles, tickets, spreadsheets, whatever the answers live in.
2. Split them into chunks. A 60-page manual is too much to hand over whole for every question, so it is cut into passages of a few hundred words, usually with a little overlap so ideas are not sliced in half.
3. Turn each chunk into an embedding. An embedding model converts text into a long list of numbers that captures its meaning. Passages about similar things end up with similar numbers, even if they use different words. "How do I get my money back" and "refund policy" land close together.
4. Store them in a vector database. The embeddings, plus the original text and details such as the source and date, go into a database built to find "nearest" matches quickly.

Question time: retrieve, augment, generate

1. Embed the question with the same embedding model.
2. Retrieve the handful of chunks whose meaning is closest to the question, typically somewhere between three and twenty.
3. Augment the prompt: the model receives an instruction ("Answer using only the sources below, and cite them"), the retrieved chunks and the question.
4. Generate the answer, ideally with references back to the chunks it used.

In code, the question stage is short. This sketch shows the shape of it, whichever libraries you use:

question = "What is our refund window for annual plans?"

# 1. Retrieve: find the passages closest in meaning to the question
hits = vector_db.search(embed(question), top_k=5)

# 2. Augment: put them in the prompt with clear instructions
sources = "\n\n".join(f"[{h.id}] {h.text}" for h in hits)
prompt = f"""Answer using only the sources below. Cite source ids.
If the answer is not in the sources, say you don't know.

{sources}

Question: {question}"""

# 3. Generate
answer = llm.generate(prompt)

RAG vs fine-tuning vs long context

RAG is one of several ways to get a model to use knowledge it was not trained on. They solve different problems, and mixing them up is the most common and most expensive mistake teams make.

The rule of thumb: fine-tuning teaches a model how to behave, RAG gives it something to read. If the problem is that the model does not know your facts, fine-tuning is the wrong tool. Our guide on how to fine-tune an LLM covers the cases where it is the right one.

Doesn't a million-token context window make RAG pointless?

Several 2026 models, including DeepSeek V4, Kimi K3 and Google's Gemini line, can read around a million tokens in one go, roughly a few thousand pages. For a handful of long documents you only need once, pasting them in is often simpler and better than building a RAG system.

It does not replace RAG for everything else. Most organisations have far more than a million tokens of material. Sending everything with every question is slow and, even with caching, much more expensive than sending five relevant passages. Models also still pay less attention to material buried in the middle of a huge prompt. In practice, the two now work together: retrieval picks the most relevant material, and a large context window means you can afford to send more of it.

Where MCP fits

The Model Context Protocol lets a model call tools, including search tools. A RAG search can be one of those tools, so the model decides when to look something up rather than searching on every question. MCP is the plumbing that connects the model to a source. RAG is the technique for finding the right passages inside that source.

Where RAG goes wrong

Most disappointing RAG systems fail at the search step, not the model. The usual culprits:

• Bad chunking. Cut a table in half or separate a heading from its paragraph and the meaning is lost. Splitting along the document's own structure (sections, headings, rows) beats cutting every 500 words.
• Search by meaning misses exact terms. Embeddings are good at concepts and poor at product codes, names and policy numbers. Ask about "SKU 4471-B" and the closest match by meaning may be a different product.
• Messy or contradictory sources. If the index holds three versions of the same policy, the model may quote the old one. RAG faithfully retrieves whatever you put in, including the mistakes.
• Questions that need the whole set. "Which of our 400 contracts have a break clause?" is not a top-five search, it is a job for processing every document.
• Security leaks. If the index ignores permissions, a junior employee can ask a question and get an answer built from a board paper. Retrieval must respect who is allowed to see what.
• Prompt injection in documents. Text inside a retrieved web page or email can carry instructions aimed at the model. Treat retrieved content as data, never as orders.

What good RAG looks like in 2026

The basic version above is a starting point. The systems that work well in production usually add some of the following.

• Hybrid search. Run a traditional keyword search alongside the meaning-based search and merge the results. This fixes most of the exact-term problem.
• Reranking. Retrieve a generous set of, say, 50 candidates, then use a second, more careful model to pick the best five. It is cheap and often the single biggest quality improvement.
• Metadata filters. Limit the search by date, department, product or permission before ranking, so the model only sees current, relevant and allowed material.
• Agentic retrieval. Instead of one search per question, the model plans: it rewrites the question, searches, reads, notices a gap and searches again. This handles complicated questions far better, at the cost of more time and tokens.
• Citations by default. Make the model cite its sources and show them in the interface, so people can check the answer in one click.
• A test set. Keep 30 to 50 real questions with known good answers and rerun them after every change. Without this, you are tuning by feel.

How to build one

Without writing code

You may already have a RAG system and not know it. Uploading files to a Claude Project, a ChatGPT project or custom GPT, or Google's NotebookLM, and asking questions about them, uses retrieval behind the scenes. For a team knowledge base of a few hundred documents this is often enough, and it is the fastest way to find out whether the idea works for you before anyone builds anything. Many workplace tools, from help desks to intranets, now include an AI search option built the same way. If you are weighing up which of these to pay for, our guide to AI tools for small business can help.

With code

A first working version takes a developer a day or two. The usual ingredients:

• An embedding model from OpenAI, Google, Cohere, Voyage or an open model you host yourself.
• A vector store. If you already run Postgres, the pgvector extension is the simplest start. Dedicated options include Pinecone, Weaviate, Qdrant and Chroma.
• A framework (optional) such as LlamaIndex or LangChain to handle loading, chunking and retrieval, though plenty of teams write the few hundred lines themselves.
• A model to write the answers. A mid-tier model is usually enough because the hard part, finding the facts, has already been done. Our 2026 model timeline lists the current options and prices.

Start with one well-defined collection, such as your help centre, and a list of real questions people ask. Get those answered well, with sources, before you add anything else. Getting a good result out of the final step is still partly a prompting job, so the techniques in our prompt engineering basics guide apply here too.

Frequently asked questions