A finance director showed me a chatbot a vendor had built for them. He typed in a question about the company’s expense policy. The bot answered with confidence, in clear English, citing a clause that did not exist. The vendor had insisted the system was grounded in the company’s documents. It had not been.
That is the moment a business owner meets the difference between an AI tool that uses your data and one that just claims to. The technical name for the first kind is retrieval-augmented generation, or RAG. By 2026, almost every vendor will tell you they use it. Whether they actually use it well is a separate question.
What is RAG?
RAG is a two-step setup. When a user asks a question, the system first searches a database of documents for relevant material. It then passes those retrieved snippets to a language model alongside the question, and the model writes an answer using that material as its context. The model still does the writing. The retrieved documents shape what it has to say.
The benefit is that an LLM on its own has no idea what your pricing is, what your customers signed last quarter, or what your refund policy says. It generates a plausible answer from patterns in its training data, which is to say, an answer about a generic company that does not exist. RAG closes that gap. Done well, the model can be confined to your actual content and asked to cite the documents it draws from.
The two pieces of plumbing underneath are an embedding model, which converts your documents into mathematical representations that capture meaning, and a vector database, which stores those representations so they can be searched by similarity. Vendors like Pinecone, Weaviate, AWS Bedrock Knowledge Bases, and many others sell this layer. The vendor selling you the product on top usually does not build it themselves.
Why it matters for your business
The first thing it changes is hallucination risk. A standard LLM hallucinates because it has no source of truth: it predicts the most plausible next word based on training data. RAG gives it a source of truth. When the retrieval works, the model is anchored to your real content and is far less likely to invent. That matters most for customer-facing work, regulatory communication, and anything an auditor might later read.
The second is currency. A foundation model has a training cut-off, often six to twelve months before you encounter it. A RAG system points at live documents, which means a policy change you make this morning is visible to the chatbot this afternoon. There is no retraining cycle, no waiting for a model release. You update the source, the system uses the update.
The third is auditability. Because the system retrieves specific documents to answer a question, a well-built RAG product can show its working: the answer, plus the passages it drew from. For UK businesses subject to ICO scrutiny on automated decisions, that audit trail is the difference between an AI tool you can defend and one you cannot.
Where you will meet it
You will meet RAG in any vendor pitch where the product is described as “trained on your data”, “grounded in your documents”, or “your AI assistant for your business”. Industry estimates put RAG language in roughly 80% of B2B AI pitches in 2026. It has become the default reassurance phrase, often used loosely. Vendors mean retrieval-then-generation, but they sometimes also mean fine-tuning, sometimes both, and sometimes neither.
You will also meet it in the small print of products you may already be using. Microsoft 365 Copilot’s enterprise tier uses RAG over your SharePoint and OneDrive content. Anthropic’s Claude with Projects is a RAG wrapper over uploaded files. ChatGPT’s custom GPTs do something similar. AWS Bedrock Knowledge Bases is a managed RAG service. The vendor pitching you a sector-specific tool has often built a lightweight wrapper on top of one of these.
The interesting place to meet it is in the demo. Ask the vendor to ingest a small bundle of your real documents, then ask it five questions you genuinely care about. Watch what happens when you ask something the documents do not cover. A good RAG product says it does not know. A poor one improvises.
When to ask about it, when to ignore it
Ask about RAG when the value of the product depends on knowing your specific business: customer service over your policies, internal Q&A over your knowledge base, a sector-specific assistant claiming to understand your processes. In all these cases, retrieval quality is the product. If retrieval is poor, the product is poor, regardless of which underlying model the vendor uses.
Ignore RAG language when the work does not need your data. A general writing assistant does not need to retrieve anything. A meeting summariser is given the meeting transcript directly, no retrieval involved. A code completion tool reads your code as context without a vector database in the middle. Vendors sometimes invoke RAG anyway because the term sells. It does not always apply.
The ignore-it-when half is also worth flagging when the vendor will not show you retrieval working on your data. “We use RAG” is a marketing statement. “Here is your document, here is the question, here is the passage we retrieved, here is the answer” is the working version. If the vendor cannot or will not run that demo, the term in their pitch deck is doing work that the product is not.
Related concepts
Embedding is the mathematical step underneath RAG retrieval. Your documents are converted into vectors, which are long lists of numbers that capture meaning. Two passages that mean similar things produce similar vectors, even if they share no exact words.
Vector database is the storage layer that holds those vectors and finds the closest matches to a new query. Pinecone, Weaviate, Qdrant, and pgvector are the names you will see on a vendor’s architecture diagram. For the typical SME, this is the vendor’s problem, not yours.
Semantic search is what RAG retrieval does at the search step: finds documents by meaning rather than by keyword match. A search for “money back” can return a document about “refund policy” because the meanings are close.
Retrieval quality is the umbrella term for how well the search step works. It is not a feature listed on the pricing page, it is the actual differentiator between a product that helps and one that hallucinates with extra steps.
Fine-tuning is the alternative path. Where RAG points the model at fresh documents at query time, fine-tuning bakes patterns into the model itself by retraining on examples. The two solve different problems. RAG handles knowledge; fine-tuning handles behaviour. Most production systems use both, but for a service-led SME, RAG is almost always where to start.
The honest test of any RAG product is the question that should not have an answer. Ask the bot something your documents do not cover. The product worth buying says it does not know.
