Giving AI Access to Knowledge

Ask a model what a flight to Barcelona costs next Tuesday and you may get a confident, plausible, entirely useless answer.

Ask a model what a flight to Barcelona costs next Tuesday and you may get a confident, plausible, entirely useless answer. Not because it is careless, but because its knowledge stopped at the end of its training data, and flight prices were not in it.

This is the fundamental limit of a language model used on its own: it is a fixed store of knowledge. A person in the same position can look something up. A bare model cannot.

The fix is straightforward once you see it. Do not ask the model to remember. Go and find the information first, hand it over, and then ask the question. The model does the reading and the writing; something else does the looking up.

That pattern has a name, and you will meet it constantly: retrieval-augmented generation, usually shortened to RAG. Retrieve the material, augment the question with it, generate the answer.

Why it matters

A model with no outside access is a student sitting an exam with an out-of-date textbook.
A model with retrieval is a student who can check the current sources before answering.

Worth Remembering Retrieval does not guarantee a correct answer. What it does is ground the answer in specific sources you can inspect, and give the model access to material more recent or more specialised than its training data. If the source is wrong, or the wrong passage is retrieved, the answer can still be wrong. Retrieval improves traceability and freshness. It does not confer truth.

How it works

Retrieval adds a research step before the model writes anything. Three stages:

Retrieve. The system searches a database, an interface to another service, or a document store for material relevant to the question.
Augment. What it found is combined with the user's question, so the model receives both.
Generate. The model answers using the retrieved material together with what it already knows.

The effect is that the answer rests on material that exists and can be checked, rather than on the model's recollection.

The difference it makes

The ruling in the example below is invented for illustration and does not refer to a real decision. The point is only to show the difference between a generic answer and a grounded one.

Without retrieval. “What is the latest ruling on data privacy law?” → “Data privacy laws focus on protecting user data, with regulations such as GDPR and CCPA.” Generic, and possibly out of date.

With retrieval. The system searches a legal database, finds recent decisions, and answers: “According to the ruling in [case] dated [date], amendments now require stricter encryption for cross-border transfers” — with a link to the source. Specific, grounded, and checkable by the reader.

That last property is the one that matters in professional use. An answer you can verify is worth far more than an answer that merely sounds authoritative.

The three parts of a retrieval system

A retrieval system needs somewhere to look, something that does the looking, and the model that writes the answer. Figure 7 shows how they fit together.

Diagram of a retrieval system: a knowledge source, a retriever, augmentation and a generator — Figure 7: Retrieve, augment, generate

The knowledge source

Where the material comes from. It might be:

An internal knowledge base: handbooks, policies, product documentation.
Live web search.
A verified specialist database — medical, legal, financial.
A live service, such as the airline system that knows Tuesday's actual fares.

The retriever

The component that finds the right material before the model is asked anything. It works in one of a few ways:

Keyword search, matching the words literally.
Search by meaning, so a question about “time off” finds a document titled “annual leave policy” even though they share no words.
Vector databases, which store the embeddings from Chapter 2 and make comparison by meaning fast at scale.

The retriever is where most of these systems succeed or fail. A strong model given the wrong three paragraphs will produce a confident wrong answer. Retrieval quality is the ceiling on answer quality, and it is where the effort belongs.

The generator

The model then writes the answer from the retrieved material plus its own capability. Done well, this step:

reduces the chance of the system inventing detail;
reflects current material rather than training-era material;
reads as a coherent answer rather than a pasted extract.

Where you meet it

Customer support. An assistant retrieves the answer from internal documentation instead of guessing, which cuts waiting time and keeps answers consistent with policy. A banking assistant pulls the specific overdraft terms for the customer's own account type rather than a general description.

Healthcare. Clinical tools retrieve current guidance and research before drafting anything for a clinician to review.

Legal and compliance. Retrieval over case law lets a system reference recent decisions rather than whatever happened to be in its training data.

Search itself. Tools such as Perplexity, and the search modes in Claude and ChatGPT, retrieve across sources and present a direct answer with citations instead of a list of links. Chapter 10 takes one of these apart.

Our travel assistant. This is the change that makes it useful at all. Rather than describing what flights to Barcelona are usually like, it queries the airline's own system and reports what is actually available on Tuesday, with the price attached.

Optional Technical Detail Documents are too long to retrieve whole, so they are split into passages — chunks — before being stored. How you split matters more than newcomers expect. Chunks that are too small lose the context that made them meaningful; chunks that are too large dilute the match and waste room in the context window. Splitting on natural boundaries such as sections and headings generally beats splitting every fixed number of characters.

Key takeaways

A model's knowledge is fixed at training time. Retrieval is how you get past that.
Retrieve, augment, generate: the model answers from material you supplied.
Retrieval grounds an answer in checkable sources. It does not make the answer true.
Retrieval quality sets the ceiling on answer quality. Invest there first.
Search by meaning finds relevant material that shares no words with the question, which is what embeddings buy you.
Retrieved text is untrusted input. Treat it as data, never as instructions.

A model that can look things up is a considerable step forward. It is still, though, only answering. It reports what Tuesday's flights cost; it does not compare them against your budget, notice that the cheap one lands too late for your meeting, or book anything. To cross that line it needs tools it can act through, somewhere to keep track of where it has got to, and a reason to keep going until the job is done. That is where an AI agent begins.