How Generative AI Works

A system that reads I need to be in Barcelona by Tuesday lunchtime and replies with something sensible is doing something that deserves an explanation.

A system that reads I need to be in Barcelona by Tuesday lunchtime and replies with something sensible is doing something that deserves an explanation. This chapter gives you one — not the full mathematics, but enough that nothing in the rest of the book has to be taken on faith.

We will work from the outside in, and each idea will be attached to something you already know.

What generative AI is

Generative AI is artificial intelligence that does not only analyse or sort information. It produces something new.

One way to hold the difference:

Older AI is like a librarian. It looks things up and hands you what already exists.
Generative AI is like a writer. It composes new material from what it has learned.

So when you ask a model to write a paragraph, summarise an article or draft a description, it is not copying text out of a store somewhere. It is using patterns learned from a very large body of writing to produce something that did not exist before.

Kinds of generative AI

These systems are usually grouped by what they produce.

Text. Models such as ChatGPT, Claude, Gemini and Llama generate text from an instruction. They help with writing, summarising, research and code. Developers use tools such as GitHub Copilot and Claude Code to write and refine software.

Images. Other systems turn a written description into a picture. DALL·E, Midjourney and Stable Diffusion are used in design, marketing and the creative industries. Figure 2 was produced by typing a sentence describing what it should contain.

An image generated from a written description, showing a robot in a futuristic setting — Figure 2: An image generated from a written description

Several kinds at once. Some models handle more than one form of input and output: text, images, audio, video. You can hand such a model a photograph and ask a question about it, or a screenshot of an error and ask what went wrong. The usual word for this is multimodal, and it matters more for agents than it first appears — we return to it in Chapter 17.

What makes all of it work comes down to three things: neural networks, deep learning, and an architecture called the transformer.

Neural networks and deep learning

At the base of generative AI is the neural network. When someone says “AI works like the human brain”, this is what they are pointing at. Treat that as a teaching analogy rather than a description: the resemblance is loose, and pressing it too hard leads to bad intuitions about what these systems can do.

What a neural network is

Picture a large web of connected nodes. Each node takes numbers in, does something simple with them, and passes numbers on. Stack enough of these together and the network can pick up patterns and make predictions.

Show a network many thousands of pictures of cats and dogs and it will learn to separate them, using cues such as ear shape, fur texture and proportion. Figure 3 shows that happening: an image goes in, passes through layers of nodes, and a probability for each label comes out.

Diagram of a neural network classifying images of cats and dogs — Figure 3: A neural network learning to classify images

What deep learning adds

Deep learning is the same idea with many more layers. More layers means the network can represent more complicated patterns, because each layer builds on the representation the previous one produced.

Early layers pick up edges and textures. Later layers combine those into parts, and then into whole objects. Nobody tells the network to organise itself that way. It falls out of training.

Diagram showing deeper layers of a network producing finer representations — Figure 4: Deeper layers produce finer representations

How a language model works

A large language model is a deep learning model trained on an enormous quantity of text. Its core job is narrow, and slightly surprising: predict what comes next. Everything else it appears to do is built on that one operation.

When a search box completes how do I bake a… with cake, that is the same mechanism in miniature. Now imagine it at the scale of whole conversations, articles and programs.

Text passes through three stages, shown in Figure 5.

Step one: breaking text into tokens

The model does not read text the way you do. It breaks it into smaller units called tokens so it can work with them numerically. A token is often a word, but frequently a fragment of one.

Text: The cat sat on the mat.
Tokens: ["The", "cat", "sat", "on", "the", "mat."]

Diagram of the pipeline from input text through tokenising and embedding to a prediction — Figure 5: From text to a prediction

Step two: turning tokens into numbers

Because the model works only with numbers, each token is mapped to a list of numbers that encodes something about its meaning. Tokens used in similar ways end up close together in that numerical space. The representation is called an embedding.

This is the step that makes search by meaning possible rather than search by exact wording, and it is worth holding onto: it is the machinery behind everything in Chapter 4.

Step three: predicting what comes next

With the text as numbers, the model reads the sequence and predicts what follows, based on the patterns it absorbed during training.

Input: The cat sat on the …
Prediction: floor, sofa or mat, depending on what the model has seen.

Repeat that step, feeding each new token back in, and you get sentences, essays and working code.

The transformer

Today's models follow a long conversation and a shifting subject without losing the thread. That was not always true.

Earlier systems processed text one step at a time, carrying forward a compressed summary of everything they had read so far. That works for short spans, but the further apart two related words are, the harder it becomes for the connection between them to survive. A pronoun eleven sentences after the name it refers to was a genuine difficulty, not because the systems “forgot” in any human sense, but because their design gave long-range relationships very little room to persist.

That changed in 2017, when researchers at Google introduced the transformer.

Why it mattered

It can process the whole available context together. Rather than working through text step by step, a transformer computes the relationships between all the tokens in its context in parallel. Every word can be compared directly with every other word, however far apart they are.

This next point is important and often stated wrongly, so it is worth separating carefully. The parallelism is in reading. When the model writes, it is still sequential: it produces one token, appends it to the context, predicts the next, and repeats. Input processing can be highly parallel; generation remains sequential, token by token. That is why a long answer takes longer to appear than a short one, even though the input was read in a single pass.

It keeps track of reference. Give an earlier system Tom went to the store. He bought milk. What did he buy? and the connection could be lost. A transformer resolves he to Tom without difficulty.

It scales. Because the computation parallelises, transformers could be trained on far more data than what came before. That is what made models of today's size practical at all.

Attention

The mechanism at the centre of a transformer is attention. It lets the model weigh which tokens matter for interpreting each other token.

Think of reading a long report and needing only a handful of sentences to follow the argument. Attention does something comparable: it concentrates on the relevant parts and discounts the rest.

The sentence: The cat sat on the mat because it was tired.
The model must work out what it refers to.
Attention links it to the cat, not to the mat.

Diagram showing attention linking the pronoun it to the word cat rather than mat — Figure 6: Attention resolving a pronoun

A transformer runs very many attention computations at once, which is how it learns the dependencies that make language coherent.

Two terms you will meet constantly. Parameters are the internal numerical values a model adjusts during training; they are a rough proxy for its capacity. Context window is how much text the model can take into account at one time. A wider context window means a longer document or conversation can be handled without the beginning falling out of view.

In Practice Parameter counts and context window sizes move quickly, and vendors do not always publish them. Treat any specific figure you read as a snapshot rather than a fixed property, and check the model's own documentation when the number actually matters to a decision.

Pretraining and fine-tuning

Models do not arrive knowing things. They go through two distinct phases, and the difference between them explains a great deal about how AI products get built.

Pretraining

Pretraining is where the model learns as much as it can before being given a job. It reads an enormous, varied body of text and picks up grammar, facts, reasoning patterns and a great deal of implicit structure. What it does not have is specialisation. It knows a little about a great many things.

A pretrained model can draft a serviceable contract. It will not match one shaped by a particular firm's own legal writing.

Fine-tuning

Fine-tuning takes a pretrained model and trains it further on a smaller, higher-quality, domain-specific set of examples. The result is a model whose responses fit a particular context far better.

A support assistant fine-tuned on a company's own past conversations answers in that company's voice, in that company's format, without being told to each time.

Table 2-1: Pretrained compared with fine-tuned

	Pretrained	Fine-tuned
Purpose	General language ability	Competence at one kind of job
Training data	Very large and varied	Narrow and curated
Behaviour	Broad, generic	Shaped for one domain
Cost	Very high	Far lower

Worth Remembering There are three different ways to change what a model does for you, and they are not interchangeable. Instructions shape the request in front of it right now. Retrieval supplies knowledge it does not have, particularly anything current or private to your organisation — that is Chapter 4. Fine-tuning is for behaviour you want it to hold onto permanently: a consistent style, a strict output format, a specialised pattern that instructions cannot reliably produce. If what you need is today's prices, fine-tuning is the wrong tool no matter how much you spend on it.

Key takeaways

Generative AI produces new material rather than only sorting existing material.
Neural networks and deep learning are the computational base. The brain comparison is an analogy, not a description.
A language model's core operation is predicting the next token. Everything else is built on that.
Transformers compare every part of their context with every other part at once, but still generate one token at a time.
Attention decides which tokens matter for interpreting which.
Instructions shape the request, retrieval supplies knowledge, fine-tuning shapes lasting behaviour. Reach for them in that order.

So the machinery is not magic: it reads, it compares, it predicts. Which means the quality of what comes back depends heavily on the quality of what goes in. That turns out to be something you have a great deal of control over, and almost nobody uses well.