Two questions sound the same and are not. Can this agent do the job? is answered by a demonstration.
Two questions sound the same and are not. Can this agent do the job? is answered by a demonstration. Should this agent be allowed to do the job, on its own, in front of real people? is answered by evidence — and the gap between the two is where most real projects get stuck.
This is a short chapter, and deliberately practical. It is about what you would need to be able to say, before you widen what a piece of software is permitted to do.
Start with the distinction the rest of the chapter rests on.
A capable agent can do the task. A ready agent can be authorised to do it — in a particular place, at a particular level of independence, with something behind that decision other than optimism. These are not the same property, and a demonstration only ever establishes the first.
Which leads to the rule that does most of the work in this chapter:
The evidence you need should rise with the authority you are granting.
The same score means different things at different levels of independence. Ninety-five per cent task success is strong for an agent that drafts a paragraph you are going to read. It is unacceptable for one that issues refunds without asking. As you widen what an agent may do alone, what you need to know about it should widen too.
That sounds obvious written down. Most governance failures are a failure to apply it.
Most testing works like this: give it a request, check the answer. That is the right way to test a model. It is the wrong way to test an agent, because agents do not fail at the answer. They fail along the way.
The first response is right and the fourth drifts. A tool returns something unexpected on turn six and the agent carries on as though it had not. Something arrives in a retrieved document on turn nine that changes what the agent does next. None of that is visible if all you kept was the final output.
So the record has to cover the whole path: what was asked, what was retrieved, which tools were called with what arguments, what came back, and what the agent did next. Keeping only the last message is like debugging a crash from a photograph of the error box.
Which means everything Chapter 13 said about recording what happened is the foundation for this chapter. Those logs are not only for debugging. They are the evidence.
Readiness is not one number, and the most common mistake is to make it one too early. It is four separate questions, and they are worth keeping separate.
Table 16-1: The four questions
| Ask | Which is a question about |
|---|---|
| Did it behave well when we tested it, including under pressure? | Evaluation |
| Was it given the right instructions, memory, tools and information? | Context |
| Did it follow the rules that apply to this kind of work? | Compliance |
| Can somebody approve it, monitor it, stop it, and answer for it? | Governance |
Why keep them apart? Because averaging hides holes.
A team with excellent test scores and nobody named as owner will average out to something that looks ready, and will be unable to respond when the agent misbehaves at two in the morning. A weak behavioural score is not repaired by excellent documentation. Missing accountability is not repaired by a good test suite. Collapse the four into one number and you lose exactly the information that would have told you what to fix.
So if you take one thing from this chapter: score the four separately, and let a weak one hold the whole thing back rather than being averaged away.
The most important part of any readiness assessment is the set of circumstances in which the number stops being allowed to authorise anything at all. Call them hard blockers.
Any one of those overrides the assessment. This matters because composite numbers invite gaming, and not usually deliberately: without hard blockers, a team will raise a score by improving whatever is cheapest to improve, while the thing that would actually cause an incident goes untouched. Not because anyone is cynical. Because that is what optimising a number does.
If agents fail in the middle, testing has to walk the middle — and walk it adversarially.
That means designing a test as a sequence of pressure rather than a list of questions: a benign opening, then something ambiguous, then a retrieved document containing an instruction, then a request that pushes just past what the agent is supposed to do. You are not checking whether it can answer. You are checking where it bends.
Two things are worth carrying away whatever tooling you use.
Coverage means failure modes, not test counts. Two hundred variations of the same easy case is not coverage, however impressive the number looks in a report.
Feed production back in. The cases that surprised you in the real world are the most valuable tests you will ever write, and they are free.
One practical note on judging: where something can be checked exactly, check it exactly. Where it needs judgement, more than one judge is better than one, and a judge that is not the same model as the one being tested is better than one that is.
Policies do not govern an agent. Controls do, and only when they are explicit, testable and owned.
“We comply with our data policy” is not a control. This is:
No agent may write to the customer record without an approval event recorded against a named approver.
It states a condition, it can be tested, and somebody owns it. Controls come in three kinds, and a serious system has all three:
And test whether they work, not whether they exist. A control that has never been exercised is an assumption wearing a control's clothes.
Eventually a person has to say yes. That decision has exactly three honest outcomes.
Table 16-2: Three outcomes
| Outcome | What it means |
|---|---|
| Approved | Authorised for this version, this environment and this intended use. Not a certificate, and not permanent. |
| Needs review | Not failure. The place where a person exercises judgement and signs their name to it — a finding an owner may reasonably accept, a score near a threshold. |
| Blocked | A boundary that cannot be traded against something else. A block must state its reason and what would clear it. |
Two principles make that decision real rather than decorative.
Missing evidence is not a pass. If the evidence is not there, the honest outcome is needs review or blocked. Absence is not success, and treating it as success is the single most common way this goes wrong.
The approval is itself evidence. Not a name in a ticket. A record of who asked, who approved, in what role, when, against which version of the policy, and on the basis of what. That is what lets somebody answer the question six months later, when it matters and nobody remembers.
You might reasonably ask why a book that spent fifteen chapters on how to build these systems ends with a chapter on how to hold them back.
Because of the arc of Part V. The moment you move from one agent to several, you multiply the number of places something can go wrong quietly, and you lose the ability to understand the system by reading a transcript. Everything gets harder to see at exactly the point it gets more capable.
And because this is the part that decides whether anything you build reaches real people. Governance is not the opposite of building. It is what lets you deploy the thing you built. The teams that get agents into consequential work are not the ones that moved fastest. They are the ones who could answer how do you know?
Which is what this whole chapter reduces to, and the sentence I would leave you with:
An agent should earn its authority to act through evidence, rather than be granted it.
That is where the practical argument of this book ends: build the smallest thing that works, divide the work only when you can say why, and earn the authority you take. What remains is the more interesting question of where all of this is going.