This book is licensed for the personal use of the buyer. Please do not share or resell the file. Short quotations with attribution are welcome.
The information in this book is provided for education. It is not legal, financial or other professional advice. Product and company names are used only to describe them and belong to their owners.
This is a free sample: the introduction and first chapter.
IntroductionIntroduction
Every AI project reaches the same moment. Someone shows a demo; it handles five questions beautifully; people are impressed; a launch date is set. Then real users arrive with questions nobody anticipated, in phrasings nobody tried, about situations the demo never covered, and the system that looked brilliant turns out to be unreliable in ways nobody measured.
"It looked good in the demo" is not an evaluation. Evaluation is the discipline of measuring, systematically and honestly, whether an AI system does what it is supposed to do, for the people who will actually use it, and whether it keeps doing so over time.
This book covers evaluation for both classic predictive models and generative AI systems:
Chapter 1 explains why evaluation is the real work of AI projects.
Chapter 2 works through classic metrics with numbers.
Chapter 3 gives the statistics every evaluator needs: intervals, paired comparisons, the bootstrap and sample sizes.
Chapter 4 shows how to build test sets that reflect reality.
Chapter 5 covers evaluating generated text and other open-ended outputs.
Chapter 6 explains how to use language models as judges, carefully.
Chapters 7 and 8 cover retrieval-augmented systems and agents.
Chapter 9 moves to online evaluation: A/B tests and monitoring.
Chapter 10 shows how to evaluate fairness across groups.
Chapter 11 covers red teaming and robustness.
Chapter 12 covers speech, vision and multimodal systems.
Chapter 13 explains how to read benchmarks and leaderboards critically.
Chapter 14 follows a complete evaluation of a support assistant, from plan to monitoring.
Chapters 15 and 16 cover running human evaluation well and building an automated evaluation pipeline.
Chapter 17 turns metrics into decisions using expected cost.
Chapter 18 provides an evaluation plan template.
Exercises with answers and a glossary close the book.
Basic familiarity with machine learning is assumed; every metric is explained with a worked example.
Chapter 1Why Evaluation Is the Real Work
Evaluation drives everything else
Without good evaluation, every other decision is guesswork:
Which model should we use? You cannot compare what you do not measure.
Did the new prompt help? Without a test set, you will notice the cases it fixed and miss the ones it broke.
Is it ready to launch? "Ready" needs a definition.
Is it still working? Performance drifts silently after launch.
Teams that invest early in evaluation move faster, because each change can be checked in minutes rather than debated for days.
Start from the decision, not the metric
Before choosing metrics, answer:
What will the system's output be used for, and by whom?
What does a good outcome look like for users and the business?
What do different kinds of errors cost? Missing a fraudulent transaction and blocking a legitimate one have very different costs.
What level of performance is needed for the system to be worth using? Compared with what: the current process, a simple rule, a human?
Levels of evaluation
Level
Question
Example
Component
Does each part work?
Does retrieval find the right documents?
System (offline)
Does the whole system produce good outputs on a test set?
Are answers correct and grounded?
User (online)
Does it help real users in real use?
Do support tickets get resolved faster?
Business
Does it achieve the goal?
Lower costs, higher satisfaction, more sales?
Each level can look good while the next fails. A model with excellent offline accuracy may not change user behaviour at all. Plan evaluation at every level.
Baselines
Always compare against something: a simple rule, the previous system, a human, or a trivial model (always predicting the most common answer). A model that is 92% accurate sounds impressive until you learn that "always predict no" scores 91%.
You have reached the end of the free sample. The full book continues with 17 more chapters, plus everything else in its 81 pages.