About this book:

This book provides a technical roadmap for the rigorous evaluation of artificial intelligence agents, emphasizing that claims of intelligence and utility must be grounded in reproducible, empirical evidence. The text begins by establishing a foundational distinction between an agent’s underlying capacity to generalize (intelligence) and its realized value in specific contexts (utility). To measure these qualities, the authors propose a comprehensive taxonomy of metrics spanning functional performance, safety and risk, human satisfaction, and resource efficiency. Central to this approach is the disciplined definition of tasks, abilities, and success criteria, which prevents common pitfalls such as benchmark gaming and the conflation of narrow proxy metrics with real-world outcomes.

The middle chapters focus on the "how" of experimental design, detailing the construction of robust datasets and the protocols required for reliable human annotation. The book delves deeply into the statistical foundations of evaluation, explaining how to use power analysis, confidence intervals, and bootstrapping to quantify measurement error and uncertainty. It advocates for a tiered evaluation strategy that includes offline analysis of historical logs using counterfactual estimators, high-fidelity simulations for rare or hazardous scenarios, and live online testing—such as A/B tests and multi-armed bandits—protected by automated safety guardrails.

A significant portion of the book is dedicated to the ethical and operational dimensions of agent deployment. This includes specialized methodologies for auditing fairness and bias, probing for vulnerabilities through adversarial red-teaming, and developing explainability metrics to make "black-box" decisions interpretable to humans. The authors also highlight the practical constraints of real-world systems, offering frameworks for measuring latency, computational cost, and the quality of mixed-initiative human-agent collaboration. By aggregating these disparate signals into multi-objective composite scores, stakeholders can make more informed decisions about agent readiness.

The final chapters move from individual experiments to the broader ecosystem of benchmarks, leaderboards, and infrastructure. The text outlines principles for benchmark governance to prevent overfitting and ensure scientific integrity. It concludes with a call for standardized reporting, the use of documentation checklists (like Model Cards), and the implementation of automated infrastructure for continuous monitoring. By treating evaluation as a persistent, lifecycle-long process rather than a one-time checkpoint, the book provides the tools necessary to build and maintain agents that are demonstrably intelligent, safe, and useful.