AI agent evaluation checks the complete behavior of an agent, not just whether its final answer sounds good. A useful test set includes normal tasks, missing information, tool failures, ambiguous instructions and attempts to trigger unsafe actions. Track success rate alongside latency, tool errors and cost.
A practical starter set is 20 to 50 real tasks, each with a note on what a correct outcome looks like, including a few that should end with the agent refusing or asking a question. Score each run on whether the goal was met, whether the right tools were used in a sensible order, and what it cost. Re-running the same set after every prompt or model change shows whether you fixed one case by breaking another. Review of tool calling choices is part of this.
Run evaluations whenever prompts, models, tools or workflow logic change. Keep examples of past failures as regression tests, and include human review for outcomes that cannot be scored reliably with simple rules. Related: LLM Evaluation, AI Agent Observability.