Agent eval
Also known as: agent evaluation · evals (agents)
Agent evals systematically measure whether an agent completes its tasks — using test tasks, success criteria, and repeatable runs.
- A set of realistic tasks with defined success.
- Run the agent; check outcomes, not just outputs.
- Track success rate and cost over time.
The shift from chat evals: for an agent, the right question isn't "did it say the right thing" but "did it *do* the right thing, and did it finish?" Evals must check the end state.