Anthropic: Demystifying Evals for AI Agents(anthropic.com)

Anthropic: Demystifying Evals for AI Agents

https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

1 comments

I work on Promptfoo (an open-source eval framework). Appreciate the mention here. This post captures a lot of the hard lessons around agent evals. In particular, task ambiguity and brittle graders are things we run into constantly.