Simulation for Agentic Evaluation(yortuc.com)
yortuc.com
Simulation for Agentic Evaluation
https://yortuc.com/posts/simulation-for-agentic-evaluation/
3 comments
any thoughts on how to scale agent simulations?
Try: https://langwatch.ai/scenario/
Pretty amazing into running simulations at scale
Pretty amazing into running simulations at scale
That is cool but this seems to be only for simulating user behavior. Form the docs: `Test real agent behavior by simulating users in different scenarios and edge cases`.
What I mean by simulation is more on simulating a situation an agent may come accross in production. All read-only tool calls return simulated responses based on the environment state. Write actions change the environment according to scenario.
What I mean by simulation is more on simulating a situation an agent may come accross in production. All read-only tool calls return simulated responses based on the environment state. Write actions change the environment according to scenario.