Ask HN: What metrics do you track when building agents?
4 comments
I think cost is lagging indicator. (it truely does)
I am currently tracking tool count exposed, schema-token size, tool-call failure rate, duplicate tool calls, retries, wall time, tokens before first useful external observation, and “answered without required tool evidence", and some more.
I strongly recommend single metric: task success with evidence, segmented by task type.
I am currently tracking tool count exposed, schema-token size, tool-call failure rate, duplicate tool calls, retries, wall time, tokens before first useful external observation, and “answered without required tool evidence", and some more.
I strongly recommend single metric: task success with evidence, segmented by task type.
However, a couple of days a go I discovered we were loading ~300 tools due to a bug, affecting both costs and accuracy. It was going on for some time and I didn't realize it by looking at the costs, cause it got hidden by other changes I made in the same period.
Now I started tracking that and some other metrics, but it made me wonder what else I might be missing. So: what are you tracking? Any tips? Curious to hear