It's a huge industry, so a lot. Job is really stressful and has a lot of employee churn, so it's not really something I feel bad about. Pressing elevator buttons was a job too back then
feedback is generated based on evals.
example:
eval: function foo wasn't triggered even though [...]
feedback (exaggerated):
1. change stage prompt
2. change function description
3. add extra instructions to the end of the context
metrics are easy to generalize (e.g. call transfer rate), but baseline is different for each agent, so we're interpreting only the changes, not the absolute values (in the context of self-improvement).
Noisy is ok, but it doesn't work that well when there are multiple clear speakers and not much noise. We are planning to add speaker diarization to address this.