Hey folks. Keen to hear your thoughts on this. I’m have been working with agent orchestration patterns for a while now and cost optimisation in general. I thought it would be cool if you could see what the cost of a task would be with different models before you commit. So far I have trained a model on >50k coding benchmark questions and I’m getting decent cost predictions. I’m enjoying the challenge and interested to know if anyone else is trying this or has ideas for making it better.
I think you’re spot on. In my experience people confuse a models ability to solve some benchmark as a sign of its usefulness. Token throughput is often just as important from my personal usage. I am excited for more diffusion models to see how progress happens there.