Reason people want the best: people want to believe their project is so advanced that they need the most clever LLM possible. To say otherwise is to admit that it's not really frontier or novel in any way. And people don't like that.
OpenRouter sometimes chooses a very expensive provider. Try the floor slug or choose directly the provider. I moved to just putting 5 dollars directly on deepseek instead of going through OR.
I mean indsight is 20/20, but saying that is like saying "everyone will just use the best tools".
That's not what we see most places in the world for most types of resources.
We try to keep things simple. Everything has risks. No stall, run async, backward compatible. DB handles rollback via transactions. Happy to expand if interested.
Agreed. And the gained time either goes toward 1) more experiments, or 2) leisure, which makes you sharper in the lab and happier overall.
Not sure the "I saved $17,000 so far" framing is the most useful way to look at it, but it's a cool project and I love that people are doing this kind of thing.
The future will have LLMs running local at your laptop/devices. If not almost exclusively then at least for 90-95% of the tasks. Speculative decoding is just one technique out of many existing and more to come that will make this even more viable. The gap is closing on both fronts. Software gets faster/more clever. Hardware gets faster and smaller. The single user story is the story. I'm obviously speculating myself, but that's how I see it.
Manage the budget not the impl. Top down decisions like "use a cheap model" risk optimize for the wrong things. If we lose 90% cache hit on the expensive models to context switch to a cheap one, there's no savings. Set the budget, let the devs optimize.