Definitely - when you consider how varied inference workloads will be, and the different ways to minimize costs - better prompting, SLMs, different chips, batching, etc, there will be tons of opportunity
Most people don't have the team and time to do heavy token efficiency engineering. But that's all we do. marketplace.neurometric.ai has a bunch of task specific small models, and we charge flat monthly fees. We bear the token risk.
Any thoughts about making this an open platform where people own their own data but apps are built on top of it? And users can design their own UIs too? I'd be an early paying customer of that API.