must people think it’s just GPU cost. In practice it’s coordination: model latency variance + queueing + retries under load. You don’t scale linearly, you get cascading slowdowns.
What still seems unsolved is how to safely use it on real private systems (large codebases, internal tools, etc) where you can’t risk leaking context even accidentally.
In our experience that constraint changes the problem much more than the choice of runtime or SDK.
Interesting approach. One thing I’ve been running into is that even with good alerts, the harder problem ends up being when to actually be exposed. You can get the signal right and still underperform just because of timing and volatility.