Oh that's very interesting, how ready for production is it? It only works for TF right?
> If you need a few dozen inferences per second per server, this is the cheapest way. And you're not depending on a proprietary solution whose parent company could go out of business in a year.
Definitely the cheapest way.
We've been in business for more than a year already actually :)
Yes! A more in-depth blog post is coming soon. We do host the hardware ourselves, for complete control over the GPUs. We found a great infrastructure provider that is also experiencing shortages.
We don't have any cold start delay! In our custom environment, you can do exactly what you are describing (running both CPU and GPU code). We provide you with access to the GPU and the CUDA libraries installed. It's basically lambda (minus the cold start) with GPU access.
We can scale a lot very quickly depending on how much you need.
Definitely not a CTF challenge. The product is still very early on and a lot of features are hidden/hard to find (that's one of the reasons the signup isn't open).
1. No it's not open source, but I'm not excluding it from the roadmap :) The selling point is really the easiness of it.
2. Yes! We will be releasing tokens soon so that anyone can interact with the API.
3. Models can be up to 1GB, each model gets their own server , we only do real-time predictions for now. Meaning the models are constantly running waiting for a request
I couldn't agree more, I started building a ML hosting platform to solve your second point. I'm thinking of building a managed NN service as it is a common pain point.
It helps us figure out what got done and where we are in our roadmap