> I'm assuming by the pricing it's "serverless" inference, what's the cold-start time like?
Yeah, you could probably call it serverless inference. However, due to the fact that all fine-tuned models are trained on the same base model(s), we have some interesting optimizations we can apply over standard "serverless" model deployment. The biggest is that we can keep the base model loaded in VRAM and only swap the trained weight deltas per request. This gives us sub-second cold-start times for inference in the average case.
> Any idea on inference costs?
Right now, we’re pricing inference at $0.5/M input tokens, $2.5/M output tokens. That’s in a similar price range but a bit lower than gpt-4o/Claude 3.5, which we consider the main models we’re "competing" with. As it’s our goal to democratize access to models/agents in the long run, we hope that we can drop the prices for inference further, which should be enabled by some other optimizations we’re currently planning.
Wow! Thanks for taking the time to think through it. Yes, you are exactly right! I couldn’t have described Augento better than this myself. We actually want to make writing a reward function completely optional and build some RLHF (Reinforcement Learning from Human Feedback) loop soon. One of our long-term goals is to bring the cost of RL down so the barrier of entry to fine-tuning big models is not as high as it currently is.
We could open-source (parts of) our platform. What specifically would you like to see open-sourced?
We thought about developing this into a piece of software you can run in your own cloud (for compliance and security) but at the moment this makes the GPU economics really difficult and would probably be only interesting/relevant to big enterprises.
Anyway, we're definitely curious to hear if anyone has interesting applications for an open-source version of Augento!
A C only kernel build without CONFIG_RUST will not build a single line of Rust code nor will it touch anything in the rust subdirectory. If you don’t want to deal with rust, you don’t have to at all.
Currently, our software stack only supports WiFi and Ethernet at the data link layer, but that’s mainly because these are easy to set up with linux/batman, not because of an architectural problem.
Other technologies are not only possible but also on our roadmap. Just recently, for example, I’ve been taking a first look into getting LoRa working with hyveOS, but this would probably be a more long-term project.
> I'm assuming by the pricing it's "serverless" inference, what's the cold-start time like?
Yeah, you could probably call it serverless inference. However, due to the fact that all fine-tuned models are trained on the same base model(s), we have some interesting optimizations we can apply over standard "serverless" model deployment. The biggest is that we can keep the base model loaded in VRAM and only swap the trained weight deltas per request. This gives us sub-second cold-start times for inference in the average case.
> Any idea on inference costs?
Right now, we’re pricing inference at $0.5/M input tokens, $2.5/M output tokens. That’s in a similar price range but a bit lower than gpt-4o/Claude 3.5, which we consider the main models we’re "competing" with. As it’s our goal to democratize access to models/agents in the long run, we hope that we can drop the prices for inference further, which should be enabled by some other optimizations we’re currently planning.