Launch HN: Cerebrium (YC W22) – Serverless Infrastructure Platform for ML/AI
34 comments
How does this compare to modal (modal.com)? Faster cold-start? Easier config? Asking because I've used modal quite a bit for everything from fine-tuning LLMs to running etl pipelines and it works well for me, and I haven't found any real competitors for them to even think of switching.
Why use modal instead of SkyPilot?
I haven't used SkyPilot so I am unfamiliar with the experience and performance.
However, some of the situations you would like to use Cerebrium over Skypilot are: - You don't want to manage you own hardware - Reduced costs: With serverless Runtime and low cold starts (unclear if SkyPiolet offers this and what the peformance is like if they do) - Rapid iteration: Unclear of the deployment process on SkyPilot and how long projects take to go live - Observability: Looks like you would just have k8s metrics at your disposal
However, some of the situations you would like to use Cerebrium over Skypilot are: - You don't want to manage you own hardware - Reduced costs: With serverless Runtime and low cold starts (unclear if SkyPiolet offers this and what the peformance is like if they do) - Rapid iteration: Unclear of the deployment process on SkyPilot and how long projects take to go live - Observability: Looks like you would just have k8s metrics at your disposal
Modal is a great platform!
In terms of cold starts, we seem to be very comparable from what users have mentioned and tests we have run.
Easier config/setup is feedback we have gotten from users since we don't have and special syntax or a "Cerebrium way" of doing things which makes migration pretty easier as well as doesn't lock you in which some engineers appreciate. We just run your Python code as is with an extra .toml setup file.
Additionally, we offer AWS Inferentia/Tranium nodes which offer a great price/performance trade-offs for many open-Source LLM's - even when using TensorRT/vLLM on Nvidia GPU's and gets rid of the scarcity problem. We plan to support TPU's and others in future.
We are listed on AWS Marketplace as well as others which means you can subtract your Cerebrium cost from your commited cloud spend.
Two things we are working on that will hopefully make us a bit different is: - GPU checkpointing - Running compute in your own cluster to use credits/for privacy concerns.
Where Modal does really shine is training/data-processing use cases which we currently don't support too well. However, we do have this on our roadmap for the near future.
In terms of cold starts, we seem to be very comparable from what users have mentioned and tests we have run.
Easier config/setup is feedback we have gotten from users since we don't have and special syntax or a "Cerebrium way" of doing things which makes migration pretty easier as well as doesn't lock you in which some engineers appreciate. We just run your Python code as is with an extra .toml setup file.
Additionally, we offer AWS Inferentia/Tranium nodes which offer a great price/performance trade-offs for many open-Source LLM's - even when using TensorRT/vLLM on Nvidia GPU's and gets rid of the scarcity problem. We plan to support TPU's and others in future.
We are listed on AWS Marketplace as well as others which means you can subtract your Cerebrium cost from your commited cloud spend.
Two things we are working on that will hopefully make us a bit different is: - GPU checkpointing - Running compute in your own cluster to use credits/for privacy concerns.
Where Modal does really shine is training/data-processing use cases which we currently don't support too well. However, we do have this on our roadmap for the near future.
We use Cerebrium for our Mixpanel for Voice AI product (https://voice.canonical.chat). Great product. So much easier to set up and more robust than other model hosting providers we've tried (especially AWS!). Really nice people on the the team, too.
Thanks Tom! Excited to to support you and the team as you grow
i had no idea what it does and just have this vague idea that they make it easy for you to deploy, host and use models. I looked the tutorials and was amazed by what can be done and decided to try it. My suggestion is to have more tutorials and perhaps one-click deployment for some really cool models. Another thing is support typescript and you will capture a big section of the developer market that does not come from ML background. After i finish an inteview with tutorial demo , it would be great to give an estimate of cost so that i know if i can afford to cost for my software
Thanks for the feedback! I like the sound of all of those:
- clearer messaging - more tutorials - one-click deploys - clear & upfront costing
We have plans to add other runtimes (like Typescript) in the future but Python is our focus for now.
- clearer messaging - more tutorials - one-click deploys - clear & upfront costing
We have plans to add other runtimes (like Typescript) in the future but Python is our focus for now.
Being a toml-n00b, why is this quoted? https://github.com/CerebriumAI/examples/blob/85815f8e09e9e77...
Related to that, it seems the syntax isn't documented https://docs.cerebrium.ai/cerebrium/environments/config-file...
Related to that, it seems the syntax isn't documented https://docs.cerebrium.ai/cerebrium/environments/config-file...
Do you mean why the individual file names aren't quoted?
You can see an example config file at the bottom of that link you attached - agreed we should probably make it more obvious
You can see an example config file at the bottom of that link you attached - agreed we should probably make it more obvious
heh, I don't need an example in the docs, the whole repo is filled with examples, but unless you expect some poor soul to do $(grep -r ^include . | sort | uniq) and guess from there, what I'm saying in that the examples -- including the bare bones one in your documentation -- do not SPECIFY what the glob syntax is. The good thing about standards is that there are so many to choose from, so: python's os.glob, golang's glob, I'm sure rust-lang has one, bash, ... I'm sure I could keep going
As for the quoting part, it's mysterious to me why a structured file would use a quoted string for what is obviously an interior structure. Imagine if you opened a file and saw
Versus the sane encoding of:
But I did preface it with my toml n00b-ness and I know that the toml folks believe they can do no wrong, so maybe that's on purpose, I dunno
As for the quoting part, it's mysterious to me why a structured file would use a quoted string for what is obviously an interior structure. Imagine if you opened a file and saw
fred = "{alpha: ['beta', 'charlie''s dog', 'delta']}"
Versus the sane encoding of:
fred:
alpha:
- beta
- charlie's dog
- delta
But I did preface it with my toml n00b-ness and I know that the toml folks believe they can do no wrong, so maybe that's on purpose, I dunno
Congrats and thank you! We’ve been a happy customer since early on. Although we don’t have much usage, our products are mostly R&D, having Cerebrium made it super easy to launch cost effectively on tight budgets and run our own models within our apps.
The support is next level - team is ready to dive into any problem, response is super fast, and has helped us solve a bunch of dev problems that a normal platform probably won’t.
Really excited to see this one grow!!
The support is next level - team is ready to dive into any problem, response is super fast, and has helped us solve a bunch of dev problems that a normal platform probably won’t.
Really excited to see this one grow!!
Thank you - appreciate the kind words! Happy to continue supporting you and the team.
Congrats on the launch!
We're definitely looking for something like this as we're looking to transition from Azure's (expensive) GPUs. I'm curious how you stack against something like Runpod's serverless offering (which seems quite a bit cheaper). Do you offer faster cold starts? How long would a ~30GB model load takes?
We're definitely looking for something like this as we're looking to transition from Azure's (expensive) GPUs. I'm curious how you stack against something like Runpod's serverless offering (which seems quite a bit cheaper). Do you offer faster cold starts? How long would a ~30GB model load takes?
Yes RunPod does have cheaper pricing than us however they don't allow you to specify your exact resources but rather charge you the full resource (see example of A100 above) so depending on your resource requirements our pricing could be competitive since we charge you only for the resources you use.
In terms of cold starts, they mentioned their cold starts are 250ms which I am not sure what workload that is on, or if we have the same measure of cold starts. We have had quite a few customers that we have told us we are quite a bit faster 2-4 seconds vs ~10 seconds although we haven't confirmed this ourselves.
For a 30GB model, we have a few ways to speed this up such as using the Tensorizer framework from Coreweave, we cache model files in our distributed caching layer but I would need to test. We see reads of up to 1GB/s. If you tell me the model you are running (if open-source) I can get results to you - you can message me on our Slack/Discord community or email me at [email protected] or
In terms of cold starts, they mentioned their cold starts are 250ms which I am not sure what workload that is on, or if we have the same measure of cold starts. We have had quite a few customers that we have told us we are quite a bit faster 2-4 seconds vs ~10 seconds although we haven't confirmed this ourselves.
For a 30GB model, we have a few ways to speed this up such as using the Tensorizer framework from Coreweave, we cache model files in our distributed caching layer but I would need to test. We see reads of up to 1GB/s. If you tell me the model you are running (if open-source) I can get results to you - you can message me on our Slack/Discord community or email me at [email protected] or
> Yes RunPod does have cheaper pricing than us however they don't allow you to specify your exact resources but rather charge you the full resource (see example of A100 above) so depending on your resource requirements our pricing could be competitive since we charge you only for the resources you use.
I may be misunderstanding your explanation a bit here, but Runpod's serverless "flex" tier looks like the same model (it only charges you for non-idle resources). And at that tier they are still 2x cheaper for A100, at your price point with them you could rent an H100.
I may be misunderstanding your explanation a bit here, but Runpod's serverless "flex" tier looks like the same model (it only charges you for non-idle resources). And at that tier they are still 2x cheaper for A100, at your price point with them you could rent an H100.
Ah I see they recently cut their pricing by 40% so you are correct - sorry about that. It seems we are more expensive compared to their new pricing
FWIW Their most expensive flex price I've ever seen for 80GB A100 was $0.00130 back in January of this year, which is still cheaper albeit by a smaller magnitude, if that's helpful at all for your own competitive market analysis.
(Congrats on the launch as well, by the way).
(Congrats on the launch as well, by the way).
Yeah Runpods cold start is definitely not 250ms, not even close. Maybe for some models idk but a huggingface model 8B params takes like 30 seconds to cold start in their serverless "flash" configuration.
Thanks for confirming! Our cold start, excluding model load is 2-4 seconds typically for HF models.
The only time it gets much longer when companies have done a lot with very specific CUDA implementations
The only time it gets much longer when companies have done a lot with very specific CUDA implementations
Very nice demo!
When you ran it the first time, it took a while to load up. Do subsequent runs go faster?
And what cloud provider are you all using under the hood? We work in a specific sector that excludes us from using certain cloud providers (ie. AWS) at my company.
When you ran it the first time, it took a while to load up. Do subsequent runs go faster?
And what cloud provider are you all using under the hood? We work in a specific sector that excludes us from using certain cloud providers (ie. AWS) at my company.
You are correct! After the first request, an image will be on a machine and it’s cached for future use. This makes subsequent container startups much faster. We also route requests to machines where the image is already cached as well as dedupe content between images in order to make startups faster
We are running on top of AWS however can run on top of any cloud provider as well as are working on you using your own cloud. Happy to hear more about your use case and see if we can help you at all - email me at [email protected].
PS: I will state that vLLM has shocking load times into VRam that we are resolving.
We are running on top of AWS however can run on top of any cloud provider as well as are working on you using your own cloud. Happy to hear more about your use case and see if we can help you at all - email me at [email protected].
PS: I will state that vLLM has shocking load times into VRam that we are resolving.
Congratulations on the launch!
I just shared this on Slack and it looks like the site description has a typo: "A serverless AI infrastructure platform [...] customers experience a 40%+ cost savings as opposed to AWS of GCP"
I just shared this on Slack and it looks like the site description has a typo: "A serverless AI infrastructure platform [...] customers experience a 40%+ cost savings as opposed to AWS of GCP"
Thank you - updated! My team makes fun of my spelling all the time!
[deleted]
Would this be a direct competitor of paperspace? If yes what do you feel are your strenghts vis-a-vis paperspace?
There are definitely some parallels between Cerebrium and paperspace, but I don't think they are a direct competitor. The biggest difference being that paperspace doesn't have a serverless offering afaik.
Cerebrium abstracts some functionality - like streaming and batching endpoints. I think you would need to build that yourself on paperspace.
Cerebrium abstracts some functionality - like streaming and batching endpoints. I think you would need to build that yourself on paperspace.
Paperspace lets you bring your own containers and will scale them automatically. I don't know whether that would qualify as "serverless".
I guess then the next question would be how quickly can they start executing your container from cold start when a workload comes in? Typically we see companies on around 30-60s
[deleted]
Good luck on your launch! Your loom drops audio after 4m25s.
Thanks for pointing that out!
which ui framework do you use? it's very nice
I think we used this UI kit: https://minimals.cc/
We are Michael & Jono, and we are building Cerebrium (https://www.cerebrium.ai), a serverless infrastructure platform for ML/AI applications - we make it easy for engineers to build, deploy and scale AI applications.
Initially, we’ve been hyper-focused on the inference side of applications, but we’re working on expanding our functionality to support training and data processing use cases—eventually covering the full AI development lifecycle.
You can watch a quick loom video of us deploying: https://www.loom.com/share/06947794b3bf4bb1bb21c87066dfcc66?...
How we got here:
Jono and I led the technical team at our previous e-commerce startup, which grew rapidly over a few years. As we scaled, we were tasked with building out ML applications to make the business more efficient. It was tough—every day felt like a defeat. We found ourselves stitching together AWS Lambda, Sagemaker, and Prefect jobs (this stack alone was enough to make me want to give up). By the time we reached production, the costs were too high to maintain. Getting these applications live required a significant upfront investment of both time and money, making it inaccessible for most startups and scale-ups to attempt. We wanted to create something that would help us (and others like us) implement ML/AI applications easily and cost-effectively.
The problem:
There are a ton of challenges to tackle to realize our vision, but we’ve initially focused on a few key ones:
1. GPUs are expensive – An A100 is 326 times the cost of a CPU, and companies are using LLMs like they’re simple APIs. Serverless instances solve this to an extent, but minimizing cold starts is difficult.
2. Local development – Engineers need local development environments to iterate quickly, but production-grade GPUs aren’t available on consumer hardware. How can we make cloud deployments feel as fast as just saving a file locally and retrying?
3. Cost to experiment - To run experiments we had to spin up EC2 instances each day, recreate our environment and run scripts. It was difficult to monitor logs, instance usage metrics as well as run large processing jobs or scale endpoints without a significant infrastructure investment. Additionally, we often forget to switch off instances which cost us money!
Our Approach
We have three core areas that we are focused on which we believe are the most important for any infrastructure platform:
1. Performance:
We have worked hard to get our added network latency <50ms and the cold start of our average workloads to 2-4 seconds. Here are a few things we did to get our cold starts so low:
- Container Runtime: We built our own container runtime that splits container images into two parts—metadata and data blobs. Metadata provides the file structure, while the actual data blobs are fetched on-demand. This allows containers to start before the full image is downloaded. In the background, we prefetch the remaining blobs.
- Caching: Once an image is on a machine, it’s cached for future use. This makes subsequent container startups much faster. We also intelligently route requests to machines where the image is already cached.
- Efficient Inference: We route requests to the optimal machines, prioritizing low-latency and high-throughput performance. If no containers are immediately available, we efficiently queue the requests through our task scheduling system.
- Distributed Storage Cache: One of the most resource-intensive parts of AI workloads is loading models into VRAM. We use NVME drives (which are much faster than network volumes), as close as possible to the machines and we orchestrate workloads to nodes that already contain the necessary model weights where possible.
2. Developer Experience
We built Cerebrium to help developers iterate as quickly as possible by streamlining the entire build and deployment process.
To get build times as low as possible, we use high-performance machines and cache layers where possible. We've reduced first-time build times to an average of 2 minutes and 24 seconds, with subsequent builds completing in just 19 seconds.
We also offer a wide range of GPU types—over 8 different options—so you can easily test performance and cost efficiency by adjusting a single line in your configuration file.
To reduce friction, we’ve kept things simple. There are no custom Python decorators, no Cerebrium specific syntax to learn. You just add a .toml file to define your hardware requirements and environment settings. This makes migrating onto or off our platform just as easy as migrating off. We aim to impress you enough that you will want to stay.
3. Stability
This is arguably more important than the first two areas - no one wants to get an email at 11pm at night or on a Saturday that their application is down or degraded. Since April, we’ve maintained 99.999% uptime. We have redundancies in place, monitoring, alerts, and a team that covers all time zones to resolve any issues quickly.
Why Is This Hard?
Building Cerebrium has been challenging because it involves solving multiple interconnected problems. It requires optimization at every step—from efficiently splitting images to fetching data on-demand without introducing latency, handling distributed caching, optimizing our network stack, and ensuring redundancies, all while holding true to the three areas mentioned above.
Pricing:
We charge you exactly for the resources you need and only charge you when your code is running ie: usage-based. For example, if you specify you need 1 A100 GPU, with 2 CPUs and 12 GB of RAM we charge you exactly for that and not a full A100 (12 CPU’s and 148GB of memory)
You can see more about our pricing here: http://www.cerebrium.ai/pricing
What’s Next?
We're builders too, and we know how crucial support can be when you're working on something new. Here's what we've put together to support teams like yours:
- $30 in free credit to start exploring. If you're onto something interesting but need more runway, just give us a shout - we’d be happy to extend that for compelling use-cases.
- We have worked hard on our docs to make onboarding easy as well as have a very elaborate Github repo covering AI voice agents, LLM optimizations and much more.
Docs: https://docs.cerebrium.ai/cerebrium/getting-started/introduc...
Github Examples: https://github.com/CerebriumAI/examples/tree/master
If you have a question or hit a snag, you can directly reach out to the engineers who built the platform—we’re here to help! We’ve also set up Slack and Discord communities where you can connect with other creators, share experiences, ask for advice, or just chat with folks building cool things.
We're looking forward to seeing what you all build and please give us feedback on what you would like us to improve/add