It's as difficult as a serverless provider to grow as it was for CPUs before GPUs came along.
Many companies overinvest in fully-owned hardware, rather than renting from clouds. Owning hardware means you underwrite unrented inventory costs and prevents you from scaling. H100 pricing is now lower than any self-hosted option, even without factoring the TCO & headcount.
(Disclaimer: I work at a GPU cloud Voltage Park -- with 24k H100s as low as $2.25/hr [0] -- but Fly.io is not the only one I've noticed purchase hardware when renting might have saved some $$$)
Absolutely - GPUs are definitely not a very liquid asset. As someone who works at a GPU neocloud provider (Voltage Park), server assets at scale definitely face a huge slippage, you can buy for $1 and get quotes for $1.50 but only be able to sell for $0.60
Congrats on the launch! This is huge, and it's really cool to see a cloud provider moving this direction - auction pricing for customers so that you always know you're getting the best deal on the market, while providing 100% utilization for you :)
I'm curious what some of the numbers mean: e.g. what does 688/1464 GPUs available indicate in the left gray box? What about there being 1040 GPUs in light gray, and 8 in dark gray?
Hmm I did include a training workload as the second chart. My test workload was relatively small so I guess if the workload I ran spends a bit less GPU time comparatively to the CPU, given equal CPU for all workloads, would be an equalizing factor.
But even looking at the Lambda Labs benchmarks, I am surprised that the H100 PCIE barely outperforms the A100 SXM, for example. And it is meant to be a replacement for the A100 PCIE. 20% generational improvement yes, but I would have expected more?
We do have a managed container hosting service, but it's built on our own backend that auto-scales nodes for you when average GPU utilization surpasses a certain point, but it's not K8s -- which would have been a pain to configure given the distributed nature of all of our servers.
We have our own supply base (sourced through https://tensordock.com/host), operate some of our own servers, and are not related to any other marketplace :)
We think we have better security & reliability than Vast.ai due to virtualization rather than Dockerization, as well as more strict access controls [1]. Additionally, you can run Windows VMs on us if you want :)
Jonathan from TensorDock (https://tensordock.com/) here - we listed two of our A100 and H100 clusters on the site.
The IB equipped on our clusters (can't speak to others) is 8x 400 Gbps. Most customers training foundational models are able to fully utilize that fabric in parallel.
I think if you offer to sell to them, they will probably give you a lowball offer.
Instead, maybe consider trying to grow it on your own, see where you can take it, and if it shows some reasonable traction, a big player would probably definitely offer to swallow you up at a good offer so that they get the better technology that's already been demonstrated to work in a production environment.
Happy to give you whatever credits you need (would $25 be enough?) to try running some cool stuff on our infrastructure. And feel free to email me the results of whatever you do at [email protected] :)
Yeah, feel free to shoot me an email for like $25 of credit. For startups, we usually do $100 (just send a request from your startup email address).
Even with $5 credits, we had people abusing the system and creating multiple accounts to cryptocurrency mine. So we chose $1 as an amount that would be so low that nobody would want to take advantage of, haha :)
Whoops, apologies for missing this! For our core cloud product, we only partner with established providers. Large-scale compute wholesalers with $5m+ of compute each in secure data centers. These companies' entire businesses are built on selling secure compute to customers like us and other medium/large businesses. Basically, this isn't some random dedicated server host off of LowEndTalk :)
We have data protection agreements with all of them, and we can also do bare metal machines on request so that you have full control over your physical machine.
Absolutely! You can use our Windows 10 instances and cloud game (like how airGPU sells cloud gaming containers on us), render blender scenes, or do CAD on remote desktops. These are dedicated virtual machines for you to use :)
Many companies overinvest in fully-owned hardware, rather than renting from clouds. Owning hardware means you underwrite unrented inventory costs and prevents you from scaling. H100 pricing is now lower than any self-hosted option, even without factoring the TCO & headcount.
(Disclaimer: I work at a GPU cloud Voltage Park -- with 24k H100s as low as $2.25/hr [0] -- but Fly.io is not the only one I've noticed purchase hardware when renting might have saved some $$$)
[0] https://dashboard.voltagepark.com/