I've built a product in this regard - specifically for fine-tuning and deploying said fine-tuned models.
You'll need GPUs for inferencing + have to quantize the model + have it hosted on the cloud. The platform I've built is around the same workflow (but all of it is automated, along with autoscaling, and you get an API endpoint; you only pay for the compute you host on).
Generally, the GPU(s) you choose will depend on how big the model is + how many tokens/sec you're looking to get out of it.
I'm a tech founder who has raised money from VCs in India. For some reason, I feel like I know the 'syndicate' you're talking about. If you're okay sharing, are they Mumbai-based?
After running my co. for over half a decade, I wouldn't touch these investors with a 10-ft pole. "Helping you raise" does not justify the additional equity being sought.
If your friend isn't in dire need of money, I'd recommend avoiding being invested in by these investors and instead, only going after Tier-1 VC firms.
Hey Christoph, thanks for trying it out - we're running this on the cloud, particularly GCP, on A100s (80g).
On your query about running these models locally, I'm not sure if just upgrading your RAM would have the same throughput as what you see on the website. You can upgrade your RAM but you might get pretty bad tokens/sec.
I've been using Otter AI for almost a year now. They've added quite a few features with querying your transcription and the likes, and it just works.
Specifically on language, it sometimes goes haywire when there is Hinglish (Hindi + English) involved, so not sure if they support de natively. Cost-effective though.
You'll need GPUs for inferencing + have to quantize the model + have it hosted on the cloud. The platform I've built is around the same workflow (but all of it is automated, along with autoscaling, and you get an API endpoint; you only pay for the compute you host on).
Generally, the GPU(s) you choose will depend on how big the model is + how many tokens/sec you're looking to get out of it.