I've built a product in this regard - specifically for fine-tuning and deploying said fine-tuned models.
You'll need GPUs for inferencing + have to quantize the model + have it hosted on the cloud. The platform I've built is around the same workflow (but all of it is automated, along with autoscaling, and you get an API endpoint; you only pay for the compute you host on).
Generally, the GPU(s) you choose will depend on how big the model is + how many tokens/sec you're looking to get out of it.
I'm a tech founder who has raised money from VCs in India. For some reason, I feel like I know the 'syndicate' you're talking about. If you're okay sharing, are they Mumbai-based?
After running my co. for over half a decade, I wouldn't touch these investors with a 10-ft pole. "Helping you raise" does not justify the additional equity being sought.
If your friend isn't in dire need of money, I'd recommend avoiding being invested in by these investors and instead, only going after Tier-1 VC firms.
Hey Christoph, thanks for trying it out - we're running this on the cloud, particularly GCP, on A100s (80g).
On your query about running these models locally, I'm not sure if just upgrading your RAM would have the same throughput as what you see on the website. You can upgrade your RAM but you might get pretty bad tokens/sec.
I've been using Otter AI for almost a year now. They've added quite a few features with querying your transcription and the likes, and it just works.
Specifically on language, it sometimes goes haywire when there is Hinglish (Hindi + English) involved, so not sure if they support de natively. Cost-effective though.
Yeah - It's at par, if not better than GPT 3.5! This is the base model (13B) with no fine-tuning or censorship.
Feel free to give it a spin for code as well! We're just the infrastructure layer here so we don't use any data for retraining these models. LLaMA 2 70B coming soon! :D
Hey hey! We have deployed this on our cloud. It’s running on 2 A10Gs on AWS in the background.
We had the tech from our MLOps platform NimbleBox.ai that let us setup a managed service on all major cloud providers so we just frankenstein-ed it to work for LLMs as well :)
The prompt engineering, specially for web search, is powered by our open-source tool ChainFury (https://chainfury.nbox.ai/)
I believe it's the way OpenAI has positioned its model.
By itself, the model follows a straightforward and concise 'answer the question' approach. However, if you want it to be creative, scientific, or empathetic, you can engineer your prompts accordingly (bear in mind, it may spew out literal garbage while trying to answer certain advanced questions scientifically).
It can also enact prominent figures so that the responses are tailored according to how the actual person would answer.
You'll need GPUs for inferencing + have to quantize the model + have it hosted on the cloud. The platform I've built is around the same workflow (but all of it is automated, along with autoscaling, and you get an API endpoint; you only pay for the compute you host on).
Generally, the GPU(s) you choose will depend on how big the model is + how many tokens/sec you're looking to get out of it.