Show HN: Chat with Zephyr 7B over voice message using whisper.cpp and llama.cpp(bastianai.com)
bastianai.com
Show HN: Chat with Zephyr 7B over voice message using whisper.cpp and llama.cpp
https://bastianai.com/blog/2023/10/chat-with-self-hosted-llm-via-discord-voice-message-transcription
Just my small weekend project, experimenting with LLMs. I made a Discord bot that uses the Zephyr 7B model to reply, and whisper.cpp to transcribe voice messages into text. So you can send it voice messages and it replies.
3 comments
This looks good, thanks for sharing. Can you share a little more about where the compute is performed, the transcription and llm? Its a local machine or a machine in cloud? If its local, i assume it cant be used by other people in server? If its on a network/cloud machine, how is the communication handled between discord and the machine? Also what modifications need to be done to do longer streams of audio data?
I'm running everything on my own PC. With a 3060 12 GB at under $300, it's very cheap to build a system capable of running inference with smaller models.
But even bigger models (13B or 30B) can be handled with a dual GPU setup (24 GB total memory at < $600), with quantization.
The Discord bot works as a client (outgoing connection to the Discord public server), so it can run from behind NAT.
But even bigger models (13B or 30B) can be handled with a dual GPU setup (24 GB total memory at < $600), with quantization.
The Discord bot works as a client (outgoing connection to the Discord public server), so it can run from behind NAT.
However if one wants to run something like this in production, it would make more sense to use a server with at least an A10, which has better performance per power consumption.