Yeah essentially this. The irony of about wanting to obscure information by submitting it to a model API isn't lost on me, but it was the easiest way I could think of. Wanted some way of making the most key content in my picture to be the only thing unblurred
I accepted the bugginess in the browser game as unavoidable, and probably had too much faith in the LLM implementations, but I did a bit more troubleshooting than mentioned. The progressive improvement over episodes (and intuitively that PPO > the others) gave me some confidence, and I've since used a similar setup on 2048 with more results showing improvement over episode: https://wandb.ai/noahpunintended/2048-raspberry-pi?nw=nwuser...
Inspired by Bill and Ted’s Excellent Adventure (https://www.youtube.com/watch?v=GiynF8NQzgo), I created an alternative to AI agents where to do a task, simply provide a future worker with some details like description, time, and place. Then assuming time travel is invented, someone will come back and do it and be paid based on the interest accrued over the original payment.
There are two models at use here, whisper tiny for transcribing audio, and then llama 3 for responding.
Whisper tiny is multi lingual (though I am using the english specific variant) and I believe llama 3 is technically capable of multi-lingual, but not sure of any benchmarks.
I think it could be made better, but for now focus is english. I'll add this to the readme though. Thanks!
I'm really inspired reading this and hope it can help! I'm planning to put more work into this. I have a few rough demos of it in action on youtube (https://www.youtube.com/watch?v=OryGVbh5JZE) which should give you an idea of the quality of it at the moment.
I think so. I plan to update the README in the coming days with more info, but realistically this is something you could run on your laptop too, barring some changes to how it accesses the microphone and camera. I assume this is the case too for other boards.
The only thing which might pose an issue is the total RAM size needed for whatever LLM is responsible for responding to you, but there's a wide variety of those available on ollama, Hugging Face, etc. that can work.
Fully depends on the model, how much conversational context you provide, but if you keep things to a bare minimum, ~< 5 seconds from message received to starting the response using Llama 3 8B. I'm also using a vision language model, https://moondream.ai/, but that takes around 45 seconds so the next idea is to take a more basic image captioning model and insert it's output into context and try to cut that time down even more.
I also tried using Vulkan, which is supposedly faster, but the times were a bit slower than normal CPU for Llama CPP.
The wake word detection is an interesting problem here. As you can see in the repo, I have a lot of mis-heard versions of the wake word in place, in this case being "Raspberry". Since the system heats up fast you need a fan, and with the microphone directly on a USB port next to the fan, I needed something distinct, and computer wasn't cutting it for this.
Changing the transcription model to something a bit better or moving the mic away from the fan could help this happen.
Additionally, since I'm streaming the LLM response, it won't take long to get your reply. Since it does it a chunk at a time, there's occasionally only parts of words that are said momentarily. Also of course depends on what model you use or what the context size is for how long you need to wait.
Hey! I'm Noah, a Full Stack & Machine Learning Engineer looking for my next step. I've been responsible for helping bring AI products to production, comfortably working with inter-disciplinary teams to get it done. Most recently worked with/working on LLMs for humanitarian needs. I'd love to find a new role with something that continues to bring ML into everyday life in responsible ways, and giving people more free time to browse websites like this :-)