Previously, I co-founded SpeakerText (speakertext.com), a interactive/social layer on top of online video, and Humanoid (gethumanoid.com), a crowdsourced virtual labor service that uses machine learning to track worker reputation on Mechanical Turk and ensure quality.
In previous lives, I fought forest fires in Montana & Alaska for the US Forest Service, worked as a freelance news reporter for the NY Times, and drove a 911 ambulance in Harlem and the South Bronx.
Dr. Alex Mathiasen, PhD wants to build a pause button for human biology – so that you can live forever. All he needs is enough GPUs to simulate quantum physics.
If you run a smaller whisper-distil variant AND you optimize the decoder to run on Apple Neural Engine, you can get latency down to ~300ms without any backend infra.
The issue is that the smaller models tend to suck, which is why the fine-tuning is valuable.
My hypothesis is that you can distill a giant model like Gemini into a tiny distilled whisper model.
but it depends on the machina you are running, which is why local AI is a PITA.
Here’s the trick: use Gemini Pro deep research to create “Advanced Hacker’s Field Guide for X” where X is the problem that you are trying to solve. Ask for all the known issues, common bugs, unintuitive patterns, etc. Get very detailed if you want.
Then feed that to Claude / Codex / Cursor. Basically, create a cheat sheet for your AI agents.
Memory usage increases quadratically with sequence length. Therefore, using shorter sequences during fine-tuning can prevent memory explosions. On my 64GB RAM machine, I'm limited to input sequences of about 2,000 tokens, considering my average output for the fine-tuning task is around 1,000 tokens (~3k tokens total).
Previously, I co-founded SpeakerText (speakertext.com), a interactive/social layer on top of online video, and Humanoid (gethumanoid.com), a crowdsourced virtual labor service that uses machine learning to track worker reputation on Mechanical Turk and ensure quality.
In previous lives, I fought forest fires in Montana & Alaska for the US Forest Service, worked as a freelance news reporter for the NY Times, and drove a 911 ambulance in Harlem and the South Bronx.