Show HN: Gemma 4 Multimodal Fine-Tuner for Apple Silicon(github.com)
github.com
Show HN: Gemma 4 Multimodal Fine-Tuner for Apple Silicon
https://github.com/mattmireles/gemma-tuner-multimodal
32 comments
I run whisper large-v3 on an m2 max 96gb and even with just inference the memory gets tight on longer audio, can only imagine what fine-tuning looks like. Does the 64gb vs 96gb make a meaningful difference for gemma 4 fine-tuning or does it just push the oom wall back a bit? Been wanting to try local fine-tuning on apple silicon but the tooling gap has kept me on inference only so far.
Hey I was literally just working on this today (I was racing ahead on an audio FT myself but OP beat me by a few hours). For audio inference definitely try running your input through VAD first to drop junk data and if necessary, as one of several preprocessing steps before sending the audio to the large model. You can check out how I did it here: https://github.com/accretional/vad/blob/main/pkg/vad/vad.go
I was using https://huggingface.co/onnx-community/pyannote-segmentation-... because with ONNX, I could run it on Intel servers with vectorized instructions, locally on my Mac, AND in-browser with transformers.js
VAD is absurdly time-effective (I think like O(10s) to segment 1hr of audio or something) and reduces the false positive rate/cost of transcription and multimodal inference since you can just pass small bits of segmented audio into another model specializing in that, then encode it as text before passing it to the expensive model.
I was using https://huggingface.co/onnx-community/pyannote-segmentation-... because with ONNX, I could run it on Intel servers with vectorized instructions, locally on my Mac, AND in-browser with transformers.js
VAD is absurdly time-effective (I think like O(10s) to segment 1hr of audio or something) and reduces the false positive rate/cost of transcription and multimodal inference since you can just pass small bits of segmented audio into another model specializing in that, then encode it as text before passing it to the expensive model.
Great minds think alike!
Also, I had a huge head start, as I spent a month or two working on this in September 2025, shelved it and dusted it back off this weekend.
Also, I had a huge head start, as I spent a month or two working on this in September 2025, shelved it and dusted it back off this weekend.
Excellent work still, your repo is much more robust and fleshed out and I am just beelining straight to audio LoRa not really knowing what I'm doing, as this is my first time attempting a ~real ML training project.
I think in https://github.com/mattmireles/gemma-tuner-multimodal/blob/m... and https://github.com/mattmireles/gemma-tuner-multimodal/blob/m... and https://github.com/mattmireles/gemma-tuner-multimodal/blob/m... you have a superset of the various cludges I have in my finetuning repo, I'm going to study this and do what I can to learn from it. Really appreciate you sharing it here!
Definitely interested in swapping notes if you are though. Probably the biggest thing that came out of this exercise for us was realizing that Apple actually has some really powerful local inference/data processing tools available locally, they just are much more marketed towards application developers so a lot of them fly under the radar.
We just published https://github.com/accretional/macos-vision to make it easy for anybody to use Apple's local OCR, image segmentation, foreground-masking, facial analysis, classification, and video tracking functionality accessible via CLI and hopefully more commonly in ML and data workloads. Hopefully you or someone else can get some use of it. I definitely will from yours!
I think in https://github.com/mattmireles/gemma-tuner-multimodal/blob/m... and https://github.com/mattmireles/gemma-tuner-multimodal/blob/m... and https://github.com/mattmireles/gemma-tuner-multimodal/blob/m... you have a superset of the various cludges I have in my finetuning repo, I'm going to study this and do what I can to learn from it. Really appreciate you sharing it here!
Definitely interested in swapping notes if you are though. Probably the biggest thing that came out of this exercise for us was realizing that Apple actually has some really powerful local inference/data processing tools available locally, they just are much more marketed towards application developers so a lot of them fly under the radar.
We just published https://github.com/accretional/macos-vision to make it easy for anybody to use Apple's local OCR, image segmentation, foreground-masking, facial analysis, classification, and video tracking functionality accessible via CLI and hopefully more commonly in ML and data workloads. Hopefully you or someone else can get some use of it. I definitely will from yours!
Look inside here: https://github.com/mattmireles/gemma-tuner-multimodal/tree/m...
Here’s the trick: use Gemini Pro deep research to create “Advanced Hacker’s Field Guide for X” where X is the problem that you are trying to solve. Ask for all the known issues, common bugs, unintuitive patterns, etc. Get very detailed if you want.
Then feed that to Claude / Codex / Cursor. Basically, create a cheat sheet for your AI agents.
This will unlock a whole new level of capability.
I’m @mattmireles on Twitter — feel free to DM me.
Here’s the trick: use Gemini Pro deep research to create “Advanced Hacker’s Field Guide for X” where X is the problem that you are trying to solve. Ask for all the known issues, common bugs, unintuitive patterns, etc. Get very detailed if you want.
Then feed that to Claude / Codex / Cursor. Basically, create a cheat sheet for your AI agents.
This will unlock a whole new level of capability.
I’m @mattmireles on Twitter — feel free to DM me.
Memory usage increases quadratically with sequence length. Therefore, using shorter sequences during fine-tuning can prevent memory explosions. On my 64GB RAM machine, I'm limited to input sequences of about 2,000 tokens, considering my average output for the fine-tuning task is around 1,000 tokens (~3k tokens total).
Ah that makes sense, quadratic scaling is brutal. So with 96gb i'd probably get somewhere around 4-5k total sequence length before hitting the wall, which is still pretty limiting for anything multimodal. Do you do any gradient checkpointing or is that not worth the speed tradeoff at these sizes?
Haven’t tried yet. That’s on the do list. But good suggestion.
Shouldn't FlashAttention address the quadratic increase in memory footprint wrt.
fine-tuning/training? I'm also pretty sure that it does not apply to pure inference due to how KV-caching works.
[deleted]
re: Whisper v3 -- how is this possible? Whisper has a 30s context window. You have to chunk it.
Yeah sorry that was unclear on my part. I chunk at the endpoint level, whisper itself obviously processes 30s windows. The memory/latency thing I was referring to is more about processing longer files end to end through the pipeline, not a single whisper pass. My fastapi wrapper just splits the audio and runs chunks sequentially so total wall time scales linearly with file length, nothing fancy.
Wondering similar. It certainly can run beyond 30 seconds but at some point I believe the output should degrade
Plus you could do actual batch inference instead. Or if you must carry forward the context you could still do it linearly, but the mem usage shouldn’t just explode
Plus you could do actual batch inference instead. Or if you must carry forward the context you could still do it linearly, but the mem usage shouldn’t just explode
One thing worth mentioning for anyone loading model files from external
sources: GGUF and pickle-based formats (.bin, older .pt files) have been
attack vectors — CVE-2024-34359 was RCE via crafted GGUF in
llama_cpp_python (CVSS 9.8).
safetensors was created specifically to address this. Worth sticking to it and verifying checksums when loading community checkpoints. torch.load(..., weights_only=True) also helps for .pt files (PyTorch ≥ 1.13).
Just added a full section on this to a reference I maintain on AI toolchain attack techniques: https://github.com/XiaoYiWeio/ai-agent-attack-techniques
safetensors was created specifically to address this. Worth sticking to it and verifying checksums when loading community checkpoints. torch.load(..., weights_only=True) also helps for .pt files (PyTorch ≥ 1.13).
Just added a full section on this to a reference I maintain on AI toolchain attack techniques: https://github.com/XiaoYiWeio/ai-agent-attack-techniques
I’m pretty excited about the edge gallery ios app with gemma 4 on it but it seems like they hobbled it, not giving access to intents and you have to write custom plugins for web search, etc. Does anyone have a favorite way to run these usefully? ChatMCP works pretty well but only supports models via api.
Nice! I've been wanting to try local audio fine-tuning. Hopefully it works with music vocals too
Just a heads up, that I found NVIDIA Parakeet to be way better than Whisper - faster, uses less compute, the output is better, and there are more options for the output. I am using parakeet-mlx from the command line. Check it out!
I've been trying both Whisper v3 large and Parakeet in MacWhisper, and I inevitably go back to Whisper large. Which one is better depends on what you dictate, how you speak, and which languages you use.
yeah, it came out after I stared on my project last year. Only issue is that you can't fine-tune it on Apple Silicon.
> Accent, dialect, and low-resource language adaptation — adapt a base Gemma model to underrepresented voices and languages with your own labeled audio.
Is this for TTS? Have been looking for something to do a local fine tune to get a specific accent
Is this for TTS? Have been looking for something to do a local fine tune to get a specific accent
Thanks for doing this. Looks interesting, I'm going to check it out soon.
you are welcome! It was a fun side quest
> I had 15,000 hours of audio data
do you really need that much data for fine-tuning?
do you really need that much data for fine-tuning?
More data -> better, faster on-device models
The actual plan was to distill Gemini 2.5 Pro into the best on-device voice dictation model.
Pretty sure it would have worked. Alas.
The actual plan was to distill Gemini 2.5 Pro into the best on-device voice dictation model.
Pretty sure it would have worked. Alas.
Reasons for running local aside...
What is the practical latency difference you see between on-device and, say, whisper, in streaming mode, over the internet? Comparable? Seems that internet latency would be mostly negligible (assuming reasonable internet/cell coverage), or at least compensated for by the higher end hardware on the other side?
What is the practical latency difference you see between on-device and, say, whisper, in streaming mode, over the internet? Comparable? Seems that internet latency would be mostly negligible (assuming reasonable internet/cell coverage), or at least compensated for by the higher end hardware on the other side?
depends on the model!
If you run a smaller whisper-distil variant AND you optimize the decoder to run on Apple Neural Engine, you can get latency down to ~300ms without any backend infra.
The issue is that the smaller models tend to suck, which is why the fine-tuning is valuable.
My hypothesis is that you can distill a giant model like Gemini into a tiny distilled whisper model.
but it depends on the machina you are running, which is why local AI is a PITA.
If you run a smaller whisper-distil variant AND you optimize the decoder to run on Apple Neural Engine, you can get latency down to ~300ms without any backend infra.
The issue is that the smaller models tend to suck, which is why the fine-tuning is valuable.
My hypothesis is that you can distill a giant model like Gemini into a tiny distilled whisper model.
but it depends on the machina you are running, which is why local AI is a PITA.
This is super cool, will definitely try it out! Nice work
will it work for 32gb?
nice!
Gemma 3n came out, so I added that. Kinda went nuts, tbh.
Then I put it on the shelf.
When Gemma 4 came out a few days ago, I dusted it off, cleaned it up, broke out the Gemma part from the Whisper fine-tuning and added support for Gemma 4.
I'm presenting it for you here today to play with, fork and improve upon.
One thing I have learned so far: It's very easy to OOM when you fine-tune on longer sequences! My local Mac Studio has 64GB RAM, so I run out of memory constantly.
Anywho, given how much interest there is in Gemma 4, and frankly, the fact that you can't really do audio fine-tuning with MLX, that's really the reason this exists (in addition to my personal interest). I would have preferred to use MLX and not have had to make this, but here we are. Welcome to my little side quest.
And so I made this. I hope you have as much fun using it as I had fun making it.
-Matt