Helsing and Mistral announce strategic partnership in defence AIhelsing.ai3 points·by heinrichf·ปีที่แล้ว·0 comments
heinrichf·ปีที่แล้ว·discussI'm comparing Gemma3 12 B (https://ollama.com/library/gemma3; running fully on my 3060 12GB) and Mistral Small 3 24B (https://ollama.com/library/mistral-small; 10% offloaded to the CPU).- Gemma3 12B: ~100 t/s on prompt eval; 15 t/s on eval- MistralSmall3 24B: ~500 t/s on prompt eval; 10 t/s on evalDo you know what different in architecture could make the prompt eval (prefill) so much slower on the 2x smaller Gemma3 model?
heinrichf·2 ปีที่แล้ว·discussRestic has a mount subcommand that exposes all backups through a FUSE filesystem, no?
heinrichf·2 ปีที่แล้ว·discussThere is also an alternative more lightweight self-hosted server in Rust, compatible with the official clients: https://github.com/cpg314/ltapiserv-rs
- Gemma3 12B: ~100 t/s on prompt eval; 15 t/s on eval
- MistralSmall3 24B: ~500 t/s on prompt eval; 10 t/s on eval
Do you know what different in architecture could make the prompt eval (prefill) so much slower on the 2x smaller Gemma3 model?