Show HN: Inference Mixtral 8x7B in pure Rust(github.com)
github.com
Show HN: Inference Mixtral 8x7B in pure Rust
https://github.com/moritztng/mixtral
2 comments
It’s a very barebones link, but I’m curious. Is this CPU or GPU? What are some of the salient features of this particular project besides the language?
hi, it's cpu, makes full use of simd on multiple cores and i got 3 tok/s. it's also memory mapped, so it stays in ram once it's loaded. the weights are 8 bit quantized. it can run as a server and is the backend of https://x.com/paulpapers - a x bot, that reads all llm arxiv papers and shares the ones that are about running llms more efficiently.