90T/s on my iPhone llama3.2-1B-fp16(reddit.com)5 points·by darkolorin·anno scorso·1 commentsreddit.com90T/s on my iPhone llama3.2-1B-fp16https://www.reddit.com/r/LocalLLaMA/s/DscuuFbgwc1 commentsPost comment[–]darkolorin·anno scorsoreplyI made it! 90 t/s on my iPhone with llama1b fp16We completely rewrite the inference engine and did some tricks. This is a summarization with llama 3.2 1b float16. So most of the times we do much faster than MLX. lmk in comments if you wanna test the inference and I’ll post a link.
We completely rewrite the inference engine and did some tricks. This is a summarization with llama 3.2 1b float16. So most of the times we do much faster than MLX. lmk in comments if you wanna test the inference and I’ll post a link.