i also had a similar 'hello world' experience some time ago with [0] :). i manually used some SIMD instructions, and it seems the performance could align with llama.cpp. it appears that the key to performance is:
1. using SIMD on quantized matrix multiplication
2. using a busy loop instead of condition variables when splitting work among threads.
(however, i haven't had more free time to continue working on inferencing quantized models on GPU (with Vulkan), and it hasn't been updated for a long time since then.)
1. using SIMD on quantized matrix multiplication 2. using a busy loop instead of condition variables when splitting work among threads.
(however, i haven't had more free time to continue working on inferencing quantized models on GPU (with Vulkan), and it hasn't been updated for a long time since then.)
[0] https://github.com/crabml/crabml