For now it is CPU only yes, uses AVX instructions. But it's pretty fast anyway, try it out. I have it running on my mbp M1 and it's pretty decent. I think GPU support will come eventually. I wrote an app that uses the openai API and it was nice and simple to just point it at my own local service instead.
LocalAI is the OpenAI compatible API that lets you run AI models locally on your own CPU! Data never leaves your machine! No need for expensive cloud services or GPUs, LocalAI uses llama.cpp and ggml to power your AI projects!
LocalAI supports multiple models backends (such as Alpaca, Cerebras, GPT4ALL-J and StableLM) and works seamlessly with OpenAI API. Join the LocalAI community today and unleash your creativity!
I took this to extremes last year: I used it to run MAAS from Australia to sweden (which requires layer 2). Granted I used tailscale to make the WireGuard part even easier, but it was a lot of fun.
I starred this awhile back, it’s the really hard way imo, really great for learning: https://github.com/ghik/kubernetes-the-harder-way