This article is AI generated, and they didn't even fact check it.
An AI module like this can help a lot with processing for certain types of neural networks, but LLMs are not one of them.
LLM inference is basically bottlenecked by RAM bandwidth and how much RAM you have. Every token to be generated needs to iterate over the whole model, pulling it piece by piece from the RAM to the CPU, where some relatively small calculations are applied.
Having a separate NPU like this connected via PCIE makes LLMs much slower, since you're bottlenecked by a PCIE 3.0 x1 connection instead of your full memory bandwidth.
There are some magic environment variables you want to set to get ROCM to work with this technically unsupported APU:
HSA_OVERRIDE_GFX_VERSION=9.0.0
HSA_ENABLE_SDMA=0
Performance is not great, but slightly better than running inference on the CPU, with the bonus that your CPU is essentially free for other tasks even while running LLMs.
I love the MBP 2015. This MacBook model actually supports MacOS Monterey natively. That's three editions newer than Mojave. It just requires an OS reinstall. After you backup your files, you can reboot it while holding ALT+CMD+R to start online recovery mode. That will let you download and install the latest MacOS version the hardware supports.
LLM inference is basically bottlenecked by RAM bandwidth and how much RAM you have. Every token to be generated needs to iterate over the whole model, pulling it piece by piece from the RAM to the CPU, where some relatively small calculations are applied.
Having a separate NPU like this connected via PCIE makes LLMs much slower, since you're bottlenecked by a PCIE 3.0 x1 connection instead of your full memory bandwidth.