Integer multiplication was used to test LLMs reasoning capabilities, and I think Karpathy mentioned that tokenization might play a role in basic math. MathGLM was compared against GPT-4 in the article, but I couldn't figure out if MathGLM was trained with character-level tokenization or not.
It could be. But there's quite a bit of momentum behind CUDA. Plus, CUDA is just wicked fast. I wrote a WebGPU version of LLaMA inference and there's still a bit of a gap in performance between WebGPU and CUDA. Admittedly, WebGPU can't access tensor cores and I undoubtedly need to optimize further.