I can agree with the skeptics that LLM generated code is usually crap. I rarely accept its output without significant edits unless it's truly boilerplate, and I want to avoid the need for that kind of code in the first place.
For me, the killer use case is debugging. I hate wasting time debugging something that should work except for mistakes, and now I do that probably 75% less than I used to because AI does it for me.
I don't know if it makes me that much more productive, but I certainly enjoy my work more not having to do as much tedious debugging, and it feels like I waste a lot less time doing it.
Maybe you want a comparison anyways, but it won't be competitive. On Apple CPUs, SME is ~8x faster than a single regular CPU core with a good BLAS library.
The problem I've seen is this: in order to get good performance, no matter what language you use, you need to understand the hardware and how to use the instructions you want to use. It's not enough to know that you want to use tensor cores or whatever, you also need to understand the myriad low level requirements they have.
Most people that know this kind of thing don't get much value out of using a high level language to do it, and it's a huge risk because if the language fails to generate something that you want, you're stuck until a compiler team fixes and ships a patch which could take weeks or months. Even extremely fast bug fixes are still extremely slow on the timescales people want to work on.
I've spent a lot of my career trying to make high level languages for performance work well, and I've basically decided that the sweet spot for me is C++ templates: I can get the compiler to generate a lot of good code concisely, and when it fails the escape hatch of just writing some architecture specific intrinsics is right there whenever it is needed.
For me, the killer use case is debugging. I hate wasting time debugging something that should work except for mistakes, and now I do that probably 75% less than I used to because AI does it for me.
I don't know if it makes me that much more productive, but I certainly enjoy my work more not having to do as much tedious debugging, and it feels like I waste a lot less time doing it.