Wow, this sounds really cool! Out of curiosity, did you get bad results with the pure PGF backend? (And can you link to your script by any chance?) I'm always amazed that including matplotlib plots in LaTeX documents is so fraught since it's such common use case.
Also I read that matplotlib 3.5 has some sort of improved support for type 42 subsetting. I haven't had a chance to try it out yet but this could be a welcome improvement!
Perhaps surprisingly, the matrix logarithm does not satisfy log(A) + log(B) = log(AB) in general (only when AB = BA) which is why you cannot use it to "multiply by adding".
Also I read that matplotlib 3.5 has some sort of improved support for type 42 subsetting. I haven't had a chance to try it out yet but this could be a welcome improvement!