Regarding 2x faster than pytorch being a condition for tinygrad to come out of alpha:
Can they/someone else give more details as to what workloads pytorch is more than 2x slower than the hardware provides? Most of the papers use standard components and I assume pytorch is already pretty performant at implementing them at 50+% of extractable performance from typical GPUs.
If they mean more esoteric stuff that requires writing custom kernels to get good performance out of the chips, then that's a different issue.
The expression f(z) = \sum_i 1/(z-\lambda_i) is called Stieltjes transform and is heavily used in random matrix theory and similar expressions are used in other works such as Batson, Spielman and Srivastava. This is all to analyze the behavior of eigenvalues which is exactly what they were trying to understand. I'd be very surprised if Aaronson doesn't know about this.
I think it is not just the library but the huge costs associated with storage, encoding and bandwidth. YouTube has innovated significantly to make it as cheap as possible to run such a service and it is likely that it would take an enormous amount of money for any competitor to replicate it.
(Disclaimer: I work at Google but no connection to YouTube)
Can they/someone else give more details as to what workloads pytorch is more than 2x slower than the hardware provides? Most of the papers use standard components and I assume pytorch is already pretty performant at implementing them at 50+% of extractable performance from typical GPUs.
If they mean more esoteric stuff that requires writing custom kernels to get good performance out of the chips, then that's a different issue.