PyTorch 1.13(pytorch.org)
pytorch.org
PyTorch 1.13
https://pytorch.org/blog/PyTorch-1.13-release/
19 comments
I'm curious about PyTorch but never really found a problem that it would be useful for.. Any ideas?
It's used for deep learning.
> Inspired by Google® JAX, functorch is a library that offers composable vmap (vectorization) and autodiff transforms. It enables advanced autodiff use cases that would otherwise be tricky to express in PyTorch
It's very nice that they're open about borrowing ideas that they like from other projects.
It's very nice that they're open about borrowing ideas that they like from other projects.
is anyone able to comment on the experience of PyTorch's M1 support?
https://pytorch.org/docs/stable/notes/mps.html
In my very limited experience it's not a ton faster than using a CPU backend though.
In my very limited experience it's not a ton faster than using a CPU backend though.
Most things require workarounds, some things aren't possible (or we haven't found workaround yet) and it's not as fast as CUDA. But stable-diffusion inference works, and so does textual inversion training. I was also able to run training of a T5 model with just a couple of tweaks.
I'd stick with PyTorch 1.12.1 for now. 1.13 has problems with backpropagation (I get NaN gradients now when I attempt CLIP-guided diffusion -- I think this applies to training too), and some einsum formulations are 50% slower (there is a patch to fix this; I expect it'll be merged soon), making big self-attention matmuls slow and consequently making stable-diffusion inference ~6% slower.
I'd stick with PyTorch 1.12.1 for now. 1.13 has problems with backpropagation (I get NaN gradients now when I attempt CLIP-guided diffusion -- I think this applies to training too), and some einsum formulations are 50% slower (there is a patch to fix this; I expect it'll be merged soon), making big self-attention matmuls slow and consequently making stable-diffusion inference ~6% slower.
It doesn’t support a few things, and has bugs in a few of the functions but besides that it works OK. Need to make sure your batch size is high enough to fully utilise the GPU cores. I think it was maybe 2-4 times faster than the CPU in my case.
If you use miniforge (https://github.com/conda-forge/miniforge) you can just 'pip install torch' and it works.
I think you mean “conda install torch”, no? Miniforge would only impact the conda installer?
Surprisingly, no. I just tried it:
- 'pip install torch' doesn't work inside of a plain Python env (not miniforge)
- 'conda install torch' doesn't work inside miniforge
- 'pip install torch' works inside miniforge
- 'pip install torch' doesn't work inside of a plain Python env (not miniforge)
- 'conda install torch' doesn't work inside miniforge
- 'pip install torch' works inside miniforge
Huh! That’s curious! I wonder what causes the different behaviour!
What's different about miniforge in this regard? Does miniforge come with additional package indexes pre-configured?
Pretty much. Conda installed with miniforge uses a different channel, with a focus on having packages available for more architectures.
I've used it for Stable Diffusion on my M1 mbp, works with no problem. I am not a AI/ML person so my experience with M1 PyTorch is limited to just using SD.
Yeah works great, Tensorflow works great as well, but if using pip need to install tensorflow-metal and tensorflow-macos. Plus I always need to downgrade protobuf to 3.20. And couldn't get the C bindings for Tensorflow to work through metal, only through CPU.
Pretty good machine learning experience on M1.
Pretty good machine learning experience on M1.