Trying the fuse feature, seemed the most interesting:
> spaghetti ≈ trumpet
> Both spaghetti and a trumpet can be difficult to eat without making a mess—spaghetti with its long, slippery noodles, and a trumpet with its wide, flared bell.
Nvidia's enterprise GPUs are surprisingly unreliable. Working on a 128 GPU A100 cluster on AWS, 1 would fail every few days. I didn't have any insight on whether it was a hardware or software failure.
You also need the optimizer (e.g. Adam)'s state, which is usually double the parameter's size. So if using fp16, one parameter takes up 6 bytes in memory.
No it won't. Large language models are trained on 1,000 - 50,000 GPUs. No one's going to buy hundreds of Mac pros to mount them in a datacenter for training ML models.