Haha, 649 pages! Thanks for the discussion. I can understand your perspective. Maybe it would be best to add something to the blog post that discusses it from both perspectives and readers can then choose which perspective suits them better.
I should also a bit more in general about cloud computing, it seems some people agree that the post ran a bit short on that. At some point I just wanted to be done with it though — editing 10k word blog posts is not so much fun anymore!
Thank you! You have a good point. I think I would agree with you if somebody already has cloud computing skills, then the cloud is much more powerful to learn deep learning than your own GPU.
I figured that most people that start with deep learning might also lack cloud computing skills. Learning one thing at a time is easier and as such, just sticking a GPU into your desktop and focus on deep learning software / programing might yield a better experience.
I might update my blog post in the future with this detail.
Thank you, I just updated the blog post with more detailed clarification of where the data comes from.
One thing that I am quite sure of for the A100 is its transformer performance. It turns out, large transformers are so strongly bottlenecked by memory bandwidth that you can just use memory bandwidth alone to measure performance — even across GPU architectures. The error between Volta and Turning with a pure bandwidth model is less than 5%. The NVIDIA transformer A100 benchmark data shows similar scaling. So I am pretty confident on the transformer numbers.
The computer vision numbers are more dependent on the network and it is difficult to generalize across all CNNs. For example, group convolution or depth-wise separable convolution based CNNs do not scale well with better GPUs and speedups will be small (1.2 - 1.5x) whereas some other networks like ResNet get pretty straightforward improvements (1.6x-1.7x). So CNN values are less straightforward because there is more diversity between CNNs compared to transformers.
I should have been a bit clearer what went into the charts. I do not use theoretical marketing numbers but real-life benchmark data from NVIDIA and 4 other sources of benchmark between Titan V, V100, RTX 2080, RTX 2080 Ti and Titan RTX. Since I calibrate a model that needs to satisfy all sources as best as it can I think the numbers are pretty accurate.
If you put both of those benchmarks together my conclusion is quite reasonable. But I see that you could also come to your conclusion with your benchmarks. It is just a question which benchmarks are less biased and that is too difficult to evaluate.
I guess we have to wait for real data, but thanks for putting your data out there to get a discussion going.
I have a prepared draft for a blog post exactly on the topic of cloud computing and deep learning. I did not finish it as I thought that there would not be much interest in the overall question since most people will just buy GTX cards. However, it seems that there is quite a confusion going on what makes sense and what does given certain circumstances. I think I will finish that blog post now and post it in the next days.
If you want me to discuss certain questions regarding deep learning hardware and cloud computing let me know here.
Thanks for sharing your experience – this is a fair point. Often it is possible to pre-process your data and save it to disk so that you can skip this decompression/conversion/transformation step once you start training your net, but I can image applications where this is impractical or just does not work. I will add a small note to my blog about this.
These are some good points. I heard from another person that he had problems with the form factor and I will add that to the post tomorrow. I think a 2kW PSU is overkill, but you are right that more is better for PSUs.
If you want memory a good option will be to wait for the GTX Titan X which will be released in the next weeks: 12 GB RAM and it will be the fastest card by far. Overall however, I think the GTX 980 will be better for many cases still – it is just very cost effective.
If you have no desktop PC or no money for a GPU, it might be a better choice to use a EC2 instance instead of buying the hardware. You pay about $11 a week for a EC2, which is quite good once you compare it against the electricity costs that come on top of running a personal computer.
The downside is that you have a slow EC2 GPU with 4 GB RAM. Conv nets that take 3 weeks on a EC2, will take less than 2 weeks on a GTX980. If you run large conv nets, the 4 GB can be limiting (for example on ImageNet or similarly sized data sets).
Another point is that is more convenient to work on your own desktop and you can run multi-GPU nets, which is not possible on EC2 because the virtualization kills the memory bandwidth between GPUs.
If you think about it, over the long term a personal system will just be more cost efficient (you can keep a good system for years). So for deep learning researcher and those that apply deep learning this is just the most cost effective option.
A example calculation: You can buy a faster system than a EC2 for roughly $400 (GTX 580 + other parts from eBay). Together with electricity costs thats about 1 year worth of EC2, or 2 years worth if you use deep learning sporadically. A high end deep learning system will be about $1000-1400, which is about 3 years worth of EC2. So a EC2 makes good sense, if you use deep learning only sporadically and work with small data sets. If you use deep learning heavily, want a faster system or want to use multiple GPUs, a personal system will be better.
There are several cryptocurrency mining benchmarks which are bandwidth bound; those benchmarks reflect almost one to one the deep learning performance of a GPU. You can see this for example for litecoin mining:
https://litecoin.info/Mining_hardware_comparison