I don't think this is entirely true. (Although please let me know if I misunderstood your comment or if I'm wrong.) While it is true that CPUs' power usage increased significantly between the 1980s and 2006, I believe you were able to get much better performance "in the same thermal and power envelope" during that time period. There's two conflicting factors that made these energy efficiency gains less apparent in the market: power efficiency and power usage. Roughly until 2006, both Dennard Scaling and Moore's Law held, so you got ~2x performance efficiency (performance per watt/performance in the same thermal and power envelope) every 1.5 years. [1,2] Power consumption also increased at the same time [3], so you got both more performance in the same thermal and power envelope (useful for mobile/embedded chips) and greater power consumption (useful for desktop/server chips).
Thank you for the tips! Whenever I hear about new accelerators, my first question is: "how do people in the real world run fast code on this"? Because (related to your above points), you don't use an accelerator for things to just run, they have to run fast. Otherwise, you'd just use a CPU.
Detailed Question 1: do you have examples of fusion making things hard? Is there a way to nudge the compiler to not fuse or create a symbol table tracking fusion? Does fusion cause issues with the name_scopes?
Detailed Question 2: isn't Pytorch eager execution? Do you know how it compares to Tensorflow's eager execution?
General Request: I'm in a more theoretical position, writing papers on programming languages for accelerators https://aetherling.org/ and TAing courses on accelerators http://cs149.stanford.edu/fall20. So, I'm excited to see people's practical experiences using these accelerators in industry. It would be very enlightening (if you have time) to write up a comparison of tuning a model for an A100's tensor cores vs a TPU. This seems like the key trade-off in comparing architectures.
The article states that the programming model is TensorFlow or PyTorch. However, I'm not clear on the details of customizing TensorFlow or PyTorch code to run efficiently on this chip.
Does anyone here have firsthand experience using the compiler? Can you give a rough approximation of performance tuning with the compiler compared to performance tuning with compilers targeting the tensor cores on an A100 or a TPU?
The closest I came to an SGI machine was an N64, so I would be interested in hearing the stories. What did consolidating the SGI boards get you? Didn't that cause thermal issues?
Are Presidio and "the death star" you referenced above related to RenderMan? I've done research on hardware for image processing (https://aetherling.org/), so it's interesting to hear about these systems.
1. https://en.wikipedia.org/wiki/Dennard_scaling#Relation_with_...
2. https://iscaconf.org/isca2018/docs/HennessyPattersonTuringLe... - slide 25
3. http://www.edwardbosworth.com/My5155_Slides/Chapter01/ThePow... - "Here is a Clue to the Problem" graph