The importance of exponentially more computing power(arxiv.org)
arxiv.org
The importance of exponentially more computing power
https://arxiv.org/abs/2206.14007
23 comments
To the users of MILP libraries, the difference between algorithmic and hardware improvements are not so significant, and both can be called "Computing Power" in terms of number of application-specific solutions per second.
You're right, and your insight is illuminating into the real gains had by the world on the backs of hardware and software improvements, but I think you're one level deeper in the abstraction than the paper intends to be.
You're right, and your insight is illuminating into the real gains had by the world on the backs of hardware and software improvements, but I think you're one level deeper in the abstraction than the paper intends to be.
Then i must have phrased it badly. The fact that all the speed up would be attributed to FLOPs in the MILP case with that methodology means any improvement in data acquisition, management methodology, mathematical modeling, the ability to dig deeper or even the increased regulatory cost of boring which taken together sorta look linear when logarithmitized (for weather prediction or oil exploration (after a dubious data selection)) would also be attributed to computational improvement.
If the abstract is too dry, here's a press release interview with the lead author: https://news.mit.edu/2022/neil-thompson-computing-power-inno...
Neil has an impressive educational pedigree that positions him well for analyses like this. He's got (among other things) an undergraduate degree in physics, separate masters in Comp Sci and economics, and a PhD in Public Policy. He's smart and worth listening to. I'm excited to see him getting some professional traction: http://www.neil-t.com/about-me/
Neil has an impressive educational pedigree that positions him well for analyses like this. He's got (among other things) an undergraduate degree in physics, separate masters in Comp Sci and economics, and a PhD in Public Policy. He's smart and worth listening to. I'm excited to see him getting some professional traction: http://www.neil-t.com/about-me/
I saw a slide somewhere recently that showed that although compute per individual CPU/GPU is reaching an upper bound, the compute per dollar is still growing exponentially.
Gave me some hope that we'll still be able to do cool things in the coming decades even without some giant leap like quantum computers etc (although it's not as cool to have the compute in the cloud vs on your local machine, but it's something).
Gave me some hope that we'll still be able to do cool things in the coming decades even without some giant leap like quantum computers etc (although it's not as cool to have the compute in the cloud vs on your local machine, but it's something).
Quantum computers are not faster classical computers. They solve some problems exponentially faster than classical computers. For example we can break RSA keys with a quantum computer because we have a quantum algorithm (Shor's algorithm) to solve this problem.
We do not have many quantum algorithms that solve interesting problems exponentially faster. Finding these algorithms is not an easy task and we do not expect all problems to be solvable this way.
Disclaimer: I'm a researcher in quantum algorithms.
We do not have many quantum algorithms that solve interesting problems exponentially faster. Finding these algorithms is not an easy task and we do not expect all problems to be solvable this way.
Disclaimer: I'm a researcher in quantum algorithms.
Just to add, nobody expects NP-complete or harder problems (like the MILP example) to get any speedup form quantum computers at all.
That threat has been looming over me ever since I had my commodore 64 in the nineteen eighties. I'm sure it will hit some theoretical limit at some point in the future. But I wouldn't go as far as to predict next year for that.
Of course we have been approaching nano scale now for a while and my 2012 mac was only able to perform at 70% of the build speed of my 2017 model, which in turn is about the same as the cheap Linux laptop I picked up a few months ago. GPUs are one area where things are still improving rapidly because you increase performance by simply having more cores.
My cheap laptop is impressive in the sense that it does what it does without thermal throttling or even heating up a lot. Not bad for a cheap 700 Euro i5 laptop. My 2017 mac book pro was struggling with keeping things cool.
The next leap is going to be an exponentially larger number of CPU cores. We've been stuck at 4-16 or so for the last decade or so. There's no other reason for this other than legacy compilers, languages and CPU architectures. Leveraging concurrency is just hard. GPUs kept on increasing number of cores and have been doubling fps for the same job much more reliably.
All of course amazingly quick compared to my trusty old C64.
Of course we have been approaching nano scale now for a while and my 2012 mac was only able to perform at 70% of the build speed of my 2017 model, which in turn is about the same as the cheap Linux laptop I picked up a few months ago. GPUs are one area where things are still improving rapidly because you increase performance by simply having more cores.
My cheap laptop is impressive in the sense that it does what it does without thermal throttling or even heating up a lot. Not bad for a cheap 700 Euro i5 laptop. My 2017 mac book pro was struggling with keeping things cool.
The next leap is going to be an exponentially larger number of CPU cores. We've been stuck at 4-16 or so for the last decade or so. There's no other reason for this other than legacy compilers, languages and CPU architectures. Leveraging concurrency is just hard. GPUs kept on increasing number of cores and have been doubling fps for the same job much more reliably.
All of course amazingly quick compared to my trusty old C64.
> In this paper, we assemble direct quantitative evidence of the impact that computing power has had on five domains: two computing bellwethers (Chess and Go), and three economically important applications (weather prediction, protein folding, and oil exploration). Computing power explains 49%-94% of the performance improvements in these domains.
I'd like for a follow-up paper to look at some different domains, since these are AI research and scientific computing, respectively. Business computing is more directly relevant to economic growth, and with its use dating back to the 1950s it might be possible to find data on how it's improved.
I'd like for a follow-up paper to look at some different domains, since these are AI research and scientific computing, respectively. Business computing is more directly relevant to economic growth, and with its use dating back to the 1950s it might be possible to find data on how it's improved.
Economist Robert Solow famously said in 1987 that the computer age was everywhere except for productivity statistics. Wikipedia has a pretty good article about this Productivity Paradox.
https://en.wikipedia.org/wiki/Productivity_paradox#End_of_th...
One hypothesis I've heard that isn't on the Wikipedia page is that productivity from IT also requires new kinds of organizational structures. If you had many layers of middle management, and replaced paper memos with email, it wouldn't necessarily improve productivity because that wouldn't harness the full potential of computing.
https://en.wikipedia.org/wiki/Productivity_paradox#End_of_th...
One hypothesis I've heard that isn't on the Wikipedia page is that productivity from IT also requires new kinds of organizational structures. If you had many layers of middle management, and replaced paper memos with email, it wouldn't necessarily improve productivity because that wouldn't harness the full potential of computing.
It's well studied subject. There are also surveys available.
Here is slate article: https://slate.com/business/2011/03/the-productivity-paradox-...
1) effect has been surprisingly small. It's often called the "productivity paradox" Robert Solow said it in late 80s: “You can see the computer age everywhere but in the productivity statistics.”
2) The reasons are complex.
One possible reason is that productivity increase -> labor share declines -> less demand -> smaller GDP growth.
People always incorrectly assume that unemployment is the danger from automation. The real issue is declining labor share. People with jobs get smaller share of the work they do, because computers (capital asset) is replacing labor.
1) effect has been surprisingly small. It's often called the "productivity paradox" Robert Solow said it in late 80s: “You can see the computer age everywhere but in the productivity statistics.”
2) The reasons are complex.
One possible reason is that productivity increase -> labor share declines -> less demand -> smaller GDP growth.
People always incorrectly assume that unemployment is the danger from automation. The real issue is declining labor share. People with jobs get smaller share of the work they do, because computers (capital asset) is replacing labor.
I wonder if anybody measured something similar to that number on the 18th century.
Related to weather, climate science would benefit greatly from increased computational power. Current models are forced to make assumptions to predict certain processes below the grid resolution. Things like cloud processes and transport processes are usually parameterized. Similarly, computational fluid dynamics are very useful to studying climate and are limited by the available computational power.
>> we find that an exponential increase in computing power is needed to get linear improvements in these outcomes.
The scaling for weather models and FEA is not linear. To get incrementally better requires at least polynomial increases.
The scaling for weather models and FEA is not linear. To get incrementally better requires at least polynomial increases.
"weather models" "better" - how do you actually rate weather models? Exact prediction or prediction of outliers like catastrophic events? Is there more value in knowing tomorrow's temperature to be 24C or a thunderstorm happening at 15:00 ?
There is a lot of value in predicting where a hurricane is going to hit within 100km in 36 hours. Also things like flood prediction so we can drain lakes ahead of time before the waters start rising. Giving people a forewarning saves lives.
where no algebraic, and certainly beautiful solution(s), of some set of equations exists, brute force computation to the rescue!
The more computing power, the more accurate the predictions.
The more computing power, the more accurate the predictions.
Just wrong. The more correct input you have and the more correct your model of the real world is, the more accurate your predictions are.
We already see helpless activities to be better with AI (and a lot of computing) then classical optimisation theory (think solving PDEs) and failing.
Just more computing power will not help (except people whose business model relies on selling you computing power)
We already see helpless activities to be better with AI (and a lot of computing) then classical optimisation theory (think solving PDEs) and failing.
Just more computing power will not help (except people whose business model relies on selling you computing power)
Let's say that the messy problems that don't have a nice analytic solution (a beautiful all encompassing model) are more and more leaning on 'very complex systems of equations or inequations, linear or non linear' and we can only approach or simplify them to heuristics (gradient descent, optimization of all sorts) because exact solution is too compute-expensive.
A tenfold difference in power might help a bit (being able to take 10x more sensors in or data with 10x better resolution might already help).
But a 10000x increase is a game changer, even if you waste so much of it. Though it all depends how it increases... Some processes I know would be immensely better - state of the art has already quantified the huge gains, and then spent 15 years trying to make the thing runnable in real-time - if I could run hundreds of millions of complex 400x400 SVDs per second or if I could run a maximum likelihood search at the same rhythm).
A 10000x increase in perf puts some exact best answer and not just approximations or 'best we could compute, sorry' in the realm of possibilities.
I'd also say AI itself would not be viable without the huge increase in power and memory bandwidth that allows to play SGD on such a huge search space.
You need months or 200+ DGX servers (8xA100 each) and a team of mlops people babysitting the process, to train a large language model today. Isn't that a direct result of computing power increase?
I'll take any more computing power, especially if there's a 10x or 100x gain on the horizon, if they'll sell it to me.
Yes we need to be clever and not waste the computing power, but for many as-yet-not-over-optimized problems any increase is an instant win.
A tenfold difference in power might help a bit (being able to take 10x more sensors in or data with 10x better resolution might already help).
But a 10000x increase is a game changer, even if you waste so much of it. Though it all depends how it increases... Some processes I know would be immensely better - state of the art has already quantified the huge gains, and then spent 15 years trying to make the thing runnable in real-time - if I could run hundreds of millions of complex 400x400 SVDs per second or if I could run a maximum likelihood search at the same rhythm).
A 10000x increase in perf puts some exact best answer and not just approximations or 'best we could compute, sorry' in the realm of possibilities.
I'd also say AI itself would not be viable without the huge increase in power and memory bandwidth that allows to play SGD on such a huge search space.
You need months or 200+ DGX servers (8xA100 each) and a team of mlops people babysitting the process, to train a large language model today. Isn't that a direct result of computing power increase?
I'll take any more computing power, especially if there's a 10x or 100x gain on the horizon, if they'll sell it to me.
Yes we need to be clever and not waste the computing power, but for many as-yet-not-over-optimized problems any increase is an instant win.
More computing power can be a prerequisite to more accurate models. E.g weather models modelling things at smaller granularity.
Surely this is showing correlation, but not necessarily causation. There are probably many fields where improvements in X have happened, along with greater compute power. How much the improvement in X is attributable to greater computing power is surely very hard to quantify. Or am I missing something?
I agree with you.
For the oil example, I can think of a very simple explanation that has nothing to do with computing power. They state that the drilling success rate was 10% in 1940 and 70% in 2010. In other words, out of 10 exploratory drills, 9 were dry in 1940 and only 3 in 2010. And since the companies use computers to predict the presence of oil, and the computing power has increased, voila, the computing power explains the increased success.
But the alternative explanation is that over time oil could be found at higher and higher depths. The cost of exploration has increased. When it does not cost you much to drill a shallow hole (in 1940), you don't mind if you miss 9 times out of 10. But if you have to put in millions of dollars to drill to thousands of meters depths, then you think twice (or maybe 10 times) before you drill.
Any prediction system will give you an estimated probability of success. If the cost of failure is not high, you may decide to drill as soon as p > 10%. But if it's high, you may increase the threshold to 90%.
Of course, I'm not saying this is all there is to it. But it's likely that this explained part of the increased success. Another part is no doubt due to more experience. Another is due to algorithmic advances. Yet another to the improvement in the seismic sensors and technology used.
For the oil example, I can think of a very simple explanation that has nothing to do with computing power. They state that the drilling success rate was 10% in 1940 and 70% in 2010. In other words, out of 10 exploratory drills, 9 were dry in 1940 and only 3 in 2010. And since the companies use computers to predict the presence of oil, and the computing power has increased, voila, the computing power explains the increased success.
But the alternative explanation is that over time oil could be found at higher and higher depths. The cost of exploration has increased. When it does not cost you much to drill a shallow hole (in 1940), you don't mind if you miss 9 times out of 10. But if you have to put in millions of dollars to drill to thousands of meters depths, then you think twice (or maybe 10 times) before you drill.
Any prediction system will give you an estimated probability of success. If the cost of failure is not high, you may decide to drill as soon as p > 10%. But if it's high, you may increase the threshold to 90%.
Of course, I'm not saying this is all there is to it. But it's likely that this explained part of the increased success. Another part is no doubt due to more experience. Another is due to algorithmic advances. Yet another to the improvement in the seismic sensors and technology used.
> Yet another to the improvement in the seismic sensors and technology used.
This is a huge one. Almost all of that geoseismic downhole sensing technology uses radioactive emissions. In 1940, it was impossible to "see" anything downhole with any reliability. Nowadays it's passe.
This is a huge one. Almost all of that geoseismic downhole sensing technology uses radioactive emissions. In 1940, it was impossible to "see" anything downhole with any reliability. Nowadays it's passe.
No, they pick a set of problems where computational methods are known to have a beneficial impact and the plot every progress in that field against increased amounts of computing. Since amount of computing power used is monotonous and ELO score/Go performance/weather prediction success is trending monotonous the correlation is pretty high. However computation power is not the only thing that rose mostly monotonically during that time. At best they derived an upper bound of the contribution of more computing power to better outcomes.
For example in Mixed Integer Linear Programming studies were done to measure algorithmic vs hardware speedup. "On average, we found out that for solving LP/MILP, computer hardware got about 20 times faster, and the algorithms improved by a factor of about nine for LP and around 50 for MILP, which gives a total speed-up of about 180 and 1,000 times, respectively." https://arxiv.org/abs/2206.09787 This methodology would attribute the 1000 times effect to the increase in FLOPs alone.
And just a methodological concern, taking the logarithm of one axis, is applying a non-linear transformation, and then doing a linear fit results in distorted measure of distance between fit and data depending on the data. This effect was not discussed. It does only mess with R value so i would not feel comfortable applying that R value to derive an attribution.