Zebras hate you for no reason: Why Amdahl's law is misleading (2017)(embeddedrelated.com)
embeddedrelated.com
Zebras hate you for no reason: Why Amdahl's law is misleading (2017)
https://www.EmbeddedRelated.com/showarticle/1033.php
45 comments
The other issue he didn’t talk about was how bad we are at fractions, or what I usually call budget based tuning.
What I’ve seen over and over is people who won’t look at single digit percentage improvements because those aren’t “enough”. But the thing is that if you are trying to cut time in half, the 6% for that one method is 12% of your target. So they’re optimizing the way some people do personal finances.
Some of them get “there” in the end, but the road they take is longer and ineffective. When you are looking at four slow methods in a particular flow, you have the perspective of seeing them as parts of a whole. Maybe there’s a way you can rewrite these 4 methods into three that do less, or five where we avoid work in the common case.
If you do them one at a time, and prioritize new ones as they cross your artificial and inaccurate threshold. you won’t see these options, and if you do you’ll talk yourself out of it because you already spent a bunch of effort on what you have now (sunk cost).
What I’ve seen over and over is people who won’t look at single digit percentage improvements because those aren’t “enough”. But the thing is that if you are trying to cut time in half, the 6% for that one method is 12% of your target. So they’re optimizing the way some people do personal finances.
Some of them get “there” in the end, but the road they take is longer and ineffective. When you are looking at four slow methods in a particular flow, you have the perspective of seeing them as parts of a whole. Maybe there’s a way you can rewrite these 4 methods into three that do less, or five where we avoid work in the common case.
If you do them one at a time, and prioritize new ones as they cross your artificial and inaccurate threshold. you won’t see these options, and if you do you’ll talk yourself out of it because you already spent a bunch of effort on what you have now (sunk cost).
For the same reason, the ROI on vertical scaling is very good, and buying a bigger server is a lot of the time a very cheap and good way of solving a problem.
Well that was a frustrating read.
Is the key insight here that you shouldn't be depressed if your optimization gains are small because maybe you are mis-estimating the worth of a small optimization?
I mean i guess, but taken in the abstract the advice is basically: don't worry if you don't succeed, because you probably don't know enough to even evaluate if you failed. Which is a pretty depressing take.
Is the key insight here that you shouldn't be depressed if your optimization gains are small because maybe you are mis-estimating the worth of a small optimization?
I mean i guess, but taken in the abstract the advice is basically: don't worry if you don't succeed, because you probably don't know enough to even evaluate if you failed. Which is a pretty depressing take.
Did you not read the whole thing?
There's a whole story (about a criminal api venture) where the point is stated that speed-ups that look pretty small have knock on effects that allow for opportunities elsewhere, and that these opportunities snowball until something significant happens.
There's a whole story (about a criminal api venture) where the point is stated that speed-ups that look pretty small have knock on effects that allow for opportunities elsewhere, and that these opportunities snowball until something significant happens.
Yes i did. That's how i got to, if you dont understand the problem you might accidentally succeed.
It is true that knock on effects are a thing, and small fixes can sometimes have suprisingly big returns. However most small fixes do not, especially in programming. The art of optimizing is understanding the problem well enough to find the right 1% speed improvement to make, because you cannot do all of them and most don't.
What i took away from the article (after all the obnoxious long winded irrelavent tangents that went nowhere), is basically dont dispair if the fixes look small because you might accidentally stumble upon an important one that isn't small without realizing it. Which i find to be a depressing, two wrongs sometimes make a right, take.
It is true that knock on effects are a thing, and small fixes can sometimes have suprisingly big returns. However most small fixes do not, especially in programming. The art of optimizing is understanding the problem well enough to find the right 1% speed improvement to make, because you cannot do all of them and most don't.
What i took away from the article (after all the obnoxious long winded irrelavent tangents that went nowhere), is basically dont dispair if the fixes look small because you might accidentally stumble upon an important one that isn't small without realizing it. Which i find to be a depressing, two wrongs sometimes make a right, take.
That's why you profile. Software tends to get super complex as you add features and without a very thorough understanding of what is going on chances are that your program is doing something else than what you think it is doing, or at least that it is spending it's time on something else than what you think. By profiling you get a grip on that and this in turn allows you to spend your time on that 1% of the code where it spends 99% of the time. By the time you're done optimizing that a new bottle-neck will have arisen and so on. I've had software that was performing marginal end up being too fast to measure accurately over the course of a few iterations, and that's when I thought that I knew what was going on.
If Donald Knuth suffers from this as well then I wouldn't berate myself too much about it, his famous quote that 'premature optimization is the root of all evil' then you're more than excused to let optimization rest until the code is correct (which is far more important than that it is fast) and then use a profiler to tell you where to most efficiently apply your attention. This will get you to a very high percentage of the optimum in a relatively short time.
If Donald Knuth suffers from this as well then I wouldn't berate myself too much about it, his famous quote that 'premature optimization is the root of all evil' then you're more than excused to let optimization rest until the code is correct (which is far more important than that it is fast) and then use a profiler to tell you where to most efficiently apply your attention. This will get you to a very high percentage of the optimum in a relatively short time.
It would have been helpful to get to the point. Not eveyone has the better part of an hour to spend on a single web article.
I'd like to think I have some level of willpower and self control, but incremental games like Kittens Game are my kryptonite. There's something so compelling to me about seeing the numbers go up and I don't even fully understand why.
If you enjoy that type of game, I highly recommend Kittens Game as the pinnacle of the genre—especially since it doesn't feature any microtransactions or dark patterns, but be warned: it can be incredibly addicting.
If you enjoy that type of game, I highly recommend Kittens Game as the pinnacle of the genre—especially since it doesn't feature any microtransactions or dark patterns, but be warned: it can be incredibly addicting.
The cure to the skinner box addiction is to cheat. It's kinda like that stuff that smokers use to remove the satisfied feeling.
Take all the ones you can find that aren't DRM'd. Put them on a list to open when you get the itch and open up ye olde javascript console after playing one for a day or two.
Take all the ones you can find that aren't DRM'd. Put them on a list to open when you get the itch and open up ye olde javascript console after playing one for a day or two.
I did this but with nonogram puzzles. I built a solver to alleviate my addiction to solving them myself.
Skinners box my dude. A probabilistic reward is a greater dopamine spike than straight up cocaine use.
Very interesting idea! I think I've heard something like this before on giving reward randomly yielding to addictive behaviours, like gambling. Do you have any recommended reference on this idea?
Stanford Neuroscience Lectures. Dr. Spoolansky. Free online on Youtube. Forgot which lecture covered it. The whole class is worth a watch. Including the forbidden lecture if you can find it (lets just say it ruffles some feathers by noting some commonalities between OCD and spiritual ritualistic behavior).
> Stanford professor Robert Sapolsky gave the opening lecture of the course entitled Human Behavioral Biology and explains the basic premise of the course and how he aims to avoid categorical thinking.
https://www.youtube.com/watch?v=NNnIGh9g6fA&list=PL848F2368C...
https://www.youtube.com/watch?v=NNnIGh9g6fA&list=PL848F2368C...
The keywords you might be looking for are "partial/intermittent reinforcement" and "variable ratio schedule." The original descriptions are in a 1957 book by Ferster and Skinner, but any psychology textbook should have a description of them in the chapter on conditioning.
Works much better in dog training. Pretty sure it works better for training me too.
I love that it doesn’t make me feel punished if I ignore it for a few days.
That's why I like quant trading: you get to see real $ numbers go up (hopefully).
Gustafson’s Law reminds me of Jevon’s Paradox: https://en.m.wikipedia.org/wiki/Jevons_paradox
…namely, that every time we get more efficient at using energy, we increase the amount we produce of it. Instead of doing the same things with less energy, instead we are now able to do new things that were cost prohibitive previously.
…namely, that every time we get more efficient at using energy, we increase the amount we produce of it. Instead of doing the same things with less energy, instead we are now able to do new things that were cost prohibitive previously.
An example: artificial lighting. Despite most lighting transitioning to fluorescent then LED, total electric demand used for lighting hasn't really gone down as the cost has gone down; people just floodlight everything now. Some napkin math indicates Canadians use about ~10x as much artificial light per capita compared to 50 years ago.
It's not a law, of course, or at least not a simple one. E.g., the cost to refrigerate has come down similarly, but while air conditioning continues to increase in use, average freezer/fridge space plateaued in the 1980s. Everyone seems to now have as much freezer space as they want. Some demands are fully satisfiable and people don't necessarily consume more, even as the cost comes down.
It's not a law, of course, or at least not a simple one. E.g., the cost to refrigerate has come down similarly, but while air conditioning continues to increase in use, average freezer/fridge space plateaued in the 1980s. Everyone seems to now have as much freezer space as they want. Some demands are fully satisfiable and people don't necessarily consume more, even as the cost comes down.
There’s various types of demand - some seem to climb indefinitely (does anyone even bother turning lights off anymore?) and others have practical limits, like fridges and freezers.
Often those limits are the results of other aspects - but there’s not real downside to lighting everything with the fury of ten thousand suns. Besides the dark sky loss.
Often those limits are the results of other aspects - but there’s not real downside to lighting everything with the fury of ten thousand suns. Besides the dark sky loss.
Microeconomic theory (for whatever it's worth) explains this tidily: unless demand is perfectly inelastic, reducing unit costs will always increase demand at equilibrium. Demand is almost never perfectly inelastic.
Which leads me to think we should optimize for what we should be doing, not optimizing the thing we happen to be doing. Pointing in the right direction is more important than the velocity in a wrong direction.
Kittensgame! Stay away!
Paperclips have nothing on it. I warn you!
Now let me go reset, I think this time with a Challenge …
As to the article, it's a good point - optimizing something can provide more benefits than it seems at first, especially over time.
Paperclips have nothing on it. I warn you!
Now let me go reset, I think this time with a Challenge …
As to the article, it's a good point - optimizing something can provide more benefits than it seems at first, especially over time.
God I just heard of this game a month ago and it's like crack. I prefer Universal Paperclips because it had a nicely defined ending and more interesting phases, but Kittens Game is a nice diversion from the usual mobile fare. And a single purchase.
So many great observations buried in so many needlessly wordy examples.
What were those "great observations"?
Amdahl’s law says we’re only going to get marginal improvements in a lot of cases. But that’s ok because sometimes marginal improvements still pay off.
Increasing efficiency doesn’t always mean shorter runtime, sometimes it means more production from an equal runtime.
Lots of improvements to different places in the pipeline can pay off.
Sometimes an improvement in one part of the process can have synergistic knock-on effects in other, perhaps unexpected, parts of the larger system. It seems like these can be hard to predict without taking a deep look at the larger system and how all of its component parts interact.
Increasing efficiency doesn’t always mean shorter runtime, sometimes it means more production from an equal runtime.
Lots of improvements to different places in the pipeline can pay off.
Sometimes an improvement in one part of the process can have synergistic knock-on effects in other, perhaps unexpected, parts of the larger system. It seems like these can be hard to predict without taking a deep look at the larger system and how all of its component parts interact.
At least twice during my career I’ve had to shrink some binary output so it can fit onto a fixed size part in some embedded system.
Once it was a FTTX CPE, and it needed to ship but the binary was a couple KB too big. The programmer on the project found a little bit of savings but not enough. I looked at the code, looked at the assembly output, and found a spot where a switch/case was being transformed into a huge jump table that was mostly sparse. (The embedded toolchain compiler wasn’t great at optimization.) A little refactor to this one function was enough savings to be just under the limit for whatever chip they were using. (I feel bad for the next person who has to fix a bug in that product.)
The relevant aspect to my story is that sometimes an improvement in one area can make another area worse! But that’s ok, because even though my function rewrite was almost certainly slower, we had cpu cycles to spare but not enough storage.
Once it was a FTTX CPE, and it needed to ship but the binary was a couple KB too big. The programmer on the project found a little bit of savings but not enough. I looked at the code, looked at the assembly output, and found a spot where a switch/case was being transformed into a huge jump table that was mostly sparse. (The embedded toolchain compiler wasn’t great at optimization.) A little refactor to this one function was enough savings to be just under the limit for whatever chip they were using. (I feel bad for the next person who has to fix a bug in that product.)
The relevant aspect to my story is that sometimes an improvement in one area can make another area worse! But that’s ok, because even though my function rewrite was almost certainly slower, we had cpu cycles to spare but not enough storage.
thank you for the summary!
I’d love a summary, too. Started reading the first third but got bogged down by all the Kitten Game examples.
I think the conclusion at the bottom captures most of it.
I think the conclusion at the bottom captures most of it.
We could also fire the analysts, who are trying to optimize the optimizers by determining which optimization is optimal right now. Instead just realize that all the optimizations are important and you may not even be smart enough to optimally optimize.
> where we work hard, and get many of the tasks (not just one) to be more efficient, so that our overall process can work faster.
The main way by which we do that is to compile everything and to improve the compiler to make everything faster. And by making faster machines.
After that it's hard work. If you find that your program spends its time in small fractions of percentages all over the code, and it's compiled well, it may be very hard to keep the structure of that computation as-is while obtaining any significant improvement.
The main way by which we do that is to compile everything and to improve the compiler to make everything faster. And by making faster machines.
After that it's hard work. If you find that your program spends its time in small fractions of percentages all over the code, and it's compiled well, it may be very hard to keep the structure of that computation as-is while obtaining any significant improvement.
And hope you have (or can reformulate into) a programming language that is designed to facilitate ambitious optimizations.
I wonder if there's some better term around than "compiler optimization" for doing really ambitious program transformations, in the vein of program specialization/partial evaluation treating the program and its inputs as a single symbolic system to optimize for evaluation or kolmogorov complexity. I have a feeling that by missing this terminology we're already handicapping work in the area...
I wonder if there's some better term around than "compiler optimization" for doing really ambitious program transformations, in the vein of program specialization/partial evaluation treating the program and its inputs as a single symbolic system to optimize for evaluation or kolmogorov complexity. I have a feeling that by missing this terminology we're already handicapping work in the area...
We're also largely missing application specific optimisations. Occasionally I hear of people writing their own llvm pass to target their application domain but the potential scope vs exploitation of that strategy seems tragically underexplored.
Yes. I wonder if there's a way to plug the recent GPT-3 ish program synthesis advancements into this so humans could easily specify those by expressing just a hint of what to do. The problem is merciful in that you can handle a fair chunk of the correctness validation by comparing against the unoptimized version...
There was nothing wrong with Amdahl's law but the author simply took it from a wrong angle.
>>> Our brains just aren’t wired to like linear progress, I guess
This is more insightful than I first thought - and explains a lot about share and stocks and office life
This is more insightful than I first thought - and explains a lot about share and stocks and office life
The author wrote all that while trying to stop playing an idle game?
I suppose I better stay far, far away from idle games. For my entire life.
I suppose I better stay far, far away from idle games. For my entire life.
Have a question, who made info pictures for article, they are almost ununderstandable?
For example, if a common update transaction occurs once a second and takes 1.1 second, then the transactions from different users will overlap in time, possibly blocking each other on locks. Speeding up something like this just a little bit will drop the time to 0.9 seconds, reducing the percentage chance of overlap significantly. That has a compounding effect so the queries might suddenly take 0.5 seconds due the lack of lock contention!
There are similar effects due to sharing of CPU caches when queries run concurrently. Even if they’re independent, the hardware isn’t!
Getting to the point that they all run in isolation might not take much effort, but yield dramatic benefits.