The problem with slow tools (2021)(jeremymikkola.com)
jeremymikkola.com
The problem with slow tools (2021)
https://jeremymikkola.com/posts/2021_04_01_the_problem_with_slow_tools.html
59 comments
Kids these days! In my day, compiling took so long we could go get lunch, walking uphill in the snow both ways.
How can you walk uphill BOTH ways?
because it's a joke
House at top of hill, school at top of hill. Or in this case, lab at top of one hill, lunch at top of another hill.
[deleted]
Tell me you've never visited Seattle or San Francisco without telling me you've never visited Seattle or San Francisco...
cf. Monty Python
... and mind you, there were no fast food restaurants either. So it took a really long time.
Left out the mandatory https://xkcd.com/303/
Now it's "CI tests!"
Now it's "CI tests!"
I worked in a company where the codebase of a team took hours to compile so it was compiled every nights.
Interestingly, it may have been beneficial to focus because you had to work the entire day without compiling the code. I suppose that in this situation, fast tests is what you need.
Interestingly, it may have been beneficial to focus because you had to work the entire day without compiling the code. I suppose that in this situation, fast tests is what you need.
Reads like the dude just learned how to multiply and extrapolate and got carried away.
If you spent 2 minutes peeing, and pee 8 times a day, that's 5840 minutes spent peeing a year! Wow! I bet he'd love that one.
If you spent 2 minutes peeing, and pee 8 times a day, that's 5840 minutes spent peeing a year! Wow! I bet he'd love that one.
This is what corporate diapers are for.
As with anything in engineering, it depends.
https://www.walmart.com/browse/health/adult-diapers/depend/9...
https://www.walmart.com/browse/health/adult-diapers/depend/9...
He's doing that in the context of advocating for better tools.
If you could install an update that made you need to pee 10x less often, wouldn't you jump on that opportunity?
Especially if the urge keeps interrupting you when you're right in the middle of solving something?
If you could install an update that made you need to pee 10x less often, wouldn't you jump on that opportunity?
Especially if the urge keeps interrupting you when you're right in the middle of solving something?
No. I drink lots of water to pee more often, in fact.
It’s a nice reminder to get up and stretch, plus take my eyes off the monitor.
It’s a nice reminder to get up and stretch, plus take my eyes off the monitor.
Of course every delay is a potential interruption to focus which costs way, way more than the delay itself, so much so it's not worth calculating the actual delay's 'lost time'.
In fact even a momentary delay (my macbook now seems to often stutter for 2-3 seconds switching spaces) is an opportunity for your focus to drift (why is this damn thing so slow, I should get a new machine, actually, I should lookup the latest thinkpad, maybe I can tune memory use in the meantime, how do you tune macos memory, can you even?, etc ...).
In fact even a momentary delay (my macbook now seems to often stutter for 2-3 seconds switching spaces) is an opportunity for your focus to drift (why is this damn thing so slow, I should get a new machine, actually, I should lookup the latest thinkpad, maybe I can tune memory use in the meantime, how do you tune macos memory, can you even?, etc ...).
1000 times this.
Also, the time of focus loss grows highly super-linearly with the distracting delay time. One second could be imperceptible, once you get used to it, 15 seconds is enough to get distracted by looking at something else, like checking IM / email and spending 2-3 minutes on returning to the full focus, and 2 minutes is enough to go for a coffee / reddit / YouTube, and a potential loss of an hour of focus time.
Also, the time of focus loss grows highly super-linearly with the distracting delay time. One second could be imperceptible, once you get used to it, 15 seconds is enough to get distracted by looking at something else, like checking IM / email and spending 2-3 minutes on returning to the full focus, and 2 minutes is enough to go for a coffee / reddit / YouTube, and a potential loss of an hour of focus time.
> my macbook now seems to often stutter for 2-3 seconds switching spaces
I get so angry at this sort of thing. The M1 chips are so fast! When they came out they could open basically any program instantly. But now that they’ve been out for awhile, developers everywhere are (of course) just assuming everyone has an M1 or faster and doing even less optimization on their code. So the net effect is that software doesn’t get any faster. We just collectively spent billions of dollars on new hardware with almost no tangible benefit. (The exceptions being high performance computing - like AI or AAA video games where optimizations still matter).
Computers are fast. Slow software comes from lazy developers who steal your cpu cycles instead of doing their jobs properly.
It’s inexcusable.
I get so angry at this sort of thing. The M1 chips are so fast! When they came out they could open basically any program instantly. But now that they’ve been out for awhile, developers everywhere are (of course) just assuming everyone has an M1 or faster and doing even less optimization on their code. So the net effect is that software doesn’t get any faster. We just collectively spent billions of dollars on new hardware with almost no tangible benefit. (The exceptions being high performance computing - like AI or AAA video games where optimizations still matter).
Computers are fast. Slow software comes from lazy developers who steal your cpu cycles instead of doing their jobs properly.
It’s inexcusable.
Cue the usual arguments against performance tuning: "That's premature optimization", "It only has to be fast enough", etc.
It bothers me too. We have astonishing computing power, but outside certain applications we mostly just fritter it away. I would love to use a computer which does everything almost instantly, but instead I have to deal with counting to 2 in my head after pressing WinKey before typing my search term and go-and-make-a-cup-of-tea Gradle build times for pretty trivial projects.
When it comes to Apple though, I think it's somewhat intentional. "Why don't you have an M1/2 yet? Go buy one!"
It bothers me too. We have astonishing computing power, but outside certain applications we mostly just fritter it away. I would love to use a computer which does everything almost instantly, but instead I have to deal with counting to 2 in my head after pressing WinKey before typing my search term and go-and-make-a-cup-of-tea Gradle build times for pretty trivial projects.
When it comes to Apple though, I think it's somewhat intentional. "Why don't you have an M1/2 yet? Go buy one!"
All right, we've been complaining about this forever. How do we fix it? The best thing I can think of, as a developer, is to deliberately use slower computers to test my software. My daily-driver Windows PC is still a quad-core Skylake from 2016, with 16 GB of RAM and a ~500 GB SSD, and I wonder if even that's too much.
As a developer you can't fix it unless you own the product.
Spending time improving performance when you "should" be working on "features" is not looked upon fondly by employers.
Companies need to be defining acceptable performance (with metrics) on devices meeting certain specifications at the outset of projects, and failures on those tests need to be treated seriously.
Can it really be achieved? Maybe not.
Spending time improving performance when you "should" be working on "features" is not looked upon fondly by employers.
Companies need to be defining acceptable performance (with metrics) on devices meeting certain specifications at the outset of projects, and failures on those tests need to be treated seriously.
Can it really be achieved? Maybe not.
[deleted]
My 2011 thinkpad is easily many times slower than the m1/m2, yet, when running i3 on my thinkpad, the feel of responsiveness is a lot more snappy as in instance. So in reality, I have a lot less of what you mention on my ancient thinkpad than the m1 for most of what I do daily.
> my macbook now seems to often stutter for 2-3 seconds switching spaces
Wow, mine (M1 MBA) is less than one second. I either use command+tab or the trackpad. I’d be crossed too if it took that long for me.
Wow, mine (M1 MBA) is less than one second. I either use command+tab or the trackpad. I’d be crossed too if it took that long for me.
> Well, if you do the math, 15 minutes a day times 230 working days a year equals just shy of 60 hours - or 1.5 work weeks - wasted per year. Multiply that by the size of the engineering organization, and this starts to look pretty expensive.
I hate this type of reasoning. 15 minutes a day is 3% of the day. Multiply that by 230 working days a year and it's 3% of the year. Multiply that by the size of the engineering organization, it's 3%. Your maximum savings here is 3%, no matter how many numbers you multiply it by.
I hate this type of reasoning. 15 minutes a day is 3% of the day. Multiply that by 230 working days a year and it's 3% of the year. Multiply that by the size of the engineering organization, it's 3%. Your maximum savings here is 3%, no matter how many numbers you multiply it by.
All true statements. However, it's conceptually easier to grasp the impact of 3% of my time when I think about what I can or cannot accomplish in a 1.5 week period, rather than a 15 minute period. Some 15 minutes are jam-packed with productivity, while others are a coffee break, so I don't really understand what 15 minutes of my time is worth. Whereas, each work week is nearer to my average productivity, and I have a much clearer idea of what that's worth in terms of work output.
If a person reads this "if you do the math" reasoning and has a different reaction to the same cost depending on how it's presented, then, wasn't it a good thing that the reader was given a variety of ways to think about the cost? (Since, necessarily, the reader's reaction to at least one of the presentations was incorrect.)
If a person reads this "if you do the math" reasoning and has a different reaction to the same cost depending on how it's presented, then, wasn't it a good thing that the reader was given a variety of ways to think about the cost? (Since, necessarily, the reader's reaction to at least one of the presentations was incorrect.)
How large is your morgage rate? Imagine that you could shave 3% off its rate, say, turn 7% into 4%. Would it make a difference for you?
Same applies to most companies. Unless they are in an explosive growth phase, or in a catastrophic collapse phase, 3% change in expense / efficiency of the engineering organization is pretty noticeable.
Same applies to most companies. Unless they are in an explosive growth phase, or in a catastrophic collapse phase, 3% change in expense / efficiency of the engineering organization is pretty noticeable.
Going from 7% to 4% is shaving 43% off your mortgage rate—which of course would make a difference. But it’s not even the same order of magnitude as what we’re talking about here.
>Imagine that you could shave 3% off its rate, say, turn 7% into 4%
That's not how percentages work! 7% would turn into ~6.8% if you shave 3% from it.
That's not how percentages work! 7% would turn into ~6.8% if you shave 3% from it.
While I understand your interpretation, what I meant is subtracting percentage points, not applying percentage points upon percentage points.
My idea was to show that changing some big value by a few percentage points of that big value, e.g. of the cost of running the engineering organization, may have a serious impact on the overall balance.
Engineering organizations are expensive to run, and their cost in a tech-oriented company may even be the dominant cost. Saving even a small percentage of that cost is pretty noticeable, compared to other expenses of the company.
My idea was to show that changing some big value by a few percentage points of that big value, e.g. of the cost of running the engineering organization, may have a serious impact on the overall balance.
Engineering organizations are expensive to run, and their cost in a tech-oriented company may even be the dominant cost. Saving even a small percentage of that cost is pretty noticeable, compared to other expenses of the company.
Companies seem to tolerate the 8% of each Teams or zoom meeting lost on “wait, I can’t get my camera working”, “sorry I’m late; what did I miss?”, etc.
I’m convinced it’s a percent or two of GDP!
I’m convinced it’s a percent or two of GDP!
Actually you are ignoring the main argument here that people aren't perfect and that the risk of losing more time due to people getting distracted, reading slack, going on long coffee breaks, etc. is substantial and very real. You lose all that time; not just the build time. And having to wait is frustrating and demotivating. And let's face it, a 1 minute build is actually pretty sweet. I've been on teams where it is closer to 20. And worse. That really sucks the life out of you if you have to sit and wait for that to happen fifteen times in a day. People wasting hours per day waiting for builds, CI, etc. to happen is not cheap.
At my current place they have just replaced the build system.
If I want it deployed to the "higher" dev environments I have to watch it for 20 mins so that I can "approve" the override that my base image (that I have no control over) has security risks and I accept them.
If I want to deploy to production, I have to watch and approve the 20 min gatekeeper, them again at 40 mins, and then at 120 mins accept that I want a change ticket.
If I miss any of these then I have to restart the build process. This is an "agile" workplace.
Fuck everyone who claims they agile doesn't work, just stop calling this shit agile.
If I want it deployed to the "higher" dev environments I have to watch it for 20 mins so that I can "approve" the override that my base image (that I have no control over) has security risks and I accept them.
If I want to deploy to production, I have to watch and approve the 20 min gatekeeper, them again at 40 mins, and then at 120 mins accept that I want a change ticket.
If I miss any of these then I have to restart the build process. This is an "agile" workplace.
Fuck everyone who claims they agile doesn't work, just stop calling this shit agile.
It's scaled agile my friend. Or was it scalable agile? Whatever.
"Well, let's say you can shave 10 seconds off of the boot time. Multiply that by five million users and thats 50 million seconds, every single day. Over a year, that's probably dozens of lifetimes. So if you make it boot ten seconds faster, you've saved a dozen lives. That's really worth it, don't you think?"
https://www.folklore.org/StoryView.py?story=Saving_Lives.txt
I think of this quote whenever I am waiting for my iPhone to boot. Though I suppose Apple would argue that you don't need to reboot your iPhone every day the way you might have with older computers. There were never a billion Mac users though.
https://www.folklore.org/StoryView.py?story=Saving_Lives.txt
I think of this quote whenever I am waiting for my iPhone to boot. Though I suppose Apple would argue that you don't need to reboot your iPhone every day the way you might have with older computers. There were never a billion Mac users though.
That's why I hibernate my computers.
Problem is that Microsoft is trying to kill hibernation (so far they did the first step to hide it, then second step to hide it even better).
Why do the companies not realize that employees coming to office, waiting for computer to start, then opening their programs - is an incredible waste of time and productivity? Why companies dont push microsoft to make hibernation the default? (plus some smart feature for updates)
Problem is that Microsoft is trying to kill hibernation (so far they did the first step to hide it, then second step to hide it even better).
Why do the companies not realize that employees coming to office, waiting for computer to start, then opening their programs - is an incredible waste of time and productivity? Why companies dont push microsoft to make hibernation the default? (plus some smart feature for updates)
This trend is so incredibly frustrating and will likely be the final straw that makes me go back to linux on my desktop instead of windows.
I can put up with many things, but not this.
The machine already reboots for goodness knows what reason. Probably some update that someone decided was far more critical than my machine being up.
Also why can't updates run on windows without a reboot? We're getting to the point where other than design, windows feels like the volunteer run OS and that's not me saying the linux desktop design is bad by any measure, just that I prefer windows.
I can put up with many things, but not this.
The machine already reboots for goodness knows what reason. Probably some update that someone decided was far more critical than my machine being up.
Also why can't updates run on windows without a reboot? We're getting to the point where other than design, windows feels like the volunteer run OS and that's not me saying the linux desktop design is bad by any measure, just that I prefer windows.
What does Windows promote instead of hibernation? Ie what -- if anything -- is what the OS or users are supposed to use?
Modern hardware...
Also hitting shutdown on your windows PC does not shut it down, it effectively puts it into hibernation.
The comment above is chasing the wind, just shutdown your PC if you want it off. If you dont then put it to sleep, modern hardware sleep state is effectively off from a power consumption point of view.
Also hitting shutdown on your windows PC does not shut it down, it effectively puts it into hibernation.
The comment above is chasing the wind, just shutdown your PC if you want it off. If you dont then put it to sleep, modern hardware sleep state is effectively off from a power consumption point of view.
The idea is that modern windows (10,11,12) boot fast enough through various tricks that booting is faster - or as fast as - awaking from hibernation.
TBH It's probably not wrong, I wouldn't know because for some unknown reason my motherboard sits doing nothing for over a minute before starting the POST. Once it's past that, I can be at a linux prompt in 20 seconds, or at the windows login within 40, depending on whether I interrupt rEFInd
TBH It's probably not wrong, I wouldn't know because for some unknown reason my motherboard sits doing nothing for over a minute before starting the POST. Once it's past that, I can be at a linux prompt in 20 seconds, or at the windows login within 40, depending on whether I interrupt rEFInd
[deleted]
I ran a survey and some one-on-one interviews of 30 or so engineers at my workplace to get some real data about the impact of slow tools. IIRC, the results were something like this:
* Most engineers can maintain focus on a task for up to 10 seconds waiting for a slow tool. However a couple of engineers (myself included) will get derailed at only about 3-5 seconds.
* A tool that runs for more than a minute will cause just about every engineer to switch task while it is completing. (This means they stop working on their highest-impact work and likely spend time on lower-impact things).
* A tool that runs for more than 15 minutes will make any engineer have to start deliberate multi-tasking where they are trying to work on two major work items at once.
Also, I wouldn't have read the article or posted this comment if it weren't for the slow CI pipelines at my workplace.
* Most engineers can maintain focus on a task for up to 10 seconds waiting for a slow tool. However a couple of engineers (myself included) will get derailed at only about 3-5 seconds.
* A tool that runs for more than a minute will cause just about every engineer to switch task while it is completing. (This means they stop working on their highest-impact work and likely spend time on lower-impact things).
* A tool that runs for more than 15 minutes will make any engineer have to start deliberate multi-tasking where they are trying to work on two major work items at once.
Also, I wouldn't have read the article or posted this comment if it weren't for the slow CI pipelines at my workplace.
If everyone on the team is easily distracted then why would you focus on speeding up slow tooling instead of focusing on making your team resilient to distractions?
Ah, yes, just change human nature. That sounds much easier than speeding up a program.
You can train yourself to focus better and to resist distraction, although it definitely is hard to do. The benefit is that it works for everything, from running a script to ignoring Slack to being able to work in an open office. Improving how long how code takes to run only fixes that specific distraction.
One of the worst unspoken bits is the cognitive load associated with such long running things, e.g compile or CI, which ideally are successful but that's not always the case, so you're doing something else while - consciously or not - anxiously waiting for the other thing to complete.
And when - when! - that job fails, you can be pretty sure it racks up some cortisol as one has to interrupt† your second task, pop back to the original context, curse at oneself for forgetting that comma or whatever simple thing, rerun, rinse, and repeat.
† The alternative is to finish (or reach a checkpoint) your other task, but that has cumulative latency effects on the duration of that other task that is supposed to be your main one.
And when - when! - that job fails, you can be pretty sure it racks up some cortisol as one has to interrupt† your second task, pop back to the original context, curse at oneself for forgetting that comma or whatever simple thing, rerun, rinse, and repeat.
† The alternative is to finish (or reach a checkpoint) your other task, but that has cumulative latency effects on the duration of that other task that is supposed to be your main one.
Slow tool is annoying, but it's not critical.
Bad code, shitty micromanagement, garbage meetings is the main issues.
Bad code, shitty micromanagement, garbage meetings is the main issues.
The assertion here is that these lost bits of time are purely negative.
I come back with fresh eyes and spot things, get coffee, tidy the desk, make a call, stretch the leg, use the restroom, &c.
But I don't pass myself off as an ur-coder, so possibly I don't count.
I come back with fresh eyes and spot things, get coffee, tidy the desk, make a call, stretch the leg, use the restroom, &c.
But I don't pass myself off as an ur-coder, so possibly I don't count.
These articles about extrapolating the value of a minute of wasted time into a years worth of value, always forget to divide them by the average productive hours a day. I assume you’re not optimising for zero breaks.
The problem with multiplying all the way to the organization size is that it's always going to end up being a big number.
Imagine all the oxygen we would save if we all stopped saying "bless you" every time someone sneezes, multiplied by the population of the world, multiplied by an average time per year someone sneezes multipled by an average lifetime multipled by an average amount of people in the room who can hear the sneeze.
Imagine all the oxygen we would save if we all stopped saying "bless you" every time someone sneezes, multiplied by the population of the world, multiplied by an average time per year someone sneezes multipled by an average lifetime multipled by an average amount of people in the room who can hear the sneeze.
I'm not going to say that slow tools is not an issue, however I think slow management processes has a far greater cost to an organization, and the amount of people in management or business analysts doing psedou-work.
Which the author also indicate end of the post, how many of us don't have a few 30-60 minutes meetings a week that effectively have no value to anyone.
Compiling and deploying effectively destroys my train of thought every time.
The only thing that has helped is to reduce build times from minutes to seconds.
By disabling link-time optimization in pre-production and by throwing out Docker because it doesn't play nice with build tool caches.
The only thing that has helped is to reduce build times from minutes to seconds.
By disabling link-time optimization in pre-production and by throwing out Docker because it doesn't play nice with build tool caches.
Applications that try to do too much in one application often start to experience serious performance problems as time goes on. Almost all applications written with node.js have this problem.
Good article. It focuses on build times as the bottelneck, but actually for me most of the time that bottleneck is 2FA.
By the time I've managed to sign in to my email to double check an instruction, I've already forgotten what I was looking for...
By the time I've managed to sign in to my email to double check an instruction, I've already forgotten what I was looking for...