The other half of "Artists Ship"(paulgraham.com)
paulgraham.com
The other half of "Artists Ship"
http://www.paulgraham.com/artistsship.html
131 comments
I think this can be generalized to say that start-ups have a different cost-benefit analysis - one where losses are capped at the (relatively small) value of the company. On the other hand, large companies have to be risk-averse because the worst-case is several magnitudes worse. When you're working at ConEd or AIG, the "tiny probability/worst-case-loss" factors start to matter, because "worst-case" can include investors losing life savings, federal investigations, and jail time for your boss's boss.
I don't think greater risk accounts for all the slowness of big companies. Some (probably most) of it is simply breakage caused by their large size.
The major factor at work is simply a lack of personal responsibility towards the end product in a large company.
In a startup there's noone to blame but you, and thus no need for politics. As anyone who has worked in middle management or above in a large company can tell you half of the job is making sure you can always find someone else to blame, while taking credit for the successes of others.
In a startup there's noone to blame but you, and thus no need for politics. As anyone who has worked in middle management or above in a large company can tell you half of the job is making sure you can always find someone else to blame, while taking credit for the successes of others.
Funny that some of those large companies you mention didn't seem to get the memo about being risk-averse... ;-)
[deleted]
During their executive retreats they decided to be risk seeking instead.
I wonder if the problem at the recently troubled companies (AIG, Merrill, Lehman etc) was not enough checks in place or not enough power allotted to the employees who knew something was wrong.
I wonder if the problem at the recently troubled companies (AIG, Merrill, Lehman etc) was not enough checks in place or not enough power allotted to the employees who knew something was wrong.
From memory: the Excel bug that caused Moody's to give high ratings of mortgage securities (one of the probable causes of the whole mess) was known, but management decided not to fix it. So, greed can override checks and balances.
The issue was not a bug, it was a fundamental misunderstanding of how real estate markets work. Moody's had a model for pricing mortgages which assumed real estate prices never went down. It was basically the real life equivalent of the famous "Assume a spherical cow..." problem.
This past summer I worked as an intern for a big company. On the first day of orientation the head of the legal compliance department said very flatly to us that any individual could do far more harm to the firm than good.
This mentality, reenforced by the company's bureaucratic change management system, really did not sit well with me.
Fortunately for my summer experience, my "buddy"/summer mentor and I found a loophole which we used to "hack" the change management system so that once we got our initial approval, we could propagate changes to prod without running through the whole process again for each change (which otherwise would have been required).
This mentality, reenforced by the company's bureaucratic change management system, really did not sit well with me.
Fortunately for my summer experience, my "buddy"/summer mentor and I found a loophole which we used to "hack" the change management system so that once we got our initial approval, we could propagate changes to prod without running through the whole process again for each change (which otherwise would have been required).
For large companies with a valuable reputation, that's almost guaranteed to be true. Most people won't generate a million dollars worth of value in a given year, but nearly everyone could do that amount of damage to their company's reputation in just a few minutes.
Sure, yes, if e.g. the customers' health or safety was at risk. But I don't think the average bug in a web app would damage a company's reputation significantly. GMail occasionally shows me a message saying "Oops, an error occurred." It doesn't make me think any less of Google.
You aren't a typical non-early-adopter user. I've had people make loud, public complaints about such trivial matters as a new logo being too tall, or a text box being too wide, or an optional WYSIWYG editor feature ruining the "text-only flavor of the community". These on websites far smaller than GMail.
Look at some of the major Web2.0 kerfuffles in recent years. Off the top of my head, I can think of:
- The HD/DVD mutiny on Digg
- Public suicides on both JoelOnSoftware and Justin.TV
- Reddit storing passwords in cleartext and them getting stolen off a laptop
- Ariel Waldman and the Twitter harassment fiasco
- The Flickr censorship debate here: http://www.flickr.com/help/forum/40074/page3/
Yeah, you could argue that those services are all still around, so obviously they haven't been hurt too much. How much management time was wasted in dealing with them? How many non-users decided not to become users because of something they heard 3rd-hand about how it's a terrible company?
(And I don't think the solution is to never innovate. I can understand why a middle manager at a big company would think so, though - a PR disaster is by definition public and disastrous. In the short run, it always makes more sense to not mess with success, it's just that this doesn't lead to success in the long run.)
Look at some of the major Web2.0 kerfuffles in recent years. Off the top of my head, I can think of:
- The HD/DVD mutiny on Digg
- Public suicides on both JoelOnSoftware and Justin.TV
- Reddit storing passwords in cleartext and them getting stolen off a laptop
- Ariel Waldman and the Twitter harassment fiasco
- The Flickr censorship debate here: http://www.flickr.com/help/forum/40074/page3/
Yeah, you could argue that those services are all still around, so obviously they haven't been hurt too much. How much management time was wasted in dealing with them? How many non-users decided not to become users because of something they heard 3rd-hand about how it's a terrible company?
(And I don't think the solution is to never innovate. I can understand why a middle manager at a big company would think so, though - a PR disaster is by definition public and disastrous. In the short run, it always makes more sense to not mess with success, it's just that this doesn't lead to success in the long run.)
None of these were the kind of bug that comes from releasing code with insufficient testing. The only one that even involved code was the Reddit password problem, and that was more a design mistake than a bug.
A culture of testing is a lot more than just verifying functionality. Code reviews with experienced engineers would have caught that design flaw before it was released. Or made it a high priority to fix.
I've dealt with the security teams at large companies. They end up implementing sometimes draconian policies, but it seems that programmers refuse to learn to write secure code any other way. I have to admit that, if you consider the large company as being a legitimate entity, they're doing legitimate work.
I've dealt with the security teams at large companies. They end up implementing sometimes draconian policies, but it seems that programmers refuse to learn to write secure code any other way. I have to admit that, if you consider the large company as being a legitimate entity, they're doing legitimate work.
Privacy breach could harm a company a lot. Japan Network Security Association reported that, in 2007, total of 30 million personal information records are breached and estimated 22 billion dollars for damage/compensation in Japan.
Most of these incidents caused by employees taking out the data, but there are some cases caused by sloppy web app.
(I could find only Japanese version of the report: http://www.jnsa.org/result/2007/pol/incident/index.html )
Most of these incidents caused by employees taking out the data, but there are some cases caused by sloppy web app.
(I could find only Japanese version of the report: http://www.jnsa.org/result/2007/pol/incident/index.html )
The kind of bug big companies are concerned about is the kind that causes google.com to be down for 15 minutes. That's a significant amount of revenue. Or the sort of bug that causes a release of an infrastructure component to be rolled back, causing more work for every project touched by that release. It isn't the famous sort of bug, these don't get picked up by the news, it's the sort of bug that nobody really notices but costs the organization a lot of money or work.
Imagine if, say, an Amazon customer service rep actually yelled, screamed, and abused a customer calling about his order.
A single individual - one who is not even in a high position. But you bet your buttons that it'll be all over the internet within ours.
Sure, it probably wouldn't even cause a blip to the stock price, but it's a huge amount of harm for a single person to cause.
I once worked for a company where anybody can push changes to prod without much feedback whatsoever. Certain more senior guys watched the commit logs for any obvious shenanigans or idiocy, but generally changes were within customers' hands within a day or two (and this was a desktop app!). They suffered some horrifying stability problem for a while that necessitated some new policies. It was certainly liberating to work in that environment, but I can see why many places have safeties in place.
A single individual - one who is not even in a high position. But you bet your buttons that it'll be all over the internet within ours.
Sure, it probably wouldn't even cause a blip to the stock price, but it's a huge amount of harm for a single person to cause.
I once worked for a company where anybody can push changes to prod without much feedback whatsoever. Certain more senior guys watched the commit logs for any obvious shenanigans or idiocy, but generally changes were within customers' hands within a day or two (and this was a desktop app!). They suffered some horrifying stability problem for a while that necessitated some new policies. It was certainly liberating to work in that environment, but I can see why many places have safeties in place.
The average billion dollar company is not in the web app business.
If the error was a security vulnerability in Lotus notes that resulted in all of GM's confidential email being uploaded to thepiratebay I think IBM's reputation would take a pretty serious hit.
And this is an error that one unchecked programmer would be perfectly able to make if there were no checks in place.
And this is an error that one unchecked programmer would be perfectly able to make if there were no checks in place.
Well, what if this bug happened to allow content which was not-safe-for-children onto the front page of the most heavily trafficked web page in the world, widely promoted to Middle America as being G-rated? I speak purely hypothetically of course.
If gmail deleted all your mail it might, though.
Most people might not generate a million in value per year, but some people can generate a million in value per day.
The flip side is that if an employee does generate that much income in one year, a large company isn't set up to reward it.
Case in point: I work for a $3Bn company and a year ago I developed a feature that saved the business ~$2M that year. We know the amount because when the problem raised its head we called together a number of people to calculate what the cost would be and estimates were between $1.5M - $3.5M if we couldn't find a solution. Not loss of revenue, but we would have actually had to pay out that much in support costs. The problem was highly visible across the organization and I was able to find a way to solve the it in a way that was completely invisible to the customer, implemented it and released to the field. My manager looked good, his manager looked good. I got an "average" performance review. I was pissed and to this day I haven't regained my motivation to do that caliber of work again. I wonder if I ever will without leaving!
Case in point: I work for a $3Bn company and a year ago I developed a feature that saved the business ~$2M that year. We know the amount because when the problem raised its head we called together a number of people to calculate what the cost would be and estimates were between $1.5M - $3.5M if we couldn't find a solution. Not loss of revenue, but we would have actually had to pay out that much in support costs. The problem was highly visible across the organization and I was able to find a way to solve the it in a way that was completely invisible to the customer, implemented it and released to the field. My manager looked good, his manager looked good. I got an "average" performance review. I was pissed and to this day I haven't regained my motivation to do that caliber of work again. I wonder if I ever will without leaving!
Some things that PG didn't make explicit: web applications can have an extraordinarily tight feedback loop between customer and coder. Multiple revisions per day are easy, especially if any single user's data is rather low-value. Many a startup of this kind has zoomed past its corporate competitors simply by iterating faster. If your product is embedded firmware for a home security device, this strategy is just not available.
But here's another twist: when you start working for a giant company that provides myriad services under the same login, suddenly everyone has to move as slowly as the slowest part of the business. It doesn't matter if you are just doing Yippee! Backgammon, because who knows -- maybe you could somehow accidentally expose the data at Yippee! Payment Solutions.
This is a completely rational consequence of being a giant company and delivering multiple web services under the same login. It is possible to have slightly different authentication policies for each service. But I wonder if OpenID providers have thought about this enough.
But here's another twist: when you start working for a giant company that provides myriad services under the same login, suddenly everyone has to move as slowly as the slowest part of the business. It doesn't matter if you are just doing Yippee! Backgammon, because who knows -- maybe you could somehow accidentally expose the data at Yippee! Payment Solutions.
This is a completely rational consequence of being a giant company and delivering multiple web services under the same login. It is possible to have slightly different authentication policies for each service. But I wonder if OpenID providers have thought about this enough.
Some things that PG didn't make explicit: web applications can have an extraordinarily tight feedback loop between customer and coder.
He has made that point in other essays.
He has made that point in other essays.
[deleted]
"...not only wouldn't these guys have broken anything, they'd have gotten a lot more done."
I don't think the claim that they would not have broken anything can be backed by facts.
If you look at Microsoft, for instance, their programmers used to write very buggy code, but with the introduction of better processes, like their Secure Development Lifecycle, they saw a substantial improvement in code quality (as measured through security and reliability metrics).
The damage done by not following a quality assurance process and writing buggy code was so big that Microsoft's image will be affected for a long time, even though they have now improved the code quality.
I don't think the claim that they would not have broken anything can be backed by facts.
If you look at Microsoft, for instance, their programmers used to write very buggy code, but with the introduction of better processes, like their Secure Development Lifecycle, they saw a substantial improvement in code quality (as measured through security and reliability metrics).
The damage done by not following a quality assurance process and writing buggy code was so big that Microsoft's image will be affected for a long time, even though they have now improved the code quality.
I meant these particular guys.
My point still is valid.
Even these particular guys certainly would have written buggy code. A QA process could correct it.
Even these particular guys certainly would have written buggy code. A QA process could correct it.
I worked without testing for 2 years and released a single bug into production. In that specific case I thought something might have been wrong but they needed it ASAP so I said who cares. My father once had a tester say "Ops I stopped testing your code over a year ago" when he released a fix to some buggy code someone else wrote which still had a few bugs. After adding a formal QA process I stopped double checking how buggy my code so I could get more done but I have ended up releasing more bugs.
So I have little respect for formal QA.
So I have little respect for formal QA.
QA is really useful, but there's a big downside to it.
When you have people who's job is to test code, developers get lazy and sloppy.
Also, bugs do appear in production if you can release code the same day you wrote it, but you can fix it as soon as you find out about it. In big companies that work with milestones (sprints in agile speak), only really critical bugs can be fixed between releases. Not to mention that most bugs are fixed after code-freeze, when all the deliverables of that milestone are ready, long after you find out about it.
So I guess it's best to have QA, but also the freedom to deploy code on production whenever you want.
Also, bugs do appear in production if you can release code the same day you wrote it, but you can fix it as soon as you find out about it. In big companies that work with milestones (sprints in agile speak), only really critical bugs can be fixed between releases. Not to mention that most bugs are fixed after code-freeze, when all the deliverables of that milestone are ready, long after you find out about it.
So I guess it's best to have QA, but also the freedom to deploy code on production whenever you want.
[deleted]
I think the costs of safety checks can be more pronounced in software than in other areas of business. It's not just that good hackers are particularly irritated by them -- they're irritated for a reason.
A two-week release-process lag is so much worse than it might sound. If you can ship instantly, you can get feedback and make improvements rapidly, over and over. Sure, without a careful release process you might occasionally break things, but when you do you can fix them in minutes rather than weeks. When you insert release-process lag into that cycle, improvements that might have been made in a rapid series of releases over a day or two can take months. Sometimes those improvements just never get made, because no one is motivated to keep working on them for so long.
In software, "stable" often means "unchanging", but only rarely means "high-quality".
A two-week release-process lag is so much worse than it might sound. If you can ship instantly, you can get feedback and make improvements rapidly, over and over. Sure, without a careful release process you might occasionally break things, but when you do you can fix them in minutes rather than weeks. When you insert release-process lag into that cycle, improvements that might have been made in a rapid series of releases over a day or two can take months. Sometimes those improvements just never get made, because no one is motivated to keep working on them for so long.
In software, "stable" often means "unchanging", but only rarely means "high-quality".
Some thoughts:
As companies grow big, they like to emulate larger companies, and hence the people they like to hire are from larger companies since they understand scale -- this comes with a lot of baggage ofcourse. This in turn, most times, brings with it a culture of people who are worried more about not screwing up, covering their own ass, and thinking about getting promoted rather than doing what's right for the company or the business and screwing up as a byproduct of making quick decisions.
Re Joel Spolsky: The cost of the sale is not only the cost of the product, but what an organization would pay for a software/service, and what they would value it at. This is why most companies have a sales force, and don't advertise their prices on their website. As Joel mentions, for a lot of companies, charging less than $50k is a rounding error and not worth their time.
The flip side of people at big companies buying is based not entirely on the performance of the service or the product. As long as the service/product is reliable & ok, and it doesn't screw up in any major ways, they won't get fired for making that buying decision, rather than looking at ways of maximizing/optimizing the performance of the product -- yet another reason committees make buying decisions.
Re SOX: I like what founder's fund + facebook is doing -- letting early employees cash some of their equity out, thereby increasing the time early employees will stay with the company.
I also like Fred Wilson's thoughts on a secondary market for startup stock (similar to what goog does), this would in some sense, I guess let you be a private company, and not deal with the challenges associated with SOX compliance and at the same time not raise a highly dilutive Series D, in addition there is some sort of liquidation event for the investors.
PS: (std. Buchheit comments about Limited Life Experiences + Overgeneralization = Advice apply)
PPS: Say hi to the reddit guys for me ;-)
As companies grow big, they like to emulate larger companies, and hence the people they like to hire are from larger companies since they understand scale -- this comes with a lot of baggage ofcourse. This in turn, most times, brings with it a culture of people who are worried more about not screwing up, covering their own ass, and thinking about getting promoted rather than doing what's right for the company or the business and screwing up as a byproduct of making quick decisions.
Re Joel Spolsky: The cost of the sale is not only the cost of the product, but what an organization would pay for a software/service, and what they would value it at. This is why most companies have a sales force, and don't advertise their prices on their website. As Joel mentions, for a lot of companies, charging less than $50k is a rounding error and not worth their time.
The flip side of people at big companies buying is based not entirely on the performance of the service or the product. As long as the service/product is reliable & ok, and it doesn't screw up in any major ways, they won't get fired for making that buying decision, rather than looking at ways of maximizing/optimizing the performance of the product -- yet another reason committees make buying decisions.
Re SOX: I like what founder's fund + facebook is doing -- letting early employees cash some of their equity out, thereby increasing the time early employees will stay with the company.
I also like Fred Wilson's thoughts on a secondary market for startup stock (similar to what goog does), this would in some sense, I guess let you be a private company, and not deal with the challenges associated with SOX compliance and at the same time not raise a highly dilutive Series D, in addition there is some sort of liquidation event for the investors.
PS: (std. Buchheit comments about Limited Life Experiences + Overgeneralization = Advice apply)
PPS: Say hi to the reddit guys for me ;-)
There's an excellent prototyping policy in effect at Maxis:
In terms of time, they have a policy of permission vs. forgiveness. You need to be prepared to fail early – but that’s okay! If a prototype takes less than two days, don’t worry: just go ahead and do it. It if takes more... you should probably have permission.
http://www.gamasutra.com/php-bin/news_index.php?story=11628
In terms of time, they have a policy of permission vs. forgiveness. You need to be prepared to fail early – but that’s okay! If a prototype takes less than two days, don’t worry: just go ahead and do it. It if takes more... you should probably have permission.
http://www.gamasutra.com/php-bin/news_index.php?story=11628
We used to do releases whenever we wanted. After this took the site out one too many times, we moved to at most daily releases with an hour of QA.
This actually made things worse. Aggressive releases were often the result of a decision further up the chain that something was "critical" and needed to go out right then and there. We'd got into the pattern of expecting to be able to do this at the drop of a hat.
So now, everyone was rushing to get the "urgent fix" into the daily release. This is an effective way to ship buggy software. Extra emergency releases were frowned upon, and so it introduced a level of panic that "Oh shit, this better not screw everything up". Ironically, with anyone releasing whenever they wanted, there was less panic because you could always make a quick fix and put it out without too many people noticing or caring.
We felt as a team that we'd prefer a more relaxed schedule that also allowed testing and accountability.
Frequent releases were a result of bad management, and unrealistic stakeholder expectations, we as developers didn't necessarily care that our code wasn't being put out immediately. Obviously, it needs to go out at some point, and we've been equally demoralised by mammoth projects that have gone on for months without a release, but that's a different end of the spectrum. The important thing is regular releases, not frequent ones.
We've now decided to move to releases every 2 weeks with a 1 week QA period. We made this decision as a team of developers, it wasn't handed down from above, it just became a necessary check for the scale of our company.
I personally feel a lot more productive as a result, less panicked by the quick fix you need to drop everything to fit into the daily release, and less worried that a release will take out the site.
But sure, it comes at a cost. There's now a lot more going out with each release and more variables that could combine to cause issues, but these issues highlight deficiencies in our QA process that can be fixed. Also, getting our team to work according to this 2 week schedule has meant some significant costs to adopt better development processes, namely SCRUM, but I feel our team has benefited enormously from it. It all depends how well your team can adapt really.
So yes, there are costs associated, but it doesn't necessarily lead to demotivated developers. You need to be sure that the checks are introduced properly with their own test suite. This itself has a cost but can be extremely worthwhile. A half arsed and arbitrary change to your release schedule with no process changes can destroy you, but with a little training, the right team and the right managers, you can make things better for everyone.
This actually made things worse. Aggressive releases were often the result of a decision further up the chain that something was "critical" and needed to go out right then and there. We'd got into the pattern of expecting to be able to do this at the drop of a hat.
So now, everyone was rushing to get the "urgent fix" into the daily release. This is an effective way to ship buggy software. Extra emergency releases were frowned upon, and so it introduced a level of panic that "Oh shit, this better not screw everything up". Ironically, with anyone releasing whenever they wanted, there was less panic because you could always make a quick fix and put it out without too many people noticing or caring.
We felt as a team that we'd prefer a more relaxed schedule that also allowed testing and accountability.
Frequent releases were a result of bad management, and unrealistic stakeholder expectations, we as developers didn't necessarily care that our code wasn't being put out immediately. Obviously, it needs to go out at some point, and we've been equally demoralised by mammoth projects that have gone on for months without a release, but that's a different end of the spectrum. The important thing is regular releases, not frequent ones.
We've now decided to move to releases every 2 weeks with a 1 week QA period. We made this decision as a team of developers, it wasn't handed down from above, it just became a necessary check for the scale of our company.
I personally feel a lot more productive as a result, less panicked by the quick fix you need to drop everything to fit into the daily release, and less worried that a release will take out the site.
But sure, it comes at a cost. There's now a lot more going out with each release and more variables that could combine to cause issues, but these issues highlight deficiencies in our QA process that can be fixed. Also, getting our team to work according to this 2 week schedule has meant some significant costs to adopt better development processes, namely SCRUM, but I feel our team has benefited enormously from it. It all depends how well your team can adapt really.
So yes, there are costs associated, but it doesn't necessarily lead to demotivated developers. You need to be sure that the checks are introduced properly with their own test suite. This itself has a cost but can be extremely worthwhile. A half arsed and arbitrary change to your release schedule with no process changes can destroy you, but with a little training, the right team and the right managers, you can make things better for everyone.
Having never had a "normal" job per se (okay, I bagged groceries for a year when I was younger), I still find it hard to believe that it can take two weeks just to get something deployed. Maybe one day I'll finally truly believe all the stories I've heard -- two-week deployments, meetings about future meetings, etc. For now, some part of me still believes it's simply implausible and everyone is engaging in hyperbole...
Sure. It is often due to dysfunction, but it can even happen to good teams.
Imagine you have to fix a simple bug, like the description of an option in an HTML page.
Imagine that: the codebase is a bajillion lines of Java or C++ that no one person truly understands any more.
Imagine that: fucking up is considered bad. Maybe you're protecting important user data, or maybe it's just code for the online store, and even an hour of downtime can wipe out the day's profits.
The team is so big that your feature is scheduled for deployment along with four or five others this week.
As usual, management has skimped on QA so there are not enough of them to go around. Also, QA was added late, so there's no fast automated test suite. It takes them a full week of manual and semi-automatic testing to go through all regressions.
After that week of testing in development is over, then it goes to staging, where it's tested with real live data and staging subsystems.
At least somewhere in this, something breaks (something is always breaking) and a fix is produced. Then the new build is tested all over again, with the basic smoke tests as well as any test suites that are applicable to that unit of code which broke.
This is causing another trainwreck because we promised feature X to client Y by end of the month. What do we do? Emergency team meeting! Let's rebalance the work schedule.
Meanwhile, your change, which just a simple HTML fix, languishes as this build goes into its second round of testing.
Oh, and did we mention that you have to have submitted your code in time for translation into six languages? I'm afraid that the French translator did not produce a satisfactory translation in time, so we're going to have to hold your change back. Can you revert this and put it in next week's build? Thanks.
Imagine you have to fix a simple bug, like the description of an option in an HTML page.
Imagine that: the codebase is a bajillion lines of Java or C++ that no one person truly understands any more.
Imagine that: fucking up is considered bad. Maybe you're protecting important user data, or maybe it's just code for the online store, and even an hour of downtime can wipe out the day's profits.
The team is so big that your feature is scheduled for deployment along with four or five others this week.
As usual, management has skimped on QA so there are not enough of them to go around. Also, QA was added late, so there's no fast automated test suite. It takes them a full week of manual and semi-automatic testing to go through all regressions.
After that week of testing in development is over, then it goes to staging, where it's tested with real live data and staging subsystems.
At least somewhere in this, something breaks (something is always breaking) and a fix is produced. Then the new build is tested all over again, with the basic smoke tests as well as any test suites that are applicable to that unit of code which broke.
This is causing another trainwreck because we promised feature X to client Y by end of the month. What do we do? Emergency team meeting! Let's rebalance the work schedule.
Meanwhile, your change, which just a simple HTML fix, languishes as this build goes into its second round of testing.
Oh, and did we mention that you have to have submitted your code in time for translation into six languages? I'm afraid that the French translator did not produce a satisfactory translation in time, so we're going to have to hold your change back. Can you revert this and put it in next week's build? Thanks.
I've waited a year to launch something. Then I quit.
It's not abnormal. In rails I have deployed to a customer in 12 hours, starting from scratch. I was using my own framework though. In gwt, I have done this in ~24 hours.
These weren't simple apps either, not just scaffolding, but had complex user interfaces.
I wasn't referring to task difficulty, just the extreme time delay between when something is ready and when it is put into production.
I was referring to development & production deployments in both examples.
The delay is between when it's first claimed finished by the author(s) and when it's agreed to be ready by everyone involved. After that it can be deployed instantly...
I guess it would be possible to note in a bug tracking app "we would have deployed here" and then see how many, and what kind of bugs were found between then and actual deployment, which checks found them, and whether any of those were rollback-causers.
That would give an estimate of how much good the checks did in each case, and a way to choose which checks are ineffective for their cost. (Ish - some checks may only trigger once, ever, to be priceless).
I guess it would be possible to note in a bug tracking app "we would have deployed here" and then see how many, and what kind of bugs were found between then and actual deployment, which checks found them, and whether any of those were rollback-causers.
That would give an estimate of how much good the checks did in each case, and a way to choose which checks are ineffective for their cost. (Ish - some checks may only trigger once, ever, to be priceless).
I would add that in addition to having to show the cost of any new checks that one should have to show how the check will actually prevent the problem it proposes to prevent and the likely-hood of that problem occurring or re-curring. A major problem I've seen with checks is that they just become CYA material rather than anything truly beneficial or preventative and people will work around them.
One problem is that the cost of checks is only visible in the aggregate. The marginal cost of each new check seems to be pretty low.
I wonder if there's a way to limit the total cost without hard-and-fast silly rules.
I wonder if there's a way to limit the total cost without hard-and-fast silly rules.
Is "one in, one out" a silly rule?
e.g. you can only add a new step to a process if a step is removed from another.
e.g. you can only add a new step to a process if a step is removed from another.
Is "one in, one out" a silly rule?
Yes. ;)
When do you institute the rule -- on day one, in which case it is functionally equivalent to "do not have any checks"? On day 13? Day 7386?
What is the quantum of a "check"? Is Sarbanes-Oxley one "check" or a collection of hundreds of "checks"?
Does the limit apply department-by-department or company-wide? Your sysadmins will naturally tend to employ a lot more formal checks (most of which are hopefully enforced by tiny Perl scripts) than your R&D prototyping team. Do you force the sysadmins to compete with the accounting department for a quota of checks? Do you incentivize sysadmins to evade checks by getting tasks reassigned to the R&D prototyping team, which (alas) can't afford to have any checks?
The last thing you want to do is encourage teams of lawyerly meta-checkers to run around enforcing rules about rules. That's costly, squared.
Yes. ;)
When do you institute the rule -- on day one, in which case it is functionally equivalent to "do not have any checks"? On day 13? Day 7386?
What is the quantum of a "check"? Is Sarbanes-Oxley one "check" or a collection of hundreds of "checks"?
Does the limit apply department-by-department or company-wide? Your sysadmins will naturally tend to employ a lot more formal checks (most of which are hopefully enforced by tiny Perl scripts) than your R&D prototyping team. Do you force the sysadmins to compete with the accounting department for a quota of checks? Do you incentivize sysadmins to evade checks by getting tasks reassigned to the R&D prototyping team, which (alas) can't afford to have any checks?
The last thing you want to do is encourage teams of lawyerly meta-checkers to run around enforcing rules about rules. That's costly, squared.
"When do you institute the rule?"
When you realise you have a (potential) problem. It stops it getting worse.
"What is the quantum of a 'check'?"
More tricky, certainly. That's a potential big problem.
"Does the limit apply department-by-department or company-wide?"
Whatever makes sense.
The idea of the exercise would be basically to:
- build an appreciation throught the organisation that checks have a cost
- provide a (albeit imperfect) mechanism for exercising some control over it
Something doesn't have to be perfect to be of use.
"The last thing you want to do is encourage teams of lawyerly meta-checkers to run around enforcing rules about rules. That's costly, squared."
That's just ISO9001, isn't it? :-)
When you realise you have a (potential) problem. It stops it getting worse.
"What is the quantum of a 'check'?"
More tricky, certainly. That's a potential big problem.
"Does the limit apply department-by-department or company-wide?"
Whatever makes sense.
The idea of the exercise would be basically to:
- build an appreciation throught the organisation that checks have a cost
- provide a (albeit imperfect) mechanism for exercising some control over it
Something doesn't have to be perfect to be of use.
"The last thing you want to do is encourage teams of lawyerly meta-checkers to run around enforcing rules about rules. That's costly, squared."
That's just ISO9001, isn't it? :-)
Well yes, because that assumes that there is some magical amount of checks that must never be exceeded. Some checks are extremely valuable and should be added based on their merits alone.
That's the point I was trying to make. Something like
"always commit and run the tests before going to production" makes a lot of sense, and should be mandated. Even something like "three signed copies of an authorization form" makes sense for some projects if, for example, the cost of damages (say, lost data, downtime or annoyed paying customers) is big enough.
Measuring this, though, is the hard part. It's like trying to do the math behind "no silver bullet" and estimate communication costs (too many unknowns and conditional dependencies).
Measuring this, though, is the hard part. It's like trying to do the math behind "no silver bullet" and estimate communication costs (too many unknowns and conditional dependencies).
How about just measuring how many bits it takes to describe the checks (perl code etc + informal text). Do not make any hard and fast rules - just monitor this metric.
This is what I call "playing defense" instead of "playing offense". By nature, the people at large companies are more concerned with risk aversion than innovation. This is also how managers like to "hold power" over the real innovators by controlling the change management process. What makes a manager more qualified to deploy to production than the programmer?
Their higher up position gives them [the potential for] a better view.
To "make something people want", you need to know your customer's value system: what's important to him and what's not. This can be hard for startups, especially ycombinator types which may be very low budget, because their own value system is so different. Startups start with a company worth about $1 and want to turn it into a company worth about $1,000,000. Therefore startups value the ability to change rapidly -- that's why they want good programmers.
Big companies on the other hand, start with a company worth $100,000,000, and want to turn it into a company worth $150,000,000. They value steady process improvement that doesn't risk their existing revenue -- that's why they want good managers.
It's important to know this if you're trying to sell to a big company as well as if you're trying to compete with a big company.
Big companies on the other hand, start with a company worth $100,000,000, and want to turn it into a company worth $150,000,000. They value steady process improvement that doesn't risk their existing revenue -- that's why they want good managers.
It's important to know this if you're trying to sell to a big company as well as if you're trying to compete with a big company.
In fact, the acquirer would have been better off; not only wouldn't these guys have broken anything, they'd have gotten a lot more done.
That's where you lost me. Sure, there are companies who add checks without thinking of the consequences, but as a company goes from a startup that is not making money to a more mature company that is making money, the cost of breaking things becomes very, very high. That is why most post-startup companies add more checks, to protect their revenue stream. And any company making decent money has a revenue stream that far outweighs the amount of money that the programmers would be willing to pay to have a faster release cycle. And the risk of the company losing their revenue stream for any significant amount of time far outweighs the risk in losing decent programmers because they aren't happy with the length of the release cycle.
You seem to say that these particular programmers would never have broken anything, but I find that impossible to believe unless they are working on something that is not at all complex or is inconsequential to the revenue stream. Every programmer, even the best programmer in the world writes code with bugs in it. The more complex a system is, the more likely there are to be bugs in it and as a company grows and makes money, the more complex their systems will become.
As an example, let's say that you were responsible for a web application that brought in a million dollars of revenue a day and you had these supposedly perfect programmers that never made mistakes who were responsible for the code. Would you let them just throw out whatever code they wanted onto the servers because you trusted them to not make mistakes? Or would you be more reasonable and put some checks in place first to make sure that the application worked correctly?
Obviously there needs to be a sane balance between the risk of breaking the application and the opportunity cost of not updating the application and the frustration of slowing down development is one of the costs that should be weighed, but to say that there shouldn't be any checks added as a company matures is almost laughable.
That's where you lost me. Sure, there are companies who add checks without thinking of the consequences, but as a company goes from a startup that is not making money to a more mature company that is making money, the cost of breaking things becomes very, very high. That is why most post-startup companies add more checks, to protect their revenue stream. And any company making decent money has a revenue stream that far outweighs the amount of money that the programmers would be willing to pay to have a faster release cycle. And the risk of the company losing their revenue stream for any significant amount of time far outweighs the risk in losing decent programmers because they aren't happy with the length of the release cycle.
You seem to say that these particular programmers would never have broken anything, but I find that impossible to believe unless they are working on something that is not at all complex or is inconsequential to the revenue stream. Every programmer, even the best programmer in the world writes code with bugs in it. The more complex a system is, the more likely there are to be bugs in it and as a company grows and makes money, the more complex their systems will become.
As an example, let's say that you were responsible for a web application that brought in a million dollars of revenue a day and you had these supposedly perfect programmers that never made mistakes who were responsible for the code. Would you let them just throw out whatever code they wanted onto the servers because you trusted them to not make mistakes? Or would you be more reasonable and put some checks in place first to make sure that the application worked correctly?
Obviously there needs to be a sane balance between the risk of breaking the application and the opportunity cost of not updating the application and the frustration of slowing down development is one of the costs that should be weighed, but to say that there shouldn't be any checks added as a company matures is almost laughable.
Whenever someone in an organization proposes to add a new check, they should have to explain not just the benefit but the cost. No matter how bad a job they did of analyzing it, this meta-check would at least remind everyone there had to be a cost, and send them looking for it.
To me that's the most important paragraph here. That takes you from "let's make sure this never happens again" to a cost-benefit analysis. The issue is not whether or not to have checks, as much of the discussion here assumes. It's about realizing that not all checks are equivalent, and using that knowledge to get the greatest safety at the lowest cost.
To me that's the most important paragraph here. That takes you from "let's make sure this never happens again" to a cost-benefit analysis. The issue is not whether or not to have checks, as much of the discussion here assumes. It's about realizing that not all checks are equivalent, and using that knowledge to get the greatest safety at the lowest cost.
"... Steve Jobs's famous maxim "artists ship" works both ways. Artists aren't merely capable of shipping. They insist on it. So if you don't let people ship, you won't have any artists ..."
Steve Yegge has had the unenviable luck of working on 3 products all of which have not shipped (cited as business reasons) yet he still works for Google ~ http://blog.stackoverflow.com/2008/10/podcast-25/ So it seems there is something else going on that keeps programmers on board. Is it money, having the freedom to talking about it the process? I don't know.
Steve Yegge has had the unenviable luck of working on 3 products all of which have not shipped (cited as business reasons) yet he still works for Google ~ http://blog.stackoverflow.com/2008/10/podcast-25/ So it seems there is something else going on that keeps programmers on board. Is it money, having the freedom to talking about it the process? I don't know.
I spent 3 months at the end of recent deployment furiously developing an (arguably) game-changing app for the Army's secret intranet. I knew in advance it wouldn't actually get deployed, despite everyone up to the Task Force Commander loving it and touting its importance. The day came when it was "done" (of course I had become attached and there were a million things I wanted to add) and ready to be deployed... then the red tape tied it up until it was time for us to go. As is generally the case, our replacements weren't planning to do things our way (the country has gotten much worse since we left, imagine that) so the project was basically sunk.
I worked on it because I love programming and trying new things. Everything I was doing at the time was new to me so it was fun, engaging, and a learning experience. Plus, it beat the hell out of anything else I could have been doing in that wasteland ;)
I worked on it because I love programming and trying new things. Everything I was doing at the time was new to me so it was fun, engaging, and a learning experience. Plus, it beat the hell out of anything else I could have been doing in that wasteland ;)
"... I worked on it because I love programming and trying new things. Everything I was doing at the time was new to me so it was fun, engaging, and a learning experience. Plus, it beat the hell out of anything else I could have been doing in that wasteland ;) ..."
Good explanation.
I put it as a question as I'm wasn't conclusively sure why people would sign on to assignments that have a probability of failure. But I've since found that smart organisations allow pursuit of training and learning as a reason to keep staff and motivation. I figure the USF know a thing or 2 about motivation. Hope your putting the lessons you learned into some hack your working on :)
Good explanation.
I put it as a question as I'm wasn't conclusively sure why people would sign on to assignments that have a probability of failure. But I've since found that smart organisations allow pursuit of training and learning as a reason to keep staff and motivation. I figure the USF know a thing or 2 about motivation. Hope your putting the lessons you learned into some hack your working on :)
Paul,
Surely if the costs of checks could be "discontinuous," "non-linear," "step-change," or however else you would like to term it, then perhaps the potential cost to the company of NOT having the check in place could also be the same. Larger companies potentially have more to lose by having recurring lapses in supply, or in the ongoing case of your essay, software service. Of course, software/coding may also be a special case. Additionally, it's true that your "committee" example represents a discontinuous change in bureaucracy for that firm, which would make it much more reasonable to expect an equivalent affect on cost.
Surely if the costs of checks could be "discontinuous," "non-linear," "step-change," or however else you would like to term it, then perhaps the potential cost to the company of NOT having the check in place could also be the same. Larger companies potentially have more to lose by having recurring lapses in supply, or in the ongoing case of your essay, software service. Of course, software/coding may also be a special case. Additionally, it's true that your "committee" example represents a discontinuous change in bureaucracy for that firm, which would make it much more reasonable to expect an equivalent affect on cost.
There is a related concern I have about costs piling up due to corporate or managerial mandates. I have the impression that people above us only think about what will make their jobs easier -- what else do they need for us to supply them with so that they will be able to achieve predictability, manage or reduce costs, and generally manage the company better. When they come at it from this angle, it always seems reasonable that they ask us to do more -- provide more reports, track time against projects, report metrics -- when in fact most of what they ask actually interferes with us being able to get real work done. Since they/we do not have adequate productivity measurements in place already, the resulting reduction in productivity is never noticed, hence an invisible cost has been added without a conscious decision or cost/benefit analysis. Since I think most individual contributors recognize that this is taking place, it acts as a disincentive.
In the end I think the solution to both the article's and my concerns are the same: It is always better to ask the members of the team that experienced the problem firsthand what they think the solution would be or what they would do differently next time to avoid the problem. Especially if you phrase it as "what could you or your team have done differently to improve the outcome" and urge them to avoid proposing major process initiatives if at all possible, the improvement will be more practical and lower cost in all senses.
In the end I think the solution to both the article's and my concerns are the same: It is always better to ask the members of the team that experienced the problem firsthand what they think the solution would be or what they would do differently next time to avoid the problem. Especially if you phrase it as "what could you or your team have done differently to improve the outcome" and urge them to avoid proposing major process initiatives if at all possible, the improvement will be more practical and lower cost in all senses.
Programmers are unlike many types of workers in that the best ones actually prefer to work hard. This doesn't seem to be the case in most types of work. When I worked in fast food, we didn't prefer the busy times. And when I used to mow lawns, I definitely didn't prefer it when the grass was long after a week of rain.
Programmers, though, like it better when they write more code. Or more precisely, when they release more code. Programmers like to make a difference. Good ones, anyway.
Salesman like it best when they can sell more. When I used to sell used cars, I loved it best when I could keep pushing cars out of the lot.
Writers like it best when they can write more. They like it best when their imagination can produce thousands of stories.
The difference is that coders usually know what they will tackle and have a strategy so it is easier for them to get to the end point. Also they are almost sure they will hit the end point of a certain problem.
With other professions you cannot predict much. Your next customer is not guaranteed, your next story just does not want to show up in your brain etc....
That being said, Yes, a lot of programmers I know work hard.
Programmers, though, like it better when they write more code. Or more precisely, when they release more code. Programmers like to make a difference. Good ones, anyway.
Salesman like it best when they can sell more. When I used to sell used cars, I loved it best when I could keep pushing cars out of the lot.
Writers like it best when they can write more. They like it best when their imagination can produce thousands of stories.
The difference is that coders usually know what they will tackle and have a strategy so it is easier for them to get to the end point. Also they are almost sure they will hit the end point of a certain problem.
With other professions you cannot predict much. Your next customer is not guaranteed, your next story just does not want to show up in your brain etc....
That being said, Yes, a lot of programmers I know work hard.
Writer are like hackers, yes. But the reason most salesmen like to work hard is that they're paid on commission. That's very different.
Good point.
I suspect that what is really going on here is that workers who love their job actually prefer to work hard AND good programmers love their job. So it is probably true that good programmers actually prefer to work hard, but that is not the whole truth...
Also, since most workers don't love their job, it is probably also true that good programmers are unlike many workers. To say that programmers are unlike many TYPES OF workers we'd have to believe that a higher proportion of programmers love their job than do workers in other types of work. From what I have seen on HN (albeit sample biased) I'd say that is true.
Also, since most workers don't love their job, it is probably also true that good programmers are unlike many workers. To say that programmers are unlike many TYPES OF workers we'd have to believe that a higher proportion of programmers love their job than do workers in other types of work. From what I have seen on HN (albeit sample biased) I'd say that is true.
Yes. I've worked in fast food, and as a cashier at a convenience store, and in both cases, those who were better at the job preferred busier times ("It makes the day fly by, doncha think?").
Summary:
People like to make a difference. Good ones, anyway.
People like to make a difference. Good ones, anyway.
Touche
Not true. Good salesmen love the power, convincing others to do their will. They are more like actors or politicians. You can't fake charisma just because you might get paid more.
You can't do anything well without liking to do it. The difference between hackers and salesmen is that hacking is what hackers like most. Salesmen who have sufficient money seem empirically to prefer playing golf, whereas hackers with money tend to keep hacking.
That doesn't change the fact that while they are working, they prefer to be getting lots of sales.
They like to get paid for it, they can even be greedy. They like to get recognition for it, that can go to far too.
But they also enjoy making the sale.
They like to get paid for it, they can even be greedy. They like to get recognition for it, that can go to far too.
But they also enjoy making the sale.
maybe a better test is whether you'll do it for free (no incentives, moral/material/etc) just for the fun of it
to me, hacking > sales (no, charity/non-profit/giving away money doesn't count -> there's moral incentive there)
to me, hacking > sales (no, charity/non-profit/giving away money doesn't count -> there's moral incentive there)
I think most people want to do a good job and prefer to work hard. They want to see an impact from their labors and want their firm's customers to be delighted with their experience of interacting with the firm's products and service teams. It's our challenge as entrepreneurs to create the right environment for this.
Even in the case of the fast food worker, PG gets it wrong. When it gets slow in a restaurant, you end up cleaning the grease traps or other unpleasant jobs. I'd take the adrenaline rush of juggling a dozen orders on the grill any time.
I found this particular paragraph to be not right somehow:
Programmers are unlike many types of workers in that the best ones actually prefer to work hard. This doesn't seem to be the case in most types of work. When I worked in fast food, we didn't prefer the busy times.
Who are the people who are "best at" fast food? I think perhaps you mean that people who build things get enjoyment out of building -- perhaps more precisely, of seeing their built item in action. I think programmers in particular are used to having short feedback loops between building and trying out, and that pattern of building up and testing as you go until a larger problem is solved is a strong motivation.
I guess I'm claiming that harder-working isn't really the issue. The issue seems to be who is having more fun. And in that sense, your claims still make sense. Part of the fun of building is a short feedback cycle. More checks in the process make the feedback cycle longer, therefore less fun. That explains why teachers' like their jobs. You can see it in someone's face when they learn something new.
Programmers are unlike many types of workers in that the best ones actually prefer to work hard. This doesn't seem to be the case in most types of work. When I worked in fast food, we didn't prefer the busy times.
Who are the people who are "best at" fast food? I think perhaps you mean that people who build things get enjoyment out of building -- perhaps more precisely, of seeing their built item in action. I think programmers in particular are used to having short feedback loops between building and trying out, and that pattern of building up and testing as you go until a larger problem is solved is a strong motivation.
I guess I'm claiming that harder-working isn't really the issue. The issue seems to be who is having more fun. And in that sense, your claims still make sense. Part of the fun of building is a short feedback cycle. More checks in the process make the feedback cycle longer, therefore less fun. That explains why teachers' like their jobs. You can see it in someone's face when they learn something new.
I disagree with the claim that harm to the U.S. IPO market was "not the intention" of the "people who wrote" Sarbanes-Oxley.
I have two pieces of evidence.
First, it was well-known and easy to see that Sarbanes-Oxley would be extremely expensive. It was well-known when passed that its costs fall disproportionately on small public companies. Because IPOs by definition are small public companies, the costs fall disproportionately on them. Thus, it must have been known when enacted that the bill would harm IPOs.
Second, suppose in fact it is the case, as Graham seems to advocate, that Congress "inadvertently" harmed IPOs. His argument is that Congress just accidently happened to overlook the harm of the bill to IPOs. In that case, when the harm to IPOs became factually clear, the bill would have been changed or amended. The fact that the law was not amended even after the harm to IPOs became clear proves that it was Congress' intention all along to harm small public companies (the companies most dangerous to the large corporations who have the most lobbying pull).
I have two pieces of evidence.
First, it was well-known and easy to see that Sarbanes-Oxley would be extremely expensive. It was well-known when passed that its costs fall disproportionately on small public companies. Because IPOs by definition are small public companies, the costs fall disproportionately on them. Thus, it must have been known when enacted that the bill would harm IPOs.
Second, suppose in fact it is the case, as Graham seems to advocate, that Congress "inadvertently" harmed IPOs. His argument is that Congress just accidently happened to overlook the harm of the bill to IPOs. In that case, when the harm to IPOs became factually clear, the bill would have been changed or amended. The fact that the law was not amended even after the harm to IPOs became clear proves that it was Congress' intention all along to harm small public companies (the companies most dangerous to the large corporations who have the most lobbying pull).
Very nice - I think another reason why companies overpay - is because they don't negotiate, they don't check out the hot companies, they allow their employees to make decisions and most have never hired a company - so they go with the best name so they can say they hired them - it looks good on their resume - companies don't realize that most executives - director level and above are doing nothing more than building their resume - and their projects go unchecked, unaudited and bascially the company gets screwed - and frankly it is the company's own fault. Once a company loses it's founding partners, the transition team runs wild with all the changes they want to make and not looking for those things that made the company successful - spending is at all time high, brand suffers because the new team is too busy spending that they lose sight of what is important - the customer - gosh does that sound like AIG?
Another interesting read. I'd like to note that (while it wasn't the focus of the essay), many of the expensive "checks" aren't there for monetary reasons, but rather human safety and the dire consequences of a poor decision.
Working at Motorola on cell phone code could be a great way to give everyone in the modern world a cool new feature - or accidentally break 20+% of all phones out there. A terrible Mickey Mouse movie could destroy a brand. Take it a step further to NASA, vehicle engine design (and safety), oil pipelining and refining, etc. and the cost could be human lives.
That said, I'm very interested in the cost of these "checks" on those industries. Not monetary costs (although important), but rather the cost in stifled future innovation.
Thoughts?
Working at Motorola on cell phone code could be a great way to give everyone in the modern world a cool new feature - or accidentally break 20+% of all phones out there. A terrible Mickey Mouse movie could destroy a brand. Take it a step further to NASA, vehicle engine design (and safety), oil pipelining and refining, etc. and the cost could be human lives.
That said, I'm very interested in the cost of these "checks" on those industries. Not monetary costs (although important), but rather the cost in stifled future innovation.
Thoughts?
I guess so many checks in a big company could be considered the biggest mistake of all?
Well, as companies get larger they can afford less and less to make mistakes. If I'm running a start-up company, I can get away with changing things abruptly, because I have a smaller base of users and I have fewer people relying on my running at production-level. Also, chances are I'll be able to look at user feedback effectively.
Once you get large, and your company is providing sustenance for all your employees, people rely on your product, and you've got too many users for effective feedback-checking, you have to close up, take fewer risks. Because suddenly, people want you to move slowly. They don't want you constantly skyrocketing ahead with their playing backup. Look at any big company - even Google, which was once famous for moving quickly - and you'll see that part of what gives a big company a good reputation is its being "solid." They have to give things up for an advantage.
It's why newspapers are so relied-upon. Of course, now it's what is hurting newspapers the most. They're being beaten by the flexible Internet. But even there, we're seeing a trade-off. Look at the quality of stories by the top writers online and by the top NY Times writers, and the online writers are much more amateur. They're faster, occasionally they're more interesting, but the Internet is thus far not retaining a high level of professionalism among reporting. Similarly, start-ups are much less reliable on the whole than large companies - look at Twitter and its problems, for instance.
So I think that PG's article is right. You can't restrict people and expect them to do as well. However, too much freedom leads to less stability, so it becomes a trade-off. Everything in moderation.
Once you get large, and your company is providing sustenance for all your employees, people rely on your product, and you've got too many users for effective feedback-checking, you have to close up, take fewer risks. Because suddenly, people want you to move slowly. They don't want you constantly skyrocketing ahead with their playing backup. Look at any big company - even Google, which was once famous for moving quickly - and you'll see that part of what gives a big company a good reputation is its being "solid." They have to give things up for an advantage.
It's why newspapers are so relied-upon. Of course, now it's what is hurting newspapers the most. They're being beaten by the flexible Internet. But even there, we're seeing a trade-off. Look at the quality of stories by the top writers online and by the top NY Times writers, and the online writers are much more amateur. They're faster, occasionally they're more interesting, but the Internet is thus far not retaining a high level of professionalism among reporting. Similarly, start-ups are much less reliable on the whole than large companies - look at Twitter and its problems, for instance.
So I think that PG's article is right. You can't restrict people and expect them to do as well. However, too much freedom leads to less stability, so it becomes a trade-off. Everything in moderation.
This thinking has always kind of confused me. Why are customers/users at a big company more important than those at a startup? Just because there's more of them, now you can't make mistakes?
If you have the agility to make rapid production changes, you also have the ability to rapidly rollback. So the argument that larger companies require more checks and testing than startups isn't really valid, especially when you consider the costs.
If you have the agility to make rapid production changes, you also have the ability to rapidly rollback. So the argument that larger companies require more checks and testing than startups isn't really valid, especially when you consider the costs.
If you have the agility to make rapid production changes, you also have the ability to rapidly rollback.
This is just not true. Rollbacks are always more expensive than changes, because you can't rewind time to undo the consequences of having your software be broken for minutes, hours, or days. Worse, in the absence of "checks", the cost of making a production change tends to be roughly constant as the company grows -- it takes the Amazon sysadmin no more time to type "make deploy" than it does me -- but the cost of a rollback scales directly with the size of your company's customer base.
Within a few seconds after Amazon.com breaks S3, thousands of companies begin to lose money, and they lose money second by second until the rollback happens. Even if Amazon is only down for a minute, that's one minute of downtime multiplied by its number of customers. The larger the customer base, the larger the stakes.
And, unfortunately, the cost of downtime is nonlinear. If Amazon goes down for a mere two minutes, hundreds of peacefully sleeping system administrators will get emergency pages from their uptime-monitoring systems. They will get out of bed. They will check their logs and their failover mechanisms. They will lose a lot of sleep, and soak up a bunch of overtime pay, and a lot of their good will towards Amazon will dissipate like the morning dew. Once you lose your reputation for quality it takes a lot of work to get it back.
This is why larger companies have more controls. The controls are in place to try and pass the ever-increasing cost of a rollback back to the team that causes the rollbacks. The reason it seems so gosh-darned expensive to add a trivial feature to your flagship app is that it is expensive: If the average rollback costs $1m in revenue and every new feature is only 95% reliable, every new feature costs the company $50k to deploy.
The secret here is: If you want to deploy changes rapidly, don't work on a product that has a lot of uptime-sensitive customers! Start a different product line, or start a beta program, or found a smaller company.
This is just not true. Rollbacks are always more expensive than changes, because you can't rewind time to undo the consequences of having your software be broken for minutes, hours, or days. Worse, in the absence of "checks", the cost of making a production change tends to be roughly constant as the company grows -- it takes the Amazon sysadmin no more time to type "make deploy" than it does me -- but the cost of a rollback scales directly with the size of your company's customer base.
Within a few seconds after Amazon.com breaks S3, thousands of companies begin to lose money, and they lose money second by second until the rollback happens. Even if Amazon is only down for a minute, that's one minute of downtime multiplied by its number of customers. The larger the customer base, the larger the stakes.
And, unfortunately, the cost of downtime is nonlinear. If Amazon goes down for a mere two minutes, hundreds of peacefully sleeping system administrators will get emergency pages from their uptime-monitoring systems. They will get out of bed. They will check their logs and their failover mechanisms. They will lose a lot of sleep, and soak up a bunch of overtime pay, and a lot of their good will towards Amazon will dissipate like the morning dew. Once you lose your reputation for quality it takes a lot of work to get it back.
This is why larger companies have more controls. The controls are in place to try and pass the ever-increasing cost of a rollback back to the team that causes the rollbacks. The reason it seems so gosh-darned expensive to add a trivial feature to your flagship app is that it is expensive: If the average rollback costs $1m in revenue and every new feature is only 95% reliable, every new feature costs the company $50k to deploy.
The secret here is: If you want to deploy changes rapidly, don't work on a product that has a lot of uptime-sensitive customers! Start a different product line, or start a beta program, or found a smaller company.
S3 is a really bad example because they provide infrastructure. Their customers actually see their entire site go down. Those kinds of companies are the exception. I hope Heroku has rigorous testing and scrutinizes every change, even though they are a startup.
Let's say I own a video site and I want to add threaded comments. If I have 5 users and the site goes down for 5 minutes, those 5 users will get 5 minutes each of annoyance. If I have a million users, each of those users will get 5 minutes of annoyance each also. There is no difference to the user there. So, by adding more checks to make sure the site doesn't go down for 5 minutes when you have more users, you're saying the more users you have, more the important each user becomes. I think that's a strange way of thinking.
(The same is true here of an infrastructure service-- if S3 had 5 users and were more cavalier about their release schedule and broke something, those 5 users would exact the same net effects of downtime as if S3 had 5 million users.)
The awesome benefit of getting threaded comments developed, tested briefly, and pushed in one evening is worth the risk of 5 minutes of downtime compared to the 2 weeks of rigorous testing and approval-by-committee. No matter how many users you have.
Let's say I own a video site and I want to add threaded comments. If I have 5 users and the site goes down for 5 minutes, those 5 users will get 5 minutes each of annoyance. If I have a million users, each of those users will get 5 minutes of annoyance each also. There is no difference to the user there. So, by adding more checks to make sure the site doesn't go down for 5 minutes when you have more users, you're saying the more users you have, more the important each user becomes. I think that's a strange way of thinking.
(The same is true here of an infrastructure service-- if S3 had 5 users and were more cavalier about their release schedule and broke something, those 5 users would exact the same net effects of downtime as if S3 had 5 million users.)
The awesome benefit of getting threaded comments developed, tested briefly, and pushed in one evening is worth the risk of 5 minutes of downtime compared to the 2 weeks of rigorous testing and approval-by-committee. No matter how many users you have.
I used an infrastructure site as an example because the value proposition is easy to understand when you use a site that has a clear and simple monetization strategy. Video sharing sites are arguably an even worse example than S3, because the value of uptime is so hard to perceive or compute. It's likely that even Twitter doesn't understand the true value of a customer-hour of Twitter uptime, because the site isn't monetized and so much of the value is concentrated in the brand. Measuring that is like voodoo, only less empirical. ;)
If I have 5 users and the site goes down for 5 minutes, those 5 users will get 5 minutes each of annoyance. If I have a million users, each of those users will get 5 minutes of annoyance each also. There is no difference to the user there.
No, but there is a big difference for you! If a user is worth a dollar per year, the five-user site is worth five bucks per year, but the million-user site is worth a million bucks. If each patch to your code causes 0.1% of users to abandon your product (a number which depends on the odds that a patch will cause a rollback, and on the odds that a rollback will annoy a user enough to make them leave), patching a 5-user site costs you half a cent per year on average (most likely it has no perceptable cost, since odds are no users will leave) but each patch to a million-user site costs you $1000 per year in revenue. And that's just the linear cost. There are nonlinear consequences: one or zero annoyed users is nothing to worry about -- unless that user is Michael Arrington -- but a clique of 1000 annoyed users is potentially a movement: a critical mass of people who will all start complaining about your company on Twitter on the same day, potentially costing you your next 10,000 or 100,000 or 1 million users while simultaneously empowering your competitors, who may begin building the site that will take you down by poaching those dissatisfied users.
This is just the flip side of scalability. As a programmer you enjoy mighty economies of scale: Running a site with a million users is more expensive than running a single-user site, but it is much less than a million times as expensive. But this leverage also applies to your mistakes: a mistake that costs you a dollar when your site is small might cost you $1,000,000 when your site is big. And it's the same mistake! Typos are just as easy to make on big sites as on small ones.
Obviously, this doesn't mean that you shouldn't ever change the site. Presumably each and every one of your patches is valuable, and will bring in revenue to pay for its own insurance premiums. Right? :) But you do need to think about that calculation, because you do occasionally make mistakes. As your userbase grows, you may wish to test each patch on a subset of users to be sure they will really like it, and that the additional revenue is really going to be there. You may wish to institute tests and internal audits that lower the risk of rollbacks, or failover mechanisms to lower the cost of rollbacks. And before long, lo, you will be that which you deplore: A company with a bunch of annoying internal controls! But at least you'll have revenue to console yourself with.
If I have 5 users and the site goes down for 5 minutes, those 5 users will get 5 minutes each of annoyance. If I have a million users, each of those users will get 5 minutes of annoyance each also. There is no difference to the user there.
No, but there is a big difference for you! If a user is worth a dollar per year, the five-user site is worth five bucks per year, but the million-user site is worth a million bucks. If each patch to your code causes 0.1% of users to abandon your product (a number which depends on the odds that a patch will cause a rollback, and on the odds that a rollback will annoy a user enough to make them leave), patching a 5-user site costs you half a cent per year on average (most likely it has no perceptable cost, since odds are no users will leave) but each patch to a million-user site costs you $1000 per year in revenue. And that's just the linear cost. There are nonlinear consequences: one or zero annoyed users is nothing to worry about -- unless that user is Michael Arrington -- but a clique of 1000 annoyed users is potentially a movement: a critical mass of people who will all start complaining about your company on Twitter on the same day, potentially costing you your next 10,000 or 100,000 or 1 million users while simultaneously empowering your competitors, who may begin building the site that will take you down by poaching those dissatisfied users.
This is just the flip side of scalability. As a programmer you enjoy mighty economies of scale: Running a site with a million users is more expensive than running a single-user site, but it is much less than a million times as expensive. But this leverage also applies to your mistakes: a mistake that costs you a dollar when your site is small might cost you $1,000,000 when your site is big. And it's the same mistake! Typos are just as easy to make on big sites as on small ones.
Obviously, this doesn't mean that you shouldn't ever change the site. Presumably each and every one of your patches is valuable, and will bring in revenue to pay for its own insurance premiums. Right? :) But you do need to think about that calculation, because you do occasionally make mistakes. As your userbase grows, you may wish to test each patch on a subset of users to be sure they will really like it, and that the additional revenue is really going to be there. You may wish to institute tests and internal audits that lower the risk of rollbacks, or failover mechanisms to lower the cost of rollbacks. And before long, lo, you will be that which you deplore: A company with a bunch of annoying internal controls! But at least you'll have revenue to console yourself with.
But I think what Dustin is saying (correct me if I'm wrong) is that the multiplier applies both ways. And that the total cost of making a 5 minute downtime mistake, even to a million users, could easily be outweighed by the benefits of releasing a product/feature/site 2 weeks early. In most cases, I think large companies are risk adverse instead of risk neutral to situations like this.
I agree with both of you that it varies considerably based on what the site does (infrastructure, videos, games, etc).
I agree with both of you that it varies considerably based on what the site does (infrastructure, videos, games, etc).
In most cases, I think large companies are risk adverse instead of risk neutral to situations like this.
I'm not going to argue with that. Just because a certain increase of caution is rational doesn't mean that caution isn't being overapplied in many cases, just as PG suggests in his original post.
I'm not going to argue with that. Just because a certain increase of caution is rational doesn't mean that caution isn't being overapplied in many cases, just as PG suggests in his original post.
No, it's that most people don't want to use a product that changes often, if at all. They get it for what it is and if it changes, it means disrupting their way of using it. When you're working with a smaller product, you can iterate more and work closely with the people using it. When it gets larger, any change you make will alienate some users, so you have to plan to minimize that alienation.
You can change and roll back features quickly, but that gives an impression of instability among users. If things are constantly changing, they'll seek out something more stable. It's "lowest common denominator" thinking. It results in something that nobody dislikes, and that's the goal of larger groups. Niche companies are able to work much better, but even then there's some slowdown.
You can change and roll back features quickly, but that gives an impression of instability among users. If things are constantly changing, they'll seek out something more stable. It's "lowest common denominator" thinking. It results in something that nobody dislikes, and that's the goal of larger groups. Niche companies are able to work much better, but even then there's some slowdown.
I think this is particularly important given that often the biggest cost in using a new product\tool is learning to use it well... if it's changing all the time, with no release schedule or no attempt to stabilize the main release via betas\QA\etc, then it gets very frustrating to learn.
I think this phenomenon is caused by the culture of fear among most employees. Screwing up badly will get you fired, the greatest fear of all. Most employees have heavy mortgages and children, and require a consistent and predictable income. (This state of things is a problem in itself, but that's another topic).
The big issue is that the level of fear increases as you move up the command structure.
Let's assume one of your startups has crashed and burned. Something really spectacular must have happened if this makes potential customers of your next startup spurn you. Taking risks may have consequences, but they are temporary. But if you are a high-level manager for a large company, spectacular failure will make you unemployable, or in the best case push you several steps down the ladder - steps which took years of boredom and politics to climb, and which you will never get back. Failure is punished disproportionately, success is rarely compensated. Conservatism is the correct choice for each individual actor.
We all know bureaucracy, culture and politics - existing companies won't change in this area.
The big issue is that the level of fear increases as you move up the command structure.
Let's assume one of your startups has crashed and burned. Something really spectacular must have happened if this makes potential customers of your next startup spurn you. Taking risks may have consequences, but they are temporary. But if you are a high-level manager for a large company, spectacular failure will make you unemployable, or in the best case push you several steps down the ladder - steps which took years of boredom and politics to climb, and which you will never get back. Failure is punished disproportionately, success is rarely compensated. Conservatism is the correct choice for each individual actor.
We all know bureaucracy, culture and politics - existing companies won't change in this area.
If you have the agility to make rapid production changes, you also have the ability to rapidly rollback
It doesn't matter. If you make a mistake with other people's money (e.g. calculating a payment wrong, crediting/debiting the wrong person, etc), even if you put it right quickly, they'll start losing trust and looking at your competitors.
It doesn't matter. If you make a mistake with other people's money (e.g. calculating a payment wrong, crediting/debiting the wrong person, etc), even if you put it right quickly, they'll start losing trust and looking at your competitors.
How about a check immunity system?
For example, competent people who knows their own weaknesses well and often check their decisions with others should be immune to checks
Good programmers often let other good programmers check their codes. Good writers often let other good writers proof read. Good managers often discuss his decisions with his employees before deploy them. They should be check immune.
You can even pass out check-immune badges to encourage non-check-immune people to be more critical of themselves :)
Of course, this system itself is a check. Who decides who will get check-immunity? What's the successful-decision-ratio threshold? I think it's possible to implement a light-weight system little cost.
For example, competent people who knows their own weaknesses well and often check their decisions with others should be immune to checks
Good programmers often let other good programmers check their codes. Good writers often let other good writers proof read. Good managers often discuss his decisions with his employees before deploy them. They should be check immune.
You can even pass out check-immune badges to encourage non-check-immune people to be more critical of themselves :)
Of course, this system itself is a check. Who decides who will get check-immunity? What's the successful-decision-ratio threshold? I think it's possible to implement a light-weight system little cost.
Yes, startups make their job better than big companies. But I think that it is a part of a more general rule: people work productively in OPEN systems, and the open systems themselves encourage working hard. Open source projects are developed more rapidly, because it is hard to imagine the system which is more open. Both users and developers control the system, and every user can become a developer. In startups we have a limited number of people who can control the process of development, but they still controll it. In other words, the system is open for founders and employees. Big companies work unefficiently because they are too closed, and a great number of checks is only a part of it.
Nice writeup on half an idea. Now, time for some release engineering. How do you give good coders free (enough) reign while keeping tabs on the stupidities of the lesser ones?
In a startup, it's acceptable for service to be accidentally down for short periods while the geniuses running the show fix their latest cockup. In large companies, it isn't. How do you get the best of both worlds?
It is not by removing all safety interlocks. Sysadmins, especially good ones, can tell you in excruciating detail why not. Most programmers are not very good sysadmins, or at least not as good as they might like to believe. Coincidence?
In a startup, it's acceptable for service to be accidentally down for short periods while the geniuses running the show fix their latest cockup. In large companies, it isn't. How do you get the best of both worlds?
It is not by removing all safety interlocks. Sysadmins, especially good ones, can tell you in excruciating detail why not. Most programmers are not very good sysadmins, or at least not as good as they might like to believe. Coincidence?
This would be interesting as a concept to develop past the circumstance of programmers at work.
I would call it part of institutionalisation. You have a need to be effective via policies, checks, procedures & such. This replaces the shoot from the hip of small groups.
I think you can apply this to schools that need to teach approved courses with approved grading. Then it gets worse when they need examinations that are to be applied across an entire country.
I'm sure there are principals that can be extracted & applied to lots of places. The costs can be very serious.
I would call it part of institutionalisation. You have a need to be effective via policies, checks, procedures & such. This replaces the shoot from the hip of small groups.
I think you can apply this to schools that need to teach approved courses with approved grading. Then it gets worse when they need examinations that are to be applied across an entire country.
I'm sure there are principals that can be extracted & applied to lots of places. The costs can be very serious.
My experience is that extensive procedure comes because the small company start up approach doesn't scale to large companies. Every one I have worked at that has transitioned from from 10s to 1000s of people was failing because it tried to continue as if it was 10 wizards at it. The wizardy actually hindered others because it was opaque to them.
Interestingly the biggest company I worked for actually ran the best because perhaps its roots being 80+ years before any of the others would allow the significant difference to be tracked.
Interestingly the biggest company I worked for actually ran the best because perhaps its roots being 80+ years before any of the others would allow the significant difference to be tracked.
When I was an automotive engineer the industry instituted a Japanese "check" called Design Failure Mode Effects Analysis. For every component the engineers would have to predict the ways the design could fail in testing and what steps would have to be taken to fix them. Time spent doing DFMA's added significantly to the cost of producing a new part and worked counter to weight optimization. Do you think, twenty years of DFMA's later, that automotive design is more efficient?
"The purpose of the committee is presumably to ensure that the company doesn't waste money. And yet the result is that the company pays 10 times as much."
Well, only if they still buy every product, which would defeat the purpose of having a gatekeeper committee. If, instead, the committee approves only 1 in 25 products that would otherwise have been bought, the committee saves them money (leaving aside the cost of the time of the committee-members).
Well, only if they still buy every product, which would defeat the purpose of having a gatekeeper committee. If, instead, the committee approves only 1 in 25 products that would otherwise have been bought, the committee saves them money (leaving aside the cost of the time of the committee-members).
" And you know what? It would have been perfectly safe to let them. In fact, the acquirer would have been better off; not only wouldn't these guys have broken anything, they'd have gotten a lot more done."
My experience as a manager of a team of programmers for many years is that for sure they would love to be able to write and release w/o a QA process, they do in fact take the site down when this is allowed!
My experience as a manager of a team of programmers for many years is that for sure they would love to be able to write and release w/o a QA process, they do in fact take the site down when this is allowed!
Checks have a cost, for sure.
What about Eric Reis's split-testing axiom?
If in a startup the developer/coder is basically the customer of the enterprise because the cost of the check is paid by the rising frustration of the coder, then should split-testing be looked upon as a check?
Even if it is, the idea of an available metric to backup claims of efficacy is really compelling.
For me this is all about "Mental Poisons".
Mental Poisons are ideas that harm people, life and the universe. They are remarkably common.
Paul expresses this one very well. There's a lot more out there too.
Fossilised heirarchies and thought-free-rule-driven organisations are great for the student of poisons.
Mental Poisons are ideas that harm people, life and the universe. They are remarkably common.
Paul expresses this one very well. There's a lot more out there too.
Fossilised heirarchies and thought-free-rule-driven organisations are great for the student of poisons.
Thanks for this article Paul. Now i really know the why behind the what i observed working for a big company
http://computinglife.wordpress.com/2008/11/14/hands-free-or-...
http://computinglife.wordpress.com/2008/11/14/hands-free-or-...
Two weeks seems a bit much, but once you have a significant client base of paying customers, you simply must have some form of production control and QA. Even good developers make mistakes, and letting everyone just push their changes to prod over lunch is a recipe for pissed off clients.
As a previous startup coder who was acquired by a much larger company, I can truly say you've hit the nail on the head. I never minded working 70 hour weeks before, but now I have a hard time punching the clock 35 hours a week. I hate my new higher paying job.
Oh, wow. Yes. I've often thought that a lot of this process is just to keep the lowest common denominator from messing things up. I want to spend my life surrounded with people that don't need this, because then we could do so many more worthwhile things.
It seems that checks affect journalism too. "Citizen journalism" allows people to report much more quickly than traditional media. Bloggers don't have to submit their posts to editors and fact-checkers, the way broadcast and print journalists do.
Hi Paul
I'm surprised you didn't tie this in to due diligence checks done by VCs, and its impact on startups, etc. The parallels are strong, and you've commented on that impact before (at FOWA in 2007, I think).
Regards, Terry Jones
I'm surprised you didn't tie this in to due diligence checks done by VCs, and its impact on startups, etc. The parallels are strong, and you've commented on that impact before (at FOWA in 2007, I think).
Regards, Terry Jones
[deleted]
I'd say that the reason for bureaucracy in large organizations is that central management doesn't trust middle managers. Whereas in a startup there are a total of three bosses, and all trust each other.
The cost of the checks should balance the cost of the risks. Those are not always determinate or exactly estimable, but, statistically, that would be a simple basis of a rational approach. Wouldn't it?
Small side node:
I think it is very cool to have the comments here instead of his website. It is nice to have them consolidated and not some comments here, some there.
I think it is very cool to have the comments here instead of his website. It is nice to have them consolidated and not some comments here, some there.
The movie "Meet the Robinsons" makes a point to children that we make the most progress when we are free to fail. Thanks for making the same point for grownups.
This is exactly why we don't respond to RFPs any more.
pg worked in fast food?
Baskin Robbins, in HS.
What was your favorite flavor?
When PG worked there they only had 7 flavors :).
[deleted]
it was pre-artix ;-)
gotta start somewhere.
I guess the best check (and benefit) for startups are commit mistakes early, commit mistakes often.
i love the last paragraph of that post Paul. so true.
Wonderful. Well thought, researched and valuable for large and small companies alike. http://wildwildeastdailies.blogspot.com