Shipping is your company’s heartbeat (2013)(blog.intercom.com)
blog.intercom.com
Shipping is your company’s heartbeat (2013)
https://blog.intercom.com/shipping-is-your-companys-heartbeat/
22 comments
The other side of this coin is when you're trying to push out the first version of an MVP. Devs can get so enamored by the process of designing and perfecting your continuous delivery process that it distracts them from the actual product, not to mention wasted time writing tests for features that will only get thrown out after the first redesign. I call it continuously delivering nothing.
The basic version of continuous delivery is devs have access to the server, and push code there when it's done. You don't need a process while you're building an MVP, or it's not very minimum.
It sounds nice in theory, and maybe you could make it work. But in reality, distracting a developer from whatever they were doing has a cost. Add up ten 1 minute fixes and your whole day goes by.
From the customer's perspective though having a fix out in a couple of minutes is the golden standard. I guess that's were team organization and planning comes into play. Having a couple of dedicated resources to address these ad hoc issues, without distracting the people working on larger projects.
From the customer's perspective though having a fix out in a couple of minutes is the golden standard. I guess that's were team organization and planning comes into play. Having a couple of dedicated resources to address these ad hoc issues, without distracting the people working on larger projects.
I think the dichotomy you introduce here is false. My experience with continuous deployment is that the faster you can ship, the smaller projects get.
Instead of bundling a lot of hypotheses about possibly-valuable changes up into a big release, you focus more on the next step in the direction you're going. You release those steps one by one. (User-visibility of larger features is controlled by feature flags, but you ship and user-test in small slices.)
In that context, the whole notion of distraction is different. You have to hold a lot less state in your head. Instead that state lives in your unit tests, your acceptance tests, your backlog, and the world at large. If a given developer is releasing a couple of times a day, it's just not a big deal to find a natural stopping point, release something small, and then pick up the next slice of some longer-term effort.
Instead of bundling a lot of hypotheses about possibly-valuable changes up into a big release, you focus more on the next step in the direction you're going. You release those steps one by one. (User-visibility of larger features is controlled by feature flags, but you ship and user-test in small slices.)
In that context, the whole notion of distraction is different. You have to hold a lot less state in your head. Instead that state lives in your unit tests, your acceptance tests, your backlog, and the world at large. If a given developer is releasing a couple of times a day, it's just not a big deal to find a natural stopping point, release something small, and then pick up the next slice of some longer-term effort.
I know the feeling -- "I have to finish this <big feature we're trying to release> but all these pesky customers keep getting in the way."
Your customers should be the highest priority, all the time. If they hit a bug that prevents them from accomplishing a task in your software, that matters 1000x more than the feature they don't even care about/use/want yet. Some big project the customer doesn't know about isn't why your customer is your customer.
Obviously trivial stuff (i.e. a misspelled word or an out-of-alignment UI element) ought not require an engineer to drop everything. But "I can't edit my document" or "I can't sign up because I have a weird email" or "I can't save or send this invoice" -- all of that SHOULD require an engineer to drop everything to fix it -- right that minute. Your customers are paying you money. If there's a fly in their soup, you ought not be saying, "If you order soup tomorrow, we'll make sure there's no fly in it. In the meantime, fuck you, I have to prepare next week's menu."
My point -- if there's a bug that's blocking a customer from doing what they're paying you to enable them to do, then that's all that matters at that moment, anything less and you're either understaffed, or your priorities are all wrong. Worry about today's customers today, worry about tomorrow's customers later.
I guess some companies have the luxury of prioritizing their roadmap over the needs of the people that actually are paying your company money -- but mine sure doesn't. We treat every customer's problem as the most important thing, because it is -- they're the ones that give us money. It would be like a restaurant ignoring a customer's request for a refill of water because they're busy planning next week's menu. You do that enough times and there'll be no need for next week's menu.
The customer isn't always right, but they should always come first.
Your customers should be the highest priority, all the time. If they hit a bug that prevents them from accomplishing a task in your software, that matters 1000x more than the feature they don't even care about/use/want yet. Some big project the customer doesn't know about isn't why your customer is your customer.
Obviously trivial stuff (i.e. a misspelled word or an out-of-alignment UI element) ought not require an engineer to drop everything. But "I can't edit my document" or "I can't sign up because I have a weird email" or "I can't save or send this invoice" -- all of that SHOULD require an engineer to drop everything to fix it -- right that minute. Your customers are paying you money. If there's a fly in their soup, you ought not be saying, "If you order soup tomorrow, we'll make sure there's no fly in it. In the meantime, fuck you, I have to prepare next week's menu."
My point -- if there's a bug that's blocking a customer from doing what they're paying you to enable them to do, then that's all that matters at that moment, anything less and you're either understaffed, or your priorities are all wrong. Worry about today's customers today, worry about tomorrow's customers later.
I guess some companies have the luxury of prioritizing their roadmap over the needs of the people that actually are paying your company money -- but mine sure doesn't. We treat every customer's problem as the most important thing, because it is -- they're the ones that give us money. It would be like a restaurant ignoring a customer's request for a refill of water because they're busy planning next week's menu. You do that enough times and there'll be no need for next week's menu.
The customer isn't always right, but they should always come first.
> Having a couple of dedicated resources to address these ad hoc issues, without distracting the people working on larger projects.
Can't people take turns being oncall? It's disruptive to one's life to not be able to be away from a computer for more than N minutes, so only having to do this for limited periods is important.
And while people are not oncall they can work on development and finishing fixing any problems they discovered while oncall.
Can't people take turns being oncall? It's disruptive to one's life to not be able to be away from a computer for more than N minutes, so only having to do this for limited periods is important.
And while people are not oncall they can work on development and finishing fixing any problems they discovered while oncall.
> In a company where you ship continuously, you see this and think “Simple – I’ll tweak a test and a regex pattern, get a quick code review from my buddy beside me, merge to mainline, and 1 minute later when it’s deployed to production, reply to the customer: ‘Sorry about this, it’s fixed now, thanks for letting us know.’”
This sounds like a really bad idea. If it's not a critical bug impacting all your customers, do you have to fix it in 1 minute? Once your company gets bigger, you would want to do proper code review and QA, so your 'one-line fix' doesn't accidentally bring down the whole site.
This sounds like a really bad idea. If it's not a critical bug impacting all your customers, do you have to fix it in 1 minute? Once your company gets bigger, you would want to do proper code review and QA, so your 'one-line fix' doesn't accidentally bring down the whole site.
If it only takes 1 minute to push a fix, it usually will take about the same amount of time to revert it, if it's broken. Then you can do the fix again, but with more care and testing. If all goes well, you only have to think about this issue for 10 minutes, and then get on with your life, you don't have to spend 10 minutes a day following it through the 2 week deployment process and only then notify the customer that they can finally use the product.
Also, if you think a fix needs more care, the ability to ship quickly doesn't impede doing it slowly; but in many companies there is simply no ability to ship fast, regardless of the type of bug.
Also, if you think a fix needs more care, the ability to ship quickly doesn't impede doing it slowly; but in many companies there is simply no ability to ship fast, regardless of the type of bug.
> If it only takes 1 minute to push a fix, it usually will take about the same amount of time to revert it, if it's broken.
Unless it caused data corruption.
Unless it caused data corruption.
Yes, that's why I said usually. It's also why I said, you can take it slow if you're concerned. And something that would possibly cause data corruption is a good thing to take more time with.
In a perfect scenario you would have set up continuous integration that automatically runs all your unit, integration and end-to-end tests. Ideally your coverage with tests is so good that you can rely on a successful integration of the bug fix commit.
This of course is a ideal scenario that you have to work on seriously to be very close to it.
This of course is a ideal scenario that you have to work on seriously to be very close to it.
Continuous deployment works differently. Your approach focuses on reducing MTBF; continuous deployment focused on reducing MTTR.
For CD to work, you can't really depend on QA for regression testing. You need to have really good automated test suites. You still do want exploratory QA, but that doesn't have to happen before release.
Code review works similarly. As your quote says, he did proper code review. It was a very small change, so it doesn't take long. Strong test coverage means code review is less about finding possible bugs and more about design issues. There's no urgency to getting those before release; people can watch changelogs just as well as they can service a review queue. And with frequent releases, developers return to important code often, meaning that design improvements happen over time.
For CD to work, you can't really depend on QA for regression testing. You need to have really good automated test suites. You still do want exploratory QA, but that doesn't have to happen before release.
Code review works similarly. As your quote says, he did proper code review. It was a very small change, so it doesn't take long. Strong test coverage means code review is less about finding possible bugs and more about design issues. There's no urgency to getting those before release; people can watch changelogs just as well as they can service a review queue. And with frequent releases, developers return to important code often, meaning that design improvements happen over time.
You deploy gradually, initially to 1% or so of requests. If your metrics, KPIs, etc. don’t suffer, it continues to roll-out until 100% of requests are using that code-path - within a few hours or so.
Takes a bit of infra effort, but for a company like Intercom, that’s very easily achievable.
Takes a bit of infra effort, but for a company like Intercom, that’s very easily achievable.
this is easy and doable if your change is a front end change.
good luck trying this when its a backend change involving distributed threads, locks, resources, etc where production load is 100x more than staging/test.
good luck trying this when its a backend change involving distributed threads, locks, resources, etc where production load is 100x more than staging/test.
If the cost of downtime is high enough you can mirror some or all traffic onto a shadow copy of the infrastructure.
Obviously in those cases, you’d take more than a minute to review prior to deployment...
There are plenty of backend changes where your list of complexities do not apply.
There are plenty of backend changes where your list of complexities do not apply.
what would be some good resources to learn how to build such infrastructure? I've seen it as a feature of various platforms but would imagine it could become complex otherwise.
It wouldn’t be practical without some kind of underlying infrastructure orchestration system. Obviously some companies have gone the extra step to actually build these systems - e.g. Google made Kubernetes. For any not-huge companies, I think it pretty much just falls to choosing one of the many existing tools.
Obviously it is possible to just throw together a bunch of scripts and vaguely make it all happen, I’m not sure I’d be comfortable to do that in production.
Obviously it is possible to just throw together a bunch of scripts and vaguely make it all happen, I’m not sure I’d be comfortable to do that in production.
The simple way to do this can apply if you're mostly doing things in the context of a user. Write the change to check if a hash of the user identifier is less than a target; increase the target over time.