I Don't Want to Be on Call Anymore(honeycomb.io)
honeycomb.io
I Don't Want to Be on Call Anymore
https://www.honeycomb.io/blog/devops-on-call/
397 comments
> If you are working on-call and not making big bucks
Both of those are very relative. I've seen on-call which means daily events, and on-call which requires something once a year. There's a very different level of compensations I'd be happy with in each case.
Both of those are very relative. I've seen on-call which means daily events, and on-call which requires something once a year. There's a very different level of compensations I'd be happy with in each case.
I bill my hourly rate for all on-call hours regardless of if I'm called. If my employer wants any amount of control over my time (sobriety, proximity, etc) then they pay my regular hourly rate for it.
Most will not accept these terms, and that's the goal.
Most will not accept these terms, and that's the goal.
That's the way to go. On-call is bearable if you get paid for it. As a contractor who charged by the hour I was perfectly fine with sitting in useless meetings or fixing things at night that weren't even my fault. Doing this without extra compensation is just not OK.
And of course it should be that way: if they want your time (you not being able to do other jobs, you not being able to relax, etc) they should pay for it.
Forcing them to think in your hours can also have the nice side effect that they get their communications in order.
Forcing them to think in your hours can also have the nice side effect that they get their communications in order.
This is on point. I think the reason people do it without adequate pay is that some people like the feeling of being needed.
Actually it’s the inability to say a simple “no” in most cases.
> I don't accept jobs that want me to be on call unless fuck you money is involved, and so far nobody has offered it to me. The product that drives most of my employer's revenue is a gigantic pile of tech debt, and there are some people on call whose lives are kind of miserable.
Yeah this is the right answer. I know a lot of people who are earning more than I am in exchange for being on-call (or being de-facto on call, ahem) and it just ain't worth it.
Yeah this is the right answer. I know a lot of people who are earning more than I am in exchange for being on-call (or being de-facto on call, ahem) and it just ain't worth it.
What rates are typical for being on-call for a week? (my team has a rotating weekly roster, Australian team).
In Finland the collective-agreement that covers IT work defines this. The way I've been paid has been the same across several companies:
You get paid 50% of your normal hourly salary for every hour you're "available".
You get paid 200% of your normal hourly salary for every hour you're working on an issue, incident, outage, or problem.
So being on-call is pretty lucrative, given that you'd usually do it for a seven day period. (i.e. 9am Monday to 9am the following Monday.)
You get paid 50% of your normal hourly salary for every hour you're "available".
You get paid 200% of your normal hourly salary for every hour you're working on an issue, incident, outage, or problem.
So being on-call is pretty lucrative, given that you'd usually do it for a seven day period. (i.e. 9am Monday to 9am the following Monday.)
Australian Team, We get $20 per day (pre tax) oncall allowance, then time and 1/2 for any work done, time calculated to when you can go back to doing what you were previously. Generally works out to a minimum of 30 minutes overtime for any calls.
If you don't get 10 hours continuous break before the next work day, you fall into fatigue leave and either get paid overtime for the next days work or are stood down till you have had a 10 hr break.
Still doesn't stop our operations team being completely snowed with both workday and oncall work, but they can make a fairly good bonus wage out of it.
If you don't get 10 hours continuous break before the next work day, you fall into fatigue leave and either get paid overtime for the next days work or are stood down till you have had a 10 hr break.
Still doesn't stop our operations team being completely snowed with both workday and oncall work, but they can make a fairly good bonus wage out of it.
Your ops team might be at my old employer. Ops team had the above benefits but no one in SaaS Engineering (devs) got anything as second level support.
Yeah, we seem to contract second level support from the lowest bidder. Tick, we have a support contract, regardless of how dubious it's value.
We get a flat ~800$ for an on call week (Thursday to Thursday) and Friday is off. Nothing extra for working night, but you’re not expected to work a full day if you had an incident the night before.
At my previous employer we got $4/hour for every hour on call outside normal business hours. So $64 per work day or $96 on a weekend or holiday or $512 for a typical week. Paid quarterly.
One of my reports pointed out that he made more from on call each year than the difference between an average and outstanding performance bonus. He wasn’t wrong.
One of my reports pointed out that he made more from on call each year than the difference between an average and outstanding performance bonus. He wasn’t wrong.
$35 per day for me. 1.5x for the first 3 hours. 2.5x for every hour after.
Minimum call out fee of 2 hours regardless of the length of the call.
Also in Australia. This only started when I joined as an SRE.
Prior to that the Devs just said no it'll stay broken till we come back to work and that was largely acceptable. The on call rates only started as our reliability became more of a business need.
Minimum call out fee of 2 hours regardless of the length of the call.
Also in Australia. This only started when I joined as an SRE.
Prior to that the Devs just said no it'll stay broken till we come back to work and that was largely acceptable. The on call rates only started as our reliability became more of a business need.
what would you say is big bucks?
A surgeon friend of mine gets $1000 per day that he's not working but on-call.
And what does it go to if he gets called in?
I don’t know but it’s a hell of a lot more than anything I make.
FWIW the original definition of "fuck you money" was enough money to be able to say "fuck you" to anyone, which I conservatively estimate at a few billion, but most likely more than that.
It seems these days to just mean "multiple millions", which reduces the utility of the term.
It seems these days to just mean "multiple millions", which reduces the utility of the term.
A few millions are sufficient for you to be able to say "fuck you" to any employer or person you theoretically depend on, while still being able to live a comfortable live and - possibly - defend your right in court, even if it does not make economical sense to do so. That's what most people mean.
Having a few billion gives you some additional fuck you privileges - police officers, world leaders, other billionaires -, but you probably won't get on a level where you can say that to anyone and not fear repercussions. Also, if you can live happily on your own, you can already tell them that if you have a few million to your name.
Having a few billion gives you some additional fuck you privileges - police officers, world leaders, other billionaires -, but you probably won't get on a level where you can say that to anyone and not fear repercussions. Also, if you can live happily on your own, you can already tell them that if you have a few million to your name.
In the US, for legal protection, I would say $10M is a good number based on legal costs I have seen. Maybe $5M works too, but I would say $5M protects you from healthcare costs and not being able to earn income due to health issues, but a little more is nice for paying lawyers if you are also not able to work at the same time.
There's always a bigger fish, but a few million is sufficient for almost every interaction most people will have in their lifetimes.
Unless you live in the US and get cancer.
This got downvoted, but I think is a legitimate concern. Having a terminal disease in the US can drain bank accounts quickly. It’s a real hazard that can destroy families / relationships.
This misses the key point that on-call itself is an abusive practice regardless of how well an organization practices it.
If an organization thinks their systems should be available 24x7 they should staff people 24x7.
There is no situation where it’s ok to contact someone to do work outside of their work hours.
If an organization thinks their systems should be available 24x7 they should staff people 24x7.
There is no situation where it’s ok to contact someone to do work outside of their work hours.
"There is no situation where it’s ok to contact someone to do work outside of their work hours."
Here's one. I get paged for urgent fixes maybe three times per year. To staff all three shifts with senior engineers to cover for that would be enough to make the company unprofitable and lose many jobs. I don't much like being woken in the middle of the night for such things, but it's clearly necessary and not abusive.
Here's one. I get paged for urgent fixes maybe three times per year. To staff all three shifts with senior engineers to cover for that would be enough to make the company unprofitable and lose many jobs. I don't much like being woken in the middle of the night for such things, but it's clearly necessary and not abusive.
Yeah,my issue with being on call is that i can't ever really let loose if I'm expected to be contactable anytime 24/7...what if I'm out with friends, drunk, high, on the side of a mountain? Etc.
My hours are 9-5, after that is entirely me time and if you expect me to br contact able and in s state of contribute you're going to be disappointed.
No amount of money is worth losing that me time.
My hours are 9-5, after that is entirely me time and if you expect me to br contact able and in s state of contribute you're going to be disappointed.
No amount of money is worth losing that me time.
Usually, being "on call". should mean no alcoho, and have inet + computer closed by i.e. 5-15min or so.
which also means that this should be time that is paid on some hourly rate for the lower quality of live during the on-call time.
Also, there should be a rotation, with enough people to make this bearable.
which also means that this should be time that is paid on some hourly rate for the lower quality of live during the on-call time.
Also, there should be a rotation, with enough people to make this bearable.
Are you describing the current state of on-call? Because that is what it is.
There should be no rotation; there should be no on-call in the first place. If hiring staff to work normal hours is too expensive for the company, then the company has bigger problems than a service going down.
There should be no rotation; there should be no on-call in the first place. If hiring staff to work normal hours is too expensive for the company, then the company has bigger problems than a service going down.
I'm confused. What's your plan for fixing Gmail on a Saturday night? Hope one of the devs doesn't have a social life?
Google is big enough to have teams around the world, all in their own daytime zone.
It's still the weekend wherever you are if it's Saturday in North America
If you have teams in Israel, you at least can cheat a bit on Sunday v.s Friday.
And having at least one daytime rotation agree that "well, you're expected to be on-call during business hours on Saturday" isn't nearly as crazy as "you can't relax at any time during the weekend".
And having at least one daytime rotation agree that "well, you're expected to be on-call during business hours on Saturday" isn't nearly as crazy as "you can't relax at any time during the weekend".
Or just, like... allow people to have a Sun/Mon weekend (or Mon/Tue, or whatever) instead of Sat/Sun.
There are plenty of people who would be happy with that and plenty more who would accept that arrangement if it came with extra compensation.
Instead, particularly at big companies, people get hired with some generic comp. package and then it's luck of the draw whether you have an on-call rotation or not, and if you do whether it's onerous or not.
There are plenty of people who would be happy with that and plenty more who would accept that arrangement if it came with extra compensation.
Instead, particularly at big companies, people get hired with some generic comp. package and then it's luck of the draw whether you have an on-call rotation or not, and if you do whether it's onerous or not.
Employees and employers should be able to negotiate alternative weekends in that case. These alternate days are dealt with like real weekends with the real force of law and societal expectations.
> what if I'm out with friends, drunk, high, on the side of a mountain
You can plan around it, assuming a reasonable oncall schedule. If its every other week, yeah screw that. If its a week out of 4, you can plan it and still have it better than virtually every other highly paid profession there is, and many low pay ones.
> My hours are 9-5, after that is entirely me time
Then maybe an hourly job is better than a salaried one... Of course, if you're not paid the part, it's not worth it. But if you're paid 150, 200k or more? I hope you're not expecting 9-5.
You can plan around it, assuming a reasonable oncall schedule. If its every other week, yeah screw that. If its a week out of 4, you can plan it and still have it better than virtually every other highly paid profession there is, and many low pay ones.
> My hours are 9-5, after that is entirely me time
Then maybe an hourly job is better than a salaried one... Of course, if you're not paid the part, it's not worth it. But if you're paid 150, 200k or more? I hope you're not expecting 9-5.
> But if you're paid 150, 200k or more? I hope you're not expecting 9-5.
Speak for yourself, but I would never accept a $150k a year gig that expected me to be on-call, say, 1 week out of every 6. $150k is absolutely a "9-5 and stop thinking about work when you get home" kind of salary, if we're talking California standards
Speak for yourself, but I would never accept a $150k a year gig that expected me to be on-call, say, 1 week out of every 6. $150k is absolutely a "9-5 and stop thinking about work when you get home" kind of salary, if we're talking California standards
On call typically means you gotta be online and respond within 15 minutes if paged. That's abusive. If you want a person to do that, then pay them for that. As an example, on call in Amazon works the way I described, and yes it is abusive.
Unless you're working hourly or the on-call duties weren't disclosed during salary negotiations, you're already paid for it. I'll happily take a call after hours if it means I'm in a cushy job being paid 200k/year to solve puzzles. My partner manages a Safeway and makes well under half what I do for easily twice the work, and he's effectively on-call 24/7 for his store. Is either situation ideal? No, but I recognize that even with the injustice of having to ack an occasional alert at 2am, I'm in a crazy fucking comfy spot.
Yeah, "solving puzzles". If that's the case, surely it can wait - why does it need to be solved at 2 am? Anyway, if you're gonna be a slave for $200k please go ahead, its yo choice.
[deleted]
The question is whether it's abusive or not, not whether you personally are willing to accept being abused for X amount of money.
I would think the question of whether it's "abusive" should at some level be determined by observing labor practices elsewhere. Even if you don't, I doubt anyone is going to shed any tears for something "objectively" abusive if their own work situation is much worse.
> I would think the question of whether it's "abusive" should at some level be determined by observing labor practices elsewhere
Let's take this idea to it's logical extreme then. Surely you would agree that slavery is abusive. If we imagine a society where there are only slaves and slaveowners, your position would lead one to believe that slavery is not abusive, simply because it's the status quo.
This practice is abusive because it subverts the agreement that one will exchange their labor for pay within a defined set of hours (8 hours per day) and replaces it with the expectation (not agreement) that one will be "available" 24/7, but not actually "working" unless a pager goes off. This is plainly abusive, because it destroys your ability to use your free time to do things like drink alcohol, smoke marijuana, go hiking, go sailing, go for a run, etc. because you are required to be online with 15 minutes notice.
Let's take this idea to it's logical extreme then. Surely you would agree that slavery is abusive. If we imagine a society where there are only slaves and slaveowners, your position would lead one to believe that slavery is not abusive, simply because it's the status quo.
This practice is abusive because it subverts the agreement that one will exchange their labor for pay within a defined set of hours (8 hours per day) and replaces it with the expectation (not agreement) that one will be "available" 24/7, but not actually "working" unless a pager goes off. This is plainly abusive, because it destroys your ability to use your free time to do things like drink alcohol, smoke marijuana, go hiking, go sailing, go for a run, etc. because you are required to be online with 15 minutes notice.
> This practice is abusive because it subverts the agreement that one will exchange their labor for pay within a defined set of hours (8 hours per day)
But for salaried employees, this isn't the agreement. I sometimes work four hour days. Sometimes I work odd hours. The agreement, for salaried employees, isnt about hours, it's about work getting done.
If someone consents to that, swell! I think many companies are abusive in that they don't compensate oncall work enough, but salaried oncall positions aren't inherently more abusive then having a lawyer on retainer.
But for salaried employees, this isn't the agreement. I sometimes work four hour days. Sometimes I work odd hours. The agreement, for salaried employees, isnt about hours, it's about work getting done.
If someone consents to that, swell! I think many companies are abusive in that they don't compensate oncall work enough, but salaried oncall positions aren't inherently more abusive then having a lawyer on retainer.
Agreed. There are way too many people in our industry who accept being treated like utter garbage, and that's a problem, but we should also be clear that at the end of the day people do need to learn to stand up to themselves and not to consent to insanity.
My previous job I was on-call 1 week out of every 6. My first on-call week I got paged 100+ times (obv many of these alerts were firing within a minute of each other, but even accounting for that I was still being woken up multiple times per night). That week was particularly bad but I was still woken up an average of >= once per week on my on-call weeks. My team - the sole SRE team - was on-call for everything, including of course the 6 other dev teams' services.
Even worse, when I would fight to get bullshit alerts removed (the alerts that don't actually indicate a problem), I would get incredible pushback from, of all people, my own manager! (who was the manager of the SRE team since like I mentioned we were the only SRE team)
So not only were half the alerts bullshit, but I was counter incentivized to actually fix the problems causing the alerts. One great indicator of how bad things were was that everyone else on the SRE team (except the manager but he somehow always had unusually light on-call weeks) had a 5 minute delay set for pages, because the vast majority of pages would resolve within 1-3 minutes.
Oh, and by the way we weren't given any extra compensation whatsoever. Indeed I was told it was "part of the job description", which effectively meant that SRE skills were less valuable than the devs, because the devs were making identical salaries with no on-call requirement whatsoever! So a more specialized and, at least at this organization, difficult skillset, was worth less.
Anyway, I quit that job, and never looked back. I still have a friend on that exact team who is still putting up with being on-call, despite the incredible impact it has on his ability to go out and do stuff (mid 20's guy). I hear him complain about it all the time. But, like most people, he doesn't have the balls to either (a) push for internal change (which in fairness I tried and failed, although half the reason I failed was because nobody else on my team was willing to stick their neck out and push for the change with me), or (b) quit and find a new job.
So...yeah. Kind of rambling but we have a huge problem in our industry with people who either "don't have lives", or kind of do yet have so little self esteem or whatnot that they can't actually say no and protect their personal lives. In most cases on-call is simply a case of someone getting a raw deal and being too afraid to admit it to themselves.
My previous job I was on-call 1 week out of every 6. My first on-call week I got paged 100+ times (obv many of these alerts were firing within a minute of each other, but even accounting for that I was still being woken up multiple times per night). That week was particularly bad but I was still woken up an average of >= once per week on my on-call weeks. My team - the sole SRE team - was on-call for everything, including of course the 6 other dev teams' services.
Even worse, when I would fight to get bullshit alerts removed (the alerts that don't actually indicate a problem), I would get incredible pushback from, of all people, my own manager! (who was the manager of the SRE team since like I mentioned we were the only SRE team)
So not only were half the alerts bullshit, but I was counter incentivized to actually fix the problems causing the alerts. One great indicator of how bad things were was that everyone else on the SRE team (except the manager but he somehow always had unusually light on-call weeks) had a 5 minute delay set for pages, because the vast majority of pages would resolve within 1-3 minutes.
Oh, and by the way we weren't given any extra compensation whatsoever. Indeed I was told it was "part of the job description", which effectively meant that SRE skills were less valuable than the devs, because the devs were making identical salaries with no on-call requirement whatsoever! So a more specialized and, at least at this organization, difficult skillset, was worth less.
Anyway, I quit that job, and never looked back. I still have a friend on that exact team who is still putting up with being on-call, despite the incredible impact it has on his ability to go out and do stuff (mid 20's guy). I hear him complain about it all the time. But, like most people, he doesn't have the balls to either (a) push for internal change (which in fairness I tried and failed, although half the reason I failed was because nobody else on my team was willing to stick their neck out and push for the change with me), or (b) quit and find a new job.
So...yeah. Kind of rambling but we have a huge problem in our industry with people who either "don't have lives", or kind of do yet have so little self esteem or whatnot that they can't actually say no and protect their personal lives. In most cases on-call is simply a case of someone getting a raw deal and being too afraid to admit it to themselves.
OK, that's definitely a shitty situation. But if you imagine that instead you were empowered to shut off nonsense alerts and address causes of real ones, you wouldn't have actually eliminated on-call, so I'm a little confused about how you conclude from this that "all" on-call is abusive.
What's actually happening in this thread is that people like your friend are post-hoc justifying their own abuse so that they can remain internally consistent with their past decisions. Classic cognitive dissonance.
Is it really abusive if it's up to you to choose "Work at company X, make ~10% more but you have to be on-call for 1 week every 8 weeks" vs. "Work at company Y, make ~10% less, but never be on-call"?
Honestly, I'm not even sure how it's even remotely possible to draw a parallel between slavery and that. When negotiating a salary you'll have to take these things into account. Not worth it for the pay? Find another place to work. Jobs do grow on trees for people in our industry.
Honestly, I'm not even sure how it's even remotely possible to draw a parallel between slavery and that. When negotiating a salary you'll have to take these things into account. Not worth it for the pay? Find another place to work. Jobs do grow on trees for people in our industry.
OK, fine, maybe that doesn’t work as a universal principle. But I am compensated very well for a job where I rarely have to do anything that demanding and in exchange I can’t easily make plans for one week every two months. Hardly comparable to being a helot.
[deleted]
> The question is whether it's abusive or not
The question on whether it's abusive or not very obviously depends on the details. Yes, lol, I am actually willing to accept $250k per year, unlimited time off that I use upwards of 6 weeks per year of, full benefits, working from home, in the event my boss needs to call me once a year on a Saturday.
Sweeping generalizations are not helpful, and will likely hold you back in your career. You should evaluate things on their own merits.
The question on whether it's abusive or not very obviously depends on the details. Yes, lol, I am actually willing to accept $250k per year, unlimited time off that I use upwards of 6 weeks per year of, full benefits, working from home, in the event my boss needs to call me once a year on a Saturday.
Sweeping generalizations are not helpful, and will likely hold you back in your career. You should evaluate things on their own merits.
One argument is that you are being paid for it; it’s just not broken out into two lines, but the dollars are there.
If you’re making $200K/yr, maybe $150K of that is for your job and $50K is for being on-call. That seems like a more than fair price (or at least one that’s in the ballpark) for periodic on-call service.
If you’re making $200K/yr, maybe $150K of that is for your job and $50K is for being on-call. That seems like a more than fair price (or at least one that’s in the ballpark) for periodic on-call service.
In my experience, often people paid more end up not having on-call duties. I know several people who have negotiated high salaries from the start of their employment along with a stipulation that they will never be on the on-call rotation.
In my own situation, I removed myself from on-call duties during a leave of absence, and never went back on after I returned to work. Since then, I've still gotten the same raises I got before.
In my own situation, I removed myself from on-call duties during a leave of absence, and never went back on after I returned to work. Since then, I've still gotten the same raises I got before.
An unfortunate reality of on call is that companies care about projects delivered, not disasters averted.
Unless your stepping in to right the ship when a company is on fire operationally, and can point to specific action/results you delivered to right the ship - getting pages 8x per week does nothing for your career.
Unless your stepping in to right the ship when a company is on fire operationally, and can point to specific action/results you delivered to right the ship - getting pages 8x per week does nothing for your career.
If that was the case we could easily see it in the salaries of on-call versus not on-call positions. This is clearly not the case.
That's mostly because our industry is a bit of a mess. We clearly have different people doing completely different roles that are in the same career ladder with the same title. Forget on call, some folks with the same titles and same salaries often do easier or harder jobs. That's a separate problem.
Salaries in the industry have a lot of expectations baked in, like the expectations of being able to learn and train yourself without the company having to pay for it (aside the occasional paid book). Being on call is another. That some folks can get the same pay regardless of their responsibilities or how hard their job actually is is, indeed, a problem.
Salaries in the industry have a lot of expectations baked in, like the expectations of being able to learn and train yourself without the company having to pay for it (aside the occasional paid book). Being on call is another. That some folks can get the same pay regardless of their responsibilities or how hard their job actually is is, indeed, a problem.
I interviewed at Brex, and they required getting online within 5 minutes of being paged. (I did not get the offer, but I would not have accepted the offer after learning that.) Where I currently work the response time is "ideally within 15 minutes", which is a lot more manageable, since plenty of places I like to go are within 15 minutes of my house. My job pays more than average because of on-call, it's 1 week out of 6, and there's probably an average of 1 call out of business hours per week (mode is 0).
5 minutes is absurd. You'd basically have to keep your laptop with you at all times
>>On call typically means you gotta be online and respond within 15 minutes if paged.
No place I have ever worked has this kind of oncall policy.
Most I have seen have a response time of 1hr, note resolution, but response.
Amazon is large enough they should have people staffed 24/7 just by staggering the timezone codes at the different offices. This is how Cisco TAC works, you call them you get what ever timezone code is "day time" at that time.
So I would agree that 15min on call is abusive, luckily this is not my experience as "typical" in the industry, hell even fully staffed 24/7 call centers typically do not have a 15min response time
No place I have ever worked has this kind of oncall policy.
Most I have seen have a response time of 1hr, note resolution, but response.
Amazon is large enough they should have people staffed 24/7 just by staggering the timezone codes at the different offices. This is how Cisco TAC works, you call them you get what ever timezone code is "day time" at that time.
So I would agree that 15min on call is abusive, luckily this is not my experience as "typical" in the industry, hell even fully staffed 24/7 call centers typically do not have a 15min response time
Call centers and oncall are different. Oncall is "when Facebook suddenly stops working, who is responsible for figuring out why and fixing it". These systems often have SLAs (either internal or public) in minutes. Response (ack and log on) times for these can be 30 minutes for less relevant services, but some have 15, or even 1-5 minute response times for highly critical stuff.
1-5 minutes response time is not "Oncall". It's being at work.
In France you're necessarily compensated for being on call and then paid when you are paged.
Furthermore, if you do get called, you shouldn't go back to work until 11 hours after you finished handling that call.
French salaries for senior developers are less than entry level developers in the US.
I have bootcamp students with 6 months of software development experience making more than top developers in France, lol. The few extra Euros you get for being on call hardly seems worth it.
There's a reason French workers are almost always rioting. Low pay and very few opportunities.
I have bootcamp students with 6 months of software development experience making more than top developers in France, lol. The few extra Euros you get for being on call hardly seems worth it.
There's a reason French workers are almost always rioting. Low pay and very few opportunities.
How do salaries in France compare to the US?
I don't know about France but in my country that's also factored in the law and hiring offers except for jobs that are on managerial level (them the law says you don't have clock in but you also pretty much don't get to clock out and you mostly get paid well for that), which doesn't apply for developers and that means you get paid less if there is no "on call". Basically your base pay is the same on company A and company B (assuming they are truly equal), but since company B requires "on call" that is extra work that gets compensated based on your hourly rate or double that if it is during night time, in the same way for the same job in the same company who ever works night shifts get paid twice as much as the daily shift (before taxes).
3'000 to 4500 USD monthly for a SRE
They do pay people for that - the salaries at MAANG are inflated as is.
This sounds a bit hard to believe. Mind sharing some of the $ figures of this business?
Nevertheless, I'm with the parent. This practice is abusive. When you're not working, you're not working. Being on call is working. If a company wants you to work longer hours they should make it explicit, pay for it, and comply with the local employment laws. This sort of argument that the business has no merit if they were to hire more people or pay can be applied to many exploitative situations. Most of the companies doing this are drowning in profits.
Nevertheless, I'm with the parent. This practice is abusive. When you're not working, you're not working. Being on call is working. If a company wants you to work longer hours they should make it explicit, pay for it, and comply with the local employment laws. This sort of argument that the business has no merit if they were to hire more people or pay can be applied to many exploitative situations. Most of the companies doing this are drowning in profits.
My company pays for on-call. There is a rotation, you go for a week and they pay you more for that week by several hundred AUDs and it is basically a job requirement.
Does the fact that it is discussed during hiring process and that it is paid change you opinion about being oncall?
I think that sounds fair. The concept of being on call is not unique to software engineers. You might be an emergency doctor. You might be a fighter pilot. It's possible that some software engineers need to be on call, because they support some critical infrastructure, and that can be part of their contract.
I would say the default assumption for most software engineers should be they are not on call and the company should try to structure itself around that assumption. The problem starts it's just considered the norm that software engineers are expected to work 24/7.
I would say the default assumption for most software engineers should be they are not on call and the company should try to structure itself around that assumption. The problem starts it's just considered the norm that software engineers are expected to work 24/7.
I am of the mindset that a healthy team doesn't need an on call. Group chats, tight relationships, etc. usually mean someone is free just based on the odds to handle a true emergency. If you have so few people that the ship can't run, but you're not in a startup environment that justifies having so few people, that is your real problem.
An average company with engineers who don't hate their job, staffed adequately, it should be a stars align kind of scenario where you can't get someone on a weekend. It's a simple matter of probability, and if that probability looks grim, either A. you're understaffed, or B, your staff is underengaged.
I have fixed customer outages on my phone out having a good time more than once. Usually with another engineer happy to help. Not just at one company. If you need an on call rotation, your culture sucks. I'm not advocating that people live to work, I'm advocating that people work at places where people enjoy working with their peers and their product enough that an occasional blip isn't a big deal. With the right infrastructure and the right people you should have this.
An average company with engineers who don't hate their job, staffed adequately, it should be a stars align kind of scenario where you can't get someone on a weekend. It's a simple matter of probability, and if that probability looks grim, either A. you're understaffed, or B, your staff is underengaged.
I have fixed customer outages on my phone out having a good time more than once. Usually with another engineer happy to help. Not just at one company. If you need an on call rotation, your culture sucks. I'm not advocating that people live to work, I'm advocating that people work at places where people enjoy working with their peers and their product enough that an occasional blip isn't a big deal. With the right infrastructure and the right people you should have this.
> Group chats, tight relationships, etc. usually mean someone is free just based on the odds to handle a true emergency.
Congratulations, you've made you entire team on call year round.
I don't read work chats outside of core working hours if I'm not currently on-call. In general, most people shouldn't. Doing so is awful for your stress levels and work-life balance. I did it for more than a decade, burning out twice in the meantime, and things have been much better since I stopped.
Congratulations, you've made you entire team on call year round.
I don't read work chats outside of core working hours if I'm not currently on-call. In general, most people shouldn't. Doing so is awful for your stress levels and work-life balance. I did it for more than a decade, burning out twice in the meantime, and things have been much better since I stopped.
I don't read group chats _inside_ working hours. I'm working at work, not group chatting.
If that means you're reading your chats later it sounds as if you're also working outside working hours.
You missed the point entirely.
The goal is not to need on call, because you BOTH have a stable system that requires rare off hours interventions, AND you have a team that owns that and is bothered rarely enough to not need to have a formal rotation.
On call systems with stable systems in my experience tend to be a real problem because people lose the on call urgency anyway.
TL;DR - the problem is your system, not being on call or not.
The goal is not to need on call, because you BOTH have a stable system that requires rare off hours interventions, AND you have a team that owns that and is bothered rarely enough to not need to have a formal rotation.
On call systems with stable systems in my experience tend to be a real problem because people lose the on call urgency anyway.
TL;DR - the problem is your system, not being on call or not.
I was on a team that handled a lot of things that way, and we made a deliberate, conscious shift to pull them into oncall rotations. We found that it was severely compromising work-life balance; engineers outside of the old guard thought they were expected to work most nights and most weekends, because they'd constantly see people get bonuses and accolades for some stuff they did at 6PM Saturday. I personally enjoy the "work whenever you want, play whenever you want" model, but I really don't think it works as an oncall policy.
I’m on a team of six in the same time zone. None of us are going to conveniently be watching the alerts at 4 AM Sunday. One of us has to get woken up who wasn’t planning to be well rested for anything that day.
Ah yes, the "always be working" method. That way you never need overtime!
no buddy your idea sounds like unlimited vacation days, it is worse than the original method
If they can’t afford to staff for 24x7 operation and maintain profitability that’s their problem. They should either reduce their service hours or charge more.
By working for free you’re subsidizing your organization’s unsustainable business model.
By working for free you’re subsidizing your organization’s unsustainable business model.
Getting paid 200k+ for a job that had on-call as part of stated duties (and thus baked into the comp) prior to accepting the offer is now working for free? There is hyperbole and then there is whatever this is.
Don’t like it, quit and find a company that is moving slowly and doesn’t value ownership from engineers, someone else will fill in the vacant post. Sounds like a reasonably sustainable business model to me, especially since it has worked like this for over decades at some companies.
Don’t like it, quit and find a company that is moving slowly and doesn’t value ownership from engineers, someone else will fill in the vacant post. Sounds like a reasonably sustainable business model to me, especially since it has worked like this for over decades at some companies.
There are plenty of jobs in good companies that pay well and don't require you to be a slave to your job. Companies that can move fast and treat their employees well vs. companies that exploit their employees. I am a manager in such a company and my team can move fast and own stuff during the work day and then relax on their weekend so they can get back to moving fast and owning stuff when they're back at work.
This has totally not been this way for decades.
Now if you're a founder, or you're being paid for this extra work or own a significant part of the business and benefit from it's success, that's a little bit different.
This has totally not been this way for decades.
Now if you're a founder, or you're being paid for this extra work or own a significant part of the business and benefit from it's success, that's a little bit different.
"Getting paid 200k+ for a job that had on-call as part of stated duties (and thus baked into the comp) prior to accepting the offer is now working for free? There is hyperbole and then there is whatever this is."
The lawyers at my company go home at 5 and make way more. I know a doctor who gets paid somewhere around $1500 for every day he is on call. And when they actually call him he makes even more.
The lawyers at my company go home at 5 and make way more. I know a doctor who gets paid somewhere around $1500 for every day he is on call. And when they actually call him he makes even more.
You're likely very wrong about the lawyers. They leave at 5 when nothing is urgent. But in certain crisis situations -- company being bought out, a severe injury occurred on premises, a client is suing you, etc -- those lawyers will be expected to respond 24/7 all through a full weekend or beyond.
Of course, you can still go to law school if you're really that jealous.
Of course, you can still go to law school if you're really that jealous.
How many IT people on call are actually making 200+k? The general rule I have seen is people on call get paid less as people eventually negotiate very slightly lower salaries for not being on call.
Amazing to see all these awful takes. This _is_ America I suppose.
Below a certain level I mostly agree with you. One company in my past didn't work this way, and one too many pages in the middle of the night for a business I wasn't personally invested in made it an easy job to walk away from.
That said, there is a level of seniority at which "their problem" is "your problem." If you'd chuck the business out the door rather than getting paged a few times a year, you're not ready for that level yet. This varies by company and organization size, but fundamentally you can't (and shouldn't try to) anticipate everything up front. Sometimes you need the knowledge, judgement, or simply signing authority (literal or metaphorical) of someone specific.
That said, there is a level of seniority at which "their problem" is "your problem." If you'd chuck the business out the door rather than getting paged a few times a year, you're not ready for that level yet. This varies by company and organization size, but fundamentally you can't (and shouldn't try to) anticipate everything up front. Sometimes you need the knowledge, judgement, or simply signing authority (literal or metaphorical) of someone specific.
> That said, there is a level of seniority at which "their problem" is "your problem.”
I’ve long decided that unless I have the significant stock options to show for it it’s delusional. It’s what the company wants you to think: to have all the emotional investment and none of the actual ownership benefits.
I’ve long decided that unless I have the significant stock options to show for it it’s delusional. It’s what the company wants you to think: to have all the emotional investment and none of the actual ownership benefits.
One can debate the title/comp level where this change happens but, certainly, at some executive level it's reasonable to expect to be contacted off-hours if there's some customer/PR/facility/etc. crisis. That's not to say an exec can never be out of touch; a friend told me that their former CEO had a new communications head quit because they couldn't deal with said CEO telling them that they should just deal with PR issues if he was on vacation.But, above a certain level, traveling on the weekend, being away nights, late night calls, and yes the occasional "Houston, we have a problem" moment happen.
I am quit amazed at the general reaction of this forum to this standard industry practice.
If you want a cushy 9-5 job, go work for a code factory like HCL, TCS and whatnot. That way you can build crap and wash your hands after handling it to the customer.
In a startup environment, you gotta build your shit and maintain it to. It's an environment that's not for everyone.
I've been in oncall roosters several times in the past. And it made freaking sure we wrote the best software we could.
I also have been on the management side of it, and the way I set it up is that whoever was the oncall engineer, if he had to work for more than 2 hours after normal working hours, he would get an additional PTO day. It actually worked quite well.
If you want a cushy 9-5 job, go work for a code factory like HCL, TCS and whatnot. That way you can build crap and wash your hands after handling it to the customer.
In a startup environment, you gotta build your shit and maintain it to. It's an environment that's not for everyone.
I've been in oncall roosters several times in the past. And it made freaking sure we wrote the best software we could.
I also have been on the management side of it, and the way I set it up is that whoever was the oncall engineer, if he had to work for more than 2 hours after normal working hours, he would get an additional PTO day. It actually worked quite well.
>If you want a cushy 9-5 job, go work for a code factory like HCL, TCS and whatnot. That way you can build crap and wash your hands after handling it to the customer.
You got it backwards: if you want a crappy quality of life (or have no life and you only identify with the company that sees you as a replaceable cog), and crappy products and customer service from a company that doesn't care (to plan right, to hire accordingly, to treat its staff right) go on call and overwork, producing sleep-derived crap.
Craftsmen and artisans take their time and have boundaries.
"Feature factories", startups, and mass market crap companies forego 9-to-5 and indoctrinate naive employees that they do something important by doing so.
>In a startup environment, you gotta build your shit and maintain it to. It's an environment that's not for everyone.
Yes. It's for starry-eyed naive youngsters right off the bus. The kind of people to believe they're "changing the world" by building a Facebook or Groupon.
You got it backwards: if you want a crappy quality of life (or have no life and you only identify with the company that sees you as a replaceable cog), and crappy products and customer service from a company that doesn't care (to plan right, to hire accordingly, to treat its staff right) go on call and overwork, producing sleep-derived crap.
Craftsmen and artisans take their time and have boundaries.
"Feature factories", startups, and mass market crap companies forego 9-to-5 and indoctrinate naive employees that they do something important by doing so.
>In a startup environment, you gotta build your shit and maintain it to. It's an environment that's not for everyone.
Yes. It's for starry-eyed naive youngsters right off the bus. The kind of people to believe they're "changing the world" by building a Facebook or Groupon.
> Craftsmen and artisans take their time and have boundaries.
I've been an engineer on-call for 7 years since graduating. In the meantime I handled a handful of business critical situations, developed my ability to keep cool during crisis and put my skills to the test. When I started doing it, I wasn't even paid for on-call. Now I am. The money has no bearing on me having been exploited or not. I've done it because it's cool, and when I think it's too tiring I won't do it anymore.
Am I less of an engineer, not a true craftsman (wtv that means), because of this? There's other opinions in the world, no need to be so close minded.
I've been an engineer on-call for 7 years since graduating. In the meantime I handled a handful of business critical situations, developed my ability to keep cool during crisis and put my skills to the test. When I started doing it, I wasn't even paid for on-call. Now I am. The money has no bearing on me having been exploited or not. I've done it because it's cool, and when I think it's too tiring I won't do it anymore.
Am I less of an engineer, not a true craftsman (wtv that means), because of this? There's other opinions in the world, no need to be so close minded.
When you were doing on-call without being paid, that's called wage theft. It's the single biggest kind of theft in North America: https://www.epi.org/publication/wage-theft-bigger-problem-fo...
It's also the kind of theft that insidiously convinces you that it's really cool and that you're actually OK with it because you're learning how to 'keep cool during crisis'. You know how else your employers could have taught you that? By providing actual training during working hours, while you were being paid.
It's also the kind of theft that insidiously convinces you that it's really cool and that you're actually OK with it because you're learning how to 'keep cool during crisis'. You know how else your employers could have taught you that? By providing actual training during working hours, while you were being paid.
I mean this with all the respect in the world, but I think you've gotten yourself caught in an ideological bubble here. To someone who doesn't already agree with you, this comment sounds like an argument against the concept of wage theft more than an argument against oncall. After all, you can't prove something's bad just by classifying it under a nasty-sounding label!
I purposefully did not make an argument against on-call, although I might have easily done that and others have in this thread. All I said is that doing on-call without pay is wage theft. And this is not an ideological bubble thing, as anyone will quickly understand if they ask a plumber to do on-call for their house plumbing system without pay.
A better analogy would be a plumber you pay a handsome annual salary to maintain and improve your pipes--to what degree should they owe some emergency time in return is the question in this thread.
Plumbers (or companies employing plumbers) warranty their work. Do employees warranty their work for employers? I have never heard of that arrangement in the US.
I haven’t heard of that here either. The closest I can think of are sanctions that can be levied on individual employees by professional standards organizations. I do know trades warranties aren’t provided by employees working in that trade to their employer, just to customer individuals or organizations.
I think a another analogy is that you had a plumber do some work for you a day before a leak, and in the middle of the night the pipes start leaking and you need to call him out to prevent property damage. In that case I would absolutely not expect to have to pay extra because he warranties his own work.
To be fair, if they are so far off the end that they think "man, getting called up at 2am without being paid was great because I learned how to keep my cool during a crisis!" then they are probably a lost cause.
It's fine to drink of the chalice of the corporate kool aid with moderation. Still, one should never get black out drunk out of it.
It's fine to drink of the chalice of the corporate kool aid with moderation. Still, one should never get black out drunk out of it.
This "us vs them" concept is where I think we've lost the plot. Remember that we're the experts in complex systems that we've been selling to the business to add value and we've been happily taking their money for decades. When they ask "can the technology support this?" we, like an eager child showing off our new toy, assure them that it can and beg them to let us prove it.
We can't just chunk these complex systems over the wall and say "have fun with that" and demand more money when the solution breaks. This means we failed to properly estimate total cost of ownership when we sold them the system and now they're held hostage. This is borderline legal breach of contract.
If we told them the true sustainable cost of ownership for a follow the sun support model they likely would have never purchased the solution in the first place and we'd be out of a job.
We could have lots of annoying jobs or no jobs at all. I'll happily accept the former.
We can't just chunk these complex systems over the wall and say "have fun with that" and demand more money when the solution breaks. This means we failed to properly estimate total cost of ownership when we sold them the system and now they're held hostage. This is borderline legal breach of contract.
If we told them the true sustainable cost of ownership for a follow the sun support model they likely would have never purchased the solution in the first place and we'd be out of a job.
We could have lots of annoying jobs or no jobs at all. I'll happily accept the former.
I don't know about you but I'm not 'selling' my work to the business like a black box, I am working under the direct supervision of the business and any and all work that I do is the result of a team effort and direction from management, non-engineering, and engineering staff. Let me emphasize: all the work that gets put into production is a team effort and a team responsibility.
I'm not begging anyone to let me prove my technology, I'm working with management to come up with a solution that will work for the customer. Oftentimes management will forbid or otherwise prevent me from using a better, more robust technology that would have prevented more production issues because of short-sightedness. If I pointed this out and was still told to go ahead with the short-sighted solution, and it breaks in production, whose fault is that?
I'm not suggesting the 'it's your problem now, have fun' approach, I'm suggesting the 'pay overtime for overtime work' approach. This is nowhere even near to 'breach of contract', please don't throw around lawyer-like terms when you clearly don't understand what's being discussed.
I'm not begging anyone to let me prove my technology, I'm working with management to come up with a solution that will work for the customer. Oftentimes management will forbid or otherwise prevent me from using a better, more robust technology that would have prevented more production issues because of short-sightedness. If I pointed this out and was still told to go ahead with the short-sighted solution, and it breaks in production, whose fault is that?
I'm not suggesting the 'it's your problem now, have fun' approach, I'm suggesting the 'pay overtime for overtime work' approach. This is nowhere even near to 'breach of contract', please don't throw around lawyer-like terms when you clearly don't understand what's being discussed.
>In the meantime I handled a handful of business critical situations, developed my ability to keep cool during crisis and put my skills to the test.
That's great, but you could still have done all those things if you were being compensated appropriately for the extra time you were putting in.
It's like unpaid internships. I'm sure unpaid interns often do learn useful skills and make useful connections. But the practice is still exploitative.
That's great, but you could still have done all those things if you were being compensated appropriately for the extra time you were putting in.
It's like unpaid internships. I'm sure unpaid interns often do learn useful skills and make useful connections. But the practice is still exploitative.
>The money has no bearing on me having been exploited or not. I've done it because it's cool
Convincing them of the "coolness" of it is the most common way to exploit the naive/fresh.
The same people who do so to others, wouldn't even piss if they weren't compensated for it...
(And being exploited or not is not a personal decision. If you aren't compensated for overtime, then you are exploited. It just means you're ok with it.).
Convincing them of the "coolness" of it is the most common way to exploit the naive/fresh.
The same people who do so to others, wouldn't even piss if they weren't compensated for it...
(And being exploited or not is not a personal decision. If you aren't compensated for overtime, then you are exploited. It just means you're ok with it.).
There's nothing in the world you'd do for free for a while for the chance of doing it? And I'm the naive one? Is money all there is in life in this comment section?
Also: For some people these nights might have been the only times they connected to their team mates on a different level.
I had that too, but it was for art projects, films, music, my own programming projects or some other thing that I certainly did not do for money.
The one thing that differs here is that the people I did this for/with would cross the country if I told them I am in need.
I had that too, but it was for art projects, films, music, my own programming projects or some other thing that I certainly did not do for money.
The one thing that differs here is that the people I did this for/with would cross the country if I told them I am in need.
i'd never heard of a "feature factory" before, but after reading about them, holy hell, that explains a ton of friction i had in a recent engagement.
they had a laser focus on just shipping and success theater, but little attention on actual utility of the end product. i was berated constantly for slowing things down or wanting to be more careful and do things that weren't necessarily easy but actually had potential to be useful. it's relieving to learn this is a common management deficiency.
they had a laser focus on just shipping and success theater, but little attention on actual utility of the end product. i was berated constantly for slowing things down or wanting to be more careful and do things that weren't necessarily easy but actually had potential to be useful. it's relieving to learn this is a common management deficiency.
> Craftsmen and artisans take their time and have boundaries.
Yeah but most of the people complaining only think they are Craftsmen but their day job is to write bad software to push ads
Yeah but most of the people complaining only think they are Craftsmen but their day job is to write bad software to push ads
You do understand that most startups fail, right? Clearly, on aggregate, their practices are not really working or worthy of imitation. For the ones that succeed, there's a huge element of luck. Otherwise people would make guarantees about success.
I think most experienced engineers realize on-call is a complete fool's game -- you will never get accolades for fixing shit at 2am. Literally never. Your only "reward" will be to become the escalation point to continue getting called at 2am. It's not only mentally demanding work, but also bottom-feeding work that gets no recognition. No one is doing their best work after being woken up in the middle of the night, it's all band-aid hacky fix shit that is being generated. Also in general the type of personality who would tolerate working 2+ hours beyond their working time is probably the type of person that isn't even going to use up all of their PTO to begin with, so I always find those kinds of rewards laughable.
I don't find your experience to be the general case, though that doesn't invalidate your point. Accolades are an easy intangible reward to hand out so you will often see it given for heroics. Extra cash would be better. PTO is somewhere in the middle. On teams that can rotate through on-call, bonus PTO is more likely to be used than on ones where one or two engineers shoulder the responsibility all year.
Well, putting the pipe dreams people have about a global team so nobody is ever working outside regular working hours aside (hope you've considered holidays, guys), what's the alternative? One I've seen is that there's an ops, or "devops," team, that's responsible for picking up the slack instead. But if it's unfair for engineers to have to do it, it seems even less fair for someone who doesn't even have control over what code ships to do it, and to do it without a rotation.
> what's the alternative?
Let it crash. If you can't afford to pay people enough that someone /wants/ to fix it at 2am, and you can't arrange a retainer for a contractor halfway around the world to be available if needed during their worktime, and you (owner/manager) don't want to do it yourself, then it can be off until someone starts at 9am next business day. (Most products and services aren't life critical).
Let it crash. If you can't afford to pay people enough that someone /wants/ to fix it at 2am, and you can't arrange a retainer for a contractor halfway around the world to be available if needed during their worktime, and you (owner/manager) don't want to do it yourself, then it can be off until someone starts at 9am next business day. (Most products and services aren't life critical).
I think the whole point of the article and the discussion is that many people don’t want to do it even when paid enough.
Nobody on any of the threads is arguing that you should not be paid more for being on call.
Nobody on any of the threads is arguing that you should not be paid more for being on call.
The issue with being on call constantly is, that you have to basically give up any private life you should have. Your daughters theatre play? Sad to interrupt! Your long awaited weekend trip with your girlfriend? Would be a shame if you had to cancel!
The quality of life lost by being on call is not insignificant. This is why being on call should always be the absolute exception and not a reliable, planned in, tool at your companies disposal.
If you regularily fail to organize your core business around business hours, maybe you should get someone who is good at planing projects and communicating these plans and put them in charge? Or maybe just close your company and search for a different career path if you are unable to plan your projects?
The quality of life lost by being on call is not insignificant. This is why being on call should always be the absolute exception and not a reliable, planned in, tool at your companies disposal.
If you regularily fail to organize your core business around business hours, maybe you should get someone who is good at planing projects and communicating these plans and put them in charge? Or maybe just close your company and search for a different career path if you are unable to plan your projects?
This is kind of absurd. The point of on-call is not to have regular off-hours work, but to have someone available for emergent situations. Since anybody who intends to operate a service with constant availability must always have someone who can respond to emergencies, on-call is the system used by the world's most successful tech companies, not just two-bit upstarts. It's hard to imagine anybody serious about running any kind of online business who is going to respond to an outage on Friday night by saying "we'll look at on Monday at 9:00 PT." It's unfortunate, but it's really not that different from ER doctors, firefighters, utility workers, and numerous other workers working with similar constraints.
Besides that, the rotation is known, and planned for, long in advance. So your "long-awaited weekend trip" shouldn't have been scheduled during your on-call shift in the first place.
Besides that, the rotation is known, and planned for, long in advance. So your "long-awaited weekend trip" shouldn't have been scheduled during your on-call shift in the first place.
They can pay me enough that I’ll do it but not enough that I’ll want to do it.
You've probably just described most jobs.
Yeah, and this is a relatively good one, so let's have some perspective.
I'm replying to the post saying this is an "industry standard" thing--I'm not saying that there is no disaster scenario ever. Obviously not every contingency can be planned.
However, there are certainly signals that a company respects a developers' time. Things like...was there effort put in the product around things that might minimize developer on-call time, like having more user self-serve options, or spending more time on testing, or having a proper feedback loop for recurring issues to be permanently squashes in a timely fashion, etc etc. Are managers regularly a part of these off-hour on-call issues so they're affected too, thus motivated to resolve it? Or does the company just abusively unload all of the burden on the developer and force them to use up their own personal time to maintain a product? Far too often it's the latter, and I think it should be clear to all that this is abusive practice and not "normal" or "standard" nor should any of us consider it that way.
However, there are certainly signals that a company respects a developers' time. Things like...was there effort put in the product around things that might minimize developer on-call time, like having more user self-serve options, or spending more time on testing, or having a proper feedback loop for recurring issues to be permanently squashes in a timely fashion, etc etc. Are managers regularly a part of these off-hour on-call issues so they're affected too, thus motivated to resolve it? Or does the company just abusively unload all of the burden on the developer and force them to use up their own personal time to maintain a product? Far too often it's the latter, and I think it should be clear to all that this is abusive practice and not "normal" or "standard" nor should any of us consider it that way.
Sure, but no matter how many measures you take, on-call still means you have to be available, so it’s a drag even if it’s quiet.
The number of pages is only part of the story. Even if you never get paged a single time, oncall still sucks because you have some response requirement. If I could get paged and I need to respond within 30m then it is no hikes for me this weekend.
[deleted]
FWIW, we have on-call at my work (all the developers are on PagerDuty schedule except the newest) and don’t have a written or agreed response requirement. It’s essentially best-effort, we had one instance where a page got ignored for hours by both primary and secondary responders and no one got in trouble. YMMV of course and I suspect we are laxer than most for various corporate and leadership reasons.
I do bring my computer with me when traveling out of state when on call but generally don’t worry about e.g. going for a few hours hike or whatever even if I won’t have the laptop with me. If it were stricter it would definitely be more annoying…
I do bring my computer with me when traveling out of state when on call but generally don’t worry about e.g. going for a few hours hike or whatever even if I won’t have the laptop with me. If it were stricter it would definitely be more annoying…
This one is even weirder to me. If it is best effort then this means that it is okay if the work doesn't get done. If that is the case, then why can't it be done the next morning?
My team has an "oncall" shift like this where we assign a person to be responsible for triaging bugs and other non-urgent alerts just so we don't all stare at it and wait for somebody else (or inevitably me, the lead) to handle it but they are explicitly told that this responsibility does not extend beyond working hours.
My team has an "oncall" shift like this where we assign a person to be responsible for triaging bugs and other non-urgent alerts just so we don't all stare at it and wait for somebody else (or inevitably me, the lead) to handle it but they are explicitly told that this responsibility does not extend beyond working hours.
"Here's one. I get paged for urgent fixes maybe three times per year. To staff all three shifts with senior engineers to cover for that would be enough to make the company unprofitable and lose many jobs. I don't much like being woken in the middle of the night for such things, but it's clearly necessary and not abusive."
Do you have to be always available or can you go somewhere without connection?
Do you have to be always available or can you go somewhere without connection?
That’s your thought and unfortunately it lines with the parroting by exploitative managers. But if I’m on call and expected to reply to that call whenever the bell rings I’m working all these hours and days no matter how rarely or frequently that bell rings.
On the other hand if it’s that less crucial for your company, 2-3 times a year without much monetary impact to warrant dedicated staff, I’m sure it can wait until necessary working morning just fine.
It’s not whether I’m being utilised or not, it’s that I’m being required to project my availability!
On the other hand if it’s that less crucial for your company, 2-3 times a year without much monetary impact to warrant dedicated staff, I’m sure it can wait until necessary working morning just fine.
It’s not whether I’m being utilised or not, it’s that I’m being required to project my availability!
[deleted]
The article explicitly states that’s reasonable. It’s the rotating pager duty that’s unreasonable.
This suggests a pricing issue for your uptime guarantees.
[deleted]
> If an organization thinks their systems should be available 24x7 they should staff people 24x7.
I’m a software engineer. I’m paid well partly because things like oncall are necessary. It’s priced into the compensation.
> There is no situation where it’s ok to contact someone to do work outside of their work hours.
It sounds like you should find a company that agrees with this opinion. I don’t agree.
I’m a software engineer. I’m paid well partly because things like oncall are necessary. It’s priced into the compensation.
> There is no situation where it’s ok to contact someone to do work outside of their work hours.
It sounds like you should find a company that agrees with this opinion. I don’t agree.
This attitude is very common with software engineers and I find it baffling. Is it some kind of inferiority complex? I doubt Warren Buffett worries that he is overpaid and demands to be woken up in the middle of the night for some self-flagellation. So why do so many software engineers think "I am paid well so I deserve whatever the company throws at me"? You are selling something: your skills, expertise, and time on this planet. These things are limited and valuable. Negotiate the price and terms of the sale!
I would say its a superiority complex : thinking software engineers are paid well and hence need to be ready to sacrifice more to maintain this superior position.
And of course also because the risks of bad or interrupted sleep is not recognised widely and mostly ignored.
And of course also because the risks of bad or interrupted sleep is not recognised widely and mostly ignored.
Which is why you can also see the exec team paged in the middle of the night, right? Because they're paid well and need to sacrifice?
Yes, and had their vacations cut short and other such things than rank and file employees would tolerate far less.
[deleted]
I’ve definitely seen my exec team get paged in the middle of the night, and they didn’t give any indication that it was unreasonable or uncommon. If anything they seemed to enjoy it, which to be honest I do as well. Being woken up to solve a problem (on rare occasion) can be validation that you’re an important person working on important things.
Lots of other fields have on calls or out of work expectations. That's part of the job, and you're compensated highly.
It's not like on call is a shock or something, it's not "whatever they throw at me", it's just... part of the job. My sister is a dentist, she has to be on call sometimes, it's not a shocker or something you don't know when you're signing up for the job.
You're implying that people aren't negotiating, but that's baseless. Software engineers are highly compensated because of these expectations, and we all negotiate accordingly.
It's not like on call is a shock or something, it's not "whatever they throw at me", it's just... part of the job. My sister is a dentist, she has to be on call sometimes, it's not a shocker or something you don't know when you're signing up for the job.
You're implying that people aren't negotiating, but that's baseless. Software engineers are highly compensated because of these expectations, and we all negotiate accordingly.
> Lots of other fields have on calls or out of work expectations.
Yes, and in many of them there are call-out fees and overtime. Programmers and sysadmins have convinced themselves that, as "professionals" they are not aligned with traditional working-class constructs like this.
Yes, and in many of them there are call-out fees and overtime. Programmers and sysadmins have convinced themselves that, as "professionals" they are not aligned with traditional working-class constructs like this.
> Yes, and in many of them there are call-out fees and overtime.
Why is that any better than just getting paid overall more? Lots of companies also have internal policies like "if you get called in on a weekend take a long weekend next week" etc in my experience.
Why is that any better than just getting paid overall more? Lots of companies also have internal policies like "if you get called in on a weekend take a long weekend next week" etc in my experience.
Because then the company has a financial incentive to reduce call-outs and overtime.
>we all negotiate accordingly
This is quite a sweeping statement to make. My first engineering job salary was non-negotiable. I was told to either accept it, or they 'rapidly' move to another candidate.
This is quite a sweeping statement to make. My first engineering job salary was non-negotiable. I was told to either accept it, or they 'rapidly' move to another candidate.
I meant more collectively. But of course, yes, it's sweeping. It'll change regionally, or based on experience, or company culture, etc.
But I don't think it's unfair to say, especially in the US, that software engineers are highly compensated.
But I don't think it's unfair to say, especially in the US, that software engineers are highly compensated.
While I agree that in general they shouldn’t feel bad about asking to be being Paid More (tm) you are making unwarranted assumptions about how happy they are with their current employment situation. You don’t know them, don’t know how much they’re paid, and don’t know how well or badly they deal with being on call.
There is nothing immoral about deciding you’re happy.
There is nothing immoral about deciding you’re happy.
I never said it's wrong, I just said I don't understand it. I think my post sounds a lot less moralizing than the one I am replying to. On-call is not some kind of basic virtue. It is simply part of a job description, and that's a business contract.
I personally think that a much healthier alternative would be to look for a position where one can maintain their physical and mental health, and give back extra income to worthy causes. There are many organizations much more worthy of my time and money than a bunch of managers who are too cheap to hire people for a follow-the-sun support org. But I'll accept a job with on-call if I think the rest of the deal outweighs the negatives.
It's still fair to call out on-call as a negative though. It's something to be aware of when accepting a job and for companies to keep in mind when recruiting.
I personally think that a much healthier alternative would be to look for a position where one can maintain their physical and mental health, and give back extra income to worthy causes. There are many organizations much more worthy of my time and money than a bunch of managers who are too cheap to hire people for a follow-the-sun support org. But I'll accept a job with on-call if I think the rest of the deal outweighs the negatives.
It's still fair to call out on-call as a negative though. It's something to be aware of when accepting a job and for companies to keep in mind when recruiting.
I'm honestly not sure how it's different from consultants who are only home on weekends after a week at a customer site in $RANDOM_CITY, engineers who have to basically live on a job site for weeks at a time, any number of professions that involve a lot of travel including weekend travel, or even the other salaried professions with on-call such as doctors.
If those all sound awful, by all means, do something else. But they're tradeoffs that people accept for their chosen job and compensation. My first job I would regularly be in shipyards and offshore for weeks at a time supervising some job. There was a modest allowance after some length of time but it paid better--and was almost certainly more interesting--than some routine office engineering job.
If those all sound awful, by all means, do something else. But they're tradeoffs that people accept for their chosen job and compensation. My first job I would regularly be in shipyards and offshore for weeks at a time supervising some job. There was a modest allowance after some length of time but it paid better--and was almost certainly more interesting--than some routine office engineering job.
>There is nothing immoral about deciding you’re happy.
It is when it sends a signal (about the job, market, etc) that also affects others.
It is when it sends a signal (about the job, market, etc) that also affects others.
No it is not. Because your signal is the truth for you.
Truth is not morality. "Truth for you" much less so.
All kinds of scumbags have their "personal truth" that justifies their actions too...
All kinds of scumbags have their "personal truth" that justifies their actions too...
The truth is always moral
Under what moral framework?
Not moral, practical. The majority of places that do on-call do it because it's the default.
That's the problem with on-call: it somehow took on moral undertones. And as a result it does not feel safe to speak out against it.
If on-call was widely seen as a negative (that a few people like because it gives them a sense of importance, more power to them) then there would be far fewer companies pushing for it as the default. As it stands, most people suffer silently for lack of an alternative. And the first step towards change is to make it OK to publicly say that on-call's a negative, a health hazard, and other options exist (though they may cost more).
That's the problem with on-call: it somehow took on moral undertones. And as a result it does not feel safe to speak out against it.
If on-call was widely seen as a negative (that a few people like because it gives them a sense of importance, more power to them) then there would be far fewer companies pushing for it as the default. As it stands, most people suffer silently for lack of an alternative. And the first step towards change is to make it OK to publicly say that on-call's a negative, a health hazard, and other options exist (though they may cost more).
If you believe that software engineers are not paid enough and it's morally important to do something about it then apparently I sent the best signal of all by retiring and thereby increasing demand for everyone else. But since this is absurd, I won't pat myself on the back too much.
More generally, free markets are doing absurd things all the time (see Matt Levine) which makes it hard to extend moral reasoning very far without getting the equivalent of divide-by-zero errors. So I don't think we should be all that concerned about how it affects the job market in general when negotiating with an employer. You know what you want better than you know what anyone else wants. Ask to be Paid More (tm) if that's what you want, but if you don't want to, you don't have to and people saying there is some kind of moral imperative to try to become even more wealthy at a faster rate can be ignored.
More generally, free markets are doing absurd things all the time (see Matt Levine) which makes it hard to extend moral reasoning very far without getting the equivalent of divide-by-zero errors. So I don't think we should be all that concerned about how it affects the job market in general when negotiating with an employer. You know what you want better than you know what anyone else wants. Ask to be Paid More (tm) if that's what you want, but if you don't want to, you don't have to and people saying there is some kind of moral imperative to try to become even more wealthy at a faster rate can be ignored.
The CEOs of every Fortune 500 company (and many smaller ones too) are on-call 24/7.
1. They get paid 10's of millions of $'s a year.
2. I doubt this is true. I've worked (and do work) for fortune 500 companies and have never ever heard of a CEO being paged for some sort of emergency. Presumably there can be some sort of crisis where they'll have to be involved within some reasonable time (extremely rare) but that's not exactly the same thing. That's why they have people working for them. That's not to say that some CEOs (especially for smaller companies) don't work very hard.
2. I doubt this is true. I've worked (and do work) for fortune 500 companies and have never ever heard of a CEO being paged for some sort of emergency. Presumably there can be some sort of crisis where they'll have to be involved within some reasonable time (extremely rare) but that's not exactly the same thing. That's why they have people working for them. That's not to say that some CEOs (especially for smaller companies) don't work very hard.
They definitely get involved when there's a meaningful impact on the company operations. Every business continuity plan involves top management; I've personally seen it in mid-size companies, and for the really big megacorps, just recently I was looking at a case study on the Maersk (#297 in fortune 500) management response to a cyberattack and it pretty much starts with the chairman being woken up at 4am local time. (e.g. source at https://www.bleepingcomputer.com/news/security/maersk-reinst... )
"presumably there can be some sort of crisis [...] but that's not exactly the same thing." - I'd say that it's exactly the same thing; on-call engineers are (or should be) just on an escalation path for crisis/emergency events; top managers are higher up in the escalation chain (if it's not a routine thing that others can easily resolve) but they're on escalation chain for every aspect of a large company. Of course, those events need to be rare - for example, as the original article states, up to 2-3 actual calls per year for a person being on-call; we should probably make a serious distinction between "on-call for emergencies" (which should be rare) and "routine out-of-hours support" (which might get triggered every week or even more frequently, obviously not an extraordinary event but a standard business process), which is something quite different and should have different solutions than emergency escalation.
"presumably there can be some sort of crisis [...] but that's not exactly the same thing." - I'd say that it's exactly the same thing; on-call engineers are (or should be) just on an escalation path for crisis/emergency events; top managers are higher up in the escalation chain (if it's not a routine thing that others can easily resolve) but they're on escalation chain for every aspect of a large company. Of course, those events need to be rare - for example, as the original article states, up to 2-3 actual calls per year for a person being on-call; we should probably make a serious distinction between "on-call for emergencies" (which should be rare) and "routine out-of-hours support" (which might get triggered every week or even more frequently, obviously not an extraordinary event but a standard business process), which is something quite different and should have different solutions than emergency escalation.
I also find it baffling that people still think about things the way you do.
It's not necessarily some kind of inferiority complex or anything else, it's about the free market.
It's not necessarily some kind of inferiority complex or anything else, it's about the free market.
"Free" market just means people bound by monetary considerations...
It means no one has to trade with you if they don't like what you have to offer in return. If you consider that "bound", then OK, but that language doesn't illuminate, it hides.
It is a painful slap in the face when the market doesn't price what you have to offer as high a you like. I know a friend who was convinced that her calling was to be an artist, but always had a hard time selling her works, and often complained about the unfairness of life. I guess she was "bound" by monetary considerations by not being able to make ends meet as an artist. Eventually she gave up and became a hospital lab tech and is well compensated. So the market was telling her that her skills as a hospital lab tech were much more valuable to society than her skills as an artist, even if her own preferences were otherwise.
At the same time, some other artist can buy an entire oceanside condo for one painting, because the market does value their output very highly.
That's all that we're talking about here. It is "free" but that doesn't mean that you'll be able to get whatever you want in exchange for your own output.
It is a painful slap in the face when the market doesn't price what you have to offer as high a you like. I know a friend who was convinced that her calling was to be an artist, but always had a hard time selling her works, and often complained about the unfairness of life. I guess she was "bound" by monetary considerations by not being able to make ends meet as an artist. Eventually she gave up and became a hospital lab tech and is well compensated. So the market was telling her that her skills as a hospital lab tech were much more valuable to society than her skills as an artist, even if her own preferences were otherwise.
At the same time, some other artist can buy an entire oceanside condo for one painting, because the market does value their output very highly.
That's all that we're talking about here. It is "free" but that doesn't mean that you'll be able to get whatever you want in exchange for your own output.
The free market works the other way too though. If lab techs were in high demand (as they are), and your friend's employer was demanding an unreasonable schedule, she'd be free to jump ship to a better job. Right now software engineers have that kind of market power, so why not use it? If tomorrow the market decides we are overpaid, we'll either accept it or change jobs like your friend did. What I don't get is not using the power the free market gives you because?...
Feel free to drive down your own value. This is the exactly what people are talking about when they say that they are looking for young and naive employees to exploit who won’t know better. You’re paid the big bucks because you have a very in-demand skill. Being dumb enough to answer a call at 3am is entirely orthogonal.
Please heed this. This is the lesson you don’t want to learn from experience
Why not?
Because, you undervalue your time, you open yourself for abuse. May be that project that is behind can be done by you after work, as you sometimes do work then anyways. I have seen some managers who are acutely sensitive to this and repeatedly use this to their advantage while charming the underlings. Next, being on call means it is difficult to plan that long vacation, or a unplanned night of shit faced drinking with your buddies, or unable to sit with your kid in the night when she is sick. A perspective of life as unpredictable events helps reinforce this idea. You are, with out compensation making life predictable for others while making it less predictable to you. Third, your mental and physical health. Sleep disorders because of restlessness, groggy mornings and the stress that comes with normal working hours will take a toll over long term.
Let's do the laziest back of the napkin math. Let's pretend you make $100k and work 2000 hours a year. You make $50/hr. Let's say you do 50 hours of week between trying to impress your boss and getting woken up to do call. Congratulations, you are now a $40/hr employee!
Also, in my experience, the people who aren't getting woken up at 3am (aka. your boss) don't value the time and effort needed to stabilize these systems. So they want you working on the new shiny product feature that will get them promoted. Fixing the crappy data pipeline that shouldn't alert every other night isn't a priority when you aren't the one being woken up.
Well, it should be a priority, but I also don't work at that previous job for this exact reason. I took a nice pay increase and have never had to be on-call. Don't settle.
EDIT: Original 2000 hours comes from 40 hours a week * 50 weeks. Like I said, back of the napkin math here.
Also, in my experience, the people who aren't getting woken up at 3am (aka. your boss) don't value the time and effort needed to stabilize these systems. So they want you working on the new shiny product feature that will get them promoted. Fixing the crappy data pipeline that shouldn't alert every other night isn't a priority when you aren't the one being woken up.
Well, it should be a priority, but I also don't work at that previous job for this exact reason. I took a nice pay increase and have never had to be on-call. Don't settle.
EDIT: Original 2000 hours comes from 40 hours a week * 50 weeks. Like I said, back of the napkin math here.
> Also, in my experience, the people who aren't getting woken up at 3am (aka. your boss) don't value the time and effort needed to stabilize these systems.
There's the rub. The issue isn't being on-call, it's not prioritizing making sure the system is robust so that on-call is boring.
In my last job there was no formal on-call, but if shit was going bad I'd be expected to resolve it, or track down the right resources to do so. In my current there is a formal on-call rotation. In my 15 years at the previous job I probably got called out of bed 3-4 times (and due to my roll I was the first call, if anyone got woken up, I did, and then had to wake anyone else needed). In my 7 months in the new job it hasn't happened yet.
When everything is on fire, it feels obvious to me that asking your $x00k employee to put it out isn't unreasonable. What is unreasonable is making that the plan instead of having robust fire-prevention systems and making that the exception.
There's the rub. The issue isn't being on-call, it's not prioritizing making sure the system is robust so that on-call is boring.
In my last job there was no formal on-call, but if shit was going bad I'd be expected to resolve it, or track down the right resources to do so. In my current there is a formal on-call rotation. In my 15 years at the previous job I probably got called out of bed 3-4 times (and due to my roll I was the first call, if anyone got woken up, I did, and then had to wake anyone else needed). In my 7 months in the new job it hasn't happened yet.
When everything is on fire, it feels obvious to me that asking your $x00k employee to put it out isn't unreasonable. What is unreasonable is making that the plan instead of having robust fire-prevention systems and making that the exception.
> Congratulations, you are now a $40/hr employee!
Another way of thinking about this is that you've invested 10 dollars of your paycheck into your career. I worked way more than 50 hours a week when I started my career. Subsequently, I was able to drop out of school early (by 2 years, saving 10s of thousands of dollars), get a full time job, increase my compensation by 50% within 2 years, and then by 200% the following year when I changed companies.
By your math I'm sure I was getting paid 1/3rd of my actual compensation. But that has more than paid off in terms of the investment.
Another way of thinking about this is that you've invested 10 dollars of your paycheck into your career. I worked way more than 50 hours a week when I started my career. Subsequently, I was able to drop out of school early (by 2 years, saving 10s of thousands of dollars), get a full time job, increase my compensation by 50% within 2 years, and then by 200% the following year when I changed companies.
By your math I'm sure I was getting paid 1/3rd of my actual compensation. But that has more than paid off in terms of the investment.
You're making the assumption that being on-call made some kind of difference to your career progression--as other commenters have noted, no one is doing good or interesting work during on call hours. You're just the company bitch and you get to do bitch work. Other industries require hazing like this too: finance comes to mind. But not every software firm requires on-call; you just picked the wrong company and allowed yourself to get screwed.
As a manager, yes it did. Managers are simple people. All they do most of the time is try to figure out who is critical and who is expendable (invest or tolerate). While it's not the only path to the invest bucket, putting out a fire in the middle of the night is a quick way to land there. It's often a good way to prove yourself in early career scenarios when you don't have a ton of architectural pull power.
> You're making the assumption that being on-call made some kind of difference to your career progression
Two things.
1) I didn't just put in on-call overtime. I put in "free" hours in general.
2) I'm not really making a big assumption. Unsurprisingly working a lot of hours gave a very positive impression of me and I was able to easily justify raises and promotions when I asked.
Calling on-call hazing or bitch work is stupid, I'm not engaging with this.
Two things.
1) I didn't just put in on-call overtime. I put in "free" hours in general.
2) I'm not really making a big assumption. Unsurprisingly working a lot of hours gave a very positive impression of me and I was able to easily justify raises and promotions when I asked.
Calling on-call hazing or bitch work is stupid, I'm not engaging with this.
I was on a team that had lost a lot of it's members. We worked hard to keep up and were on track to meet deadlines. Some time went by, we were working nights and weekends and every time we asked we were assured that they were hiring but "couldn't find the right person". At the same time, every week we would hear about a new hire in management. They were magically able to find some new hires as soon as deadlines started to slip, had we not worked those nights and weekends I am certain that they would've found someone to hire much earlier (the first guy they did hire said the process took three months, they were clearly dragging their feet). Too bad I learned that lesson after already spending the day of my daughters birth in long meetings about testing environments.
Why would they pay for something they can get for free?
Why would they pay for something they can get for free?
Speaking from experience, the ensuing burnout and lack of sleep leading to other health challenges isn’t worth the “life lesson”.
Downvoted for insulting someone you don’t know anything about.
Downvoted the above for hollier-than-thou preaching and missing the point.
same
Being on call is separable from your role as a software developer and can be compensated separately. My wife’s (non-technical) organization has a requirement for a junior and a senior duty officer to be available 24/7/365. These shifts are compensated and available for people to sign up for. If shifts don’t get filled, they fall to senior managers in a rotation. You can be sure these managers somehow pay enough that they only very rarely need to take shifts themselves.
>I’m a software engineer. I’m paid well partly because things like oncall are necessary. It’s priced into the compensation.
Or you priced yourself down.
Or you priced yourself down.
> It sounds like you should find a company that agrees with this opinion. I don’t agree.
I agree that on-call is necessary in certain professions (e.g, doctors). I also agree that if an employee is willing to do on-call and they are compensated accordingly, then the practice is still ethical.
However, to call someone to do work outside work hours is unreasonable. On-call is considered work time, so I am expecting to be contacted during that time. However, if I'm not on-call, then it is not time for me to work, and I shouldn't be contacted by my company and feel pressured to answer the call.
I agree that on-call is necessary in certain professions (e.g, doctors). I also agree that if an employee is willing to do on-call and they are compensated accordingly, then the practice is still ethical.
However, to call someone to do work outside work hours is unreasonable. On-call is considered work time, so I am expecting to be contacted during that time. However, if I'm not on-call, then it is not time for me to work, and I shouldn't be contacted by my company and feel pressured to answer the call.
I think a lot of founders would compare their startup to having a baby. Imagine if someone said “I want to have a baby but absolutely no waking up at 2AM”, you would be laughed at.
Likewise, lots of engineers in tech are compensated in equity so you have a stake in the business. in that case the lines on what constitutes personal time are less defined. I don’t see it as a bad thing if I’m on vacation and need to grab a laptop and help on an incident. I’ll take an extra day of vacation later or sleep in the next day. Life doesn’t have to be so inflexible.
Likewise, lots of engineers in tech are compensated in equity so you have a stake in the business. in that case the lines on what constitutes personal time are less defined. I don’t see it as a bad thing if I’m on vacation and need to grab a laptop and help on an incident. I’ll take an extra day of vacation later or sleep in the next day. Life doesn’t have to be so inflexible.
> Imagine if someone said “I want to have a baby but absolutely no waking up at 2AM”, you would be laughed at.
Okay, now imagine that I have a baby. I offer you a .0001% ownership in the baby, and you have to get (potentially, but often actually) woken up for 1 week out of every 6.
Does that sound like an arrangement that a rational, healthy, self actualized person would agree to? Of course not. You're getting all of the downside and none of the upside. In fact in many cases the "leadership" team themselves aren't even on call!
Okay, now imagine that I have a baby. I offer you a .0001% ownership in the baby, and you have to get (potentially, but often actually) woken up for 1 week out of every 6.
Does that sound like an arrangement that a rational, healthy, self actualized person would agree to? Of course not. You're getting all of the downside and none of the upside. In fact in many cases the "leadership" team themselves aren't even on call!
Parents wake up for their own babies. Very few engineers have a meaningful stake in the business. And on call hours aren't proportional to equity generally. Imagine if someone said they expected someone else to wake up for their baby in exchange for lottery tickets.
They said on call is ethical if an employee is willing and compensated accordingly. Additional paid time off is compensation.
They said on call is ethical if an employee is willing and compensated accordingly. Additional paid time off is compensation.
I partially agree. It really depends on how often I'm called when on-call.
There is a world of difference between being called twice per year, and twice per week.
...and it's fine for that to be priced in as long as those expectations are as clear as the salary when I take the job.
There is a world of difference between being called twice per year, and twice per week.
...and it's fine for that to be priced in as long as those expectations are as clear as the salary when I take the job.
Are you very young?
I feel like young people are more exploitable and don't understand what they're sacrificing by being available all the time. I was the same way at the beginning of my career. Fortunately I had a good team, but there were definitely times where my sleep and health suffered from being on-call because I didn't know how to say no.
I feel like young people are more exploitable and don't understand what they're sacrificing by being available all the time. I was the same way at the beginning of my career. Fortunately I had a good team, but there were definitely times where my sleep and health suffered from being on-call because I didn't know how to say no.
But loads of software engineers don't have oncall. And these jobs don't pay less as a rule. If you aren't at least getting overtime pay, then you are being scammed.
> I’m a software engineer. I’m paid well partly because things like oncall are necessary. It’s priced into the compensation.
It might be priced into your compensation but that's the exception. If you are paid "market rate" and on call is "priced in" then you're underselling your services by a large margin.
It might be priced into your compensation but that's the exception. If you are paid "market rate" and on call is "priced in" then you're underselling your services by a large margin.
[deleted]
Haha, That's why you still have to do on call.
[deleted]
Doctors are on call. Firemen are on call.
There are plenty of jobs that are not on call. Lots of enterprise positions do not require you to be on call. If you work on a site that requires 24/7 uptime, why is it so hard to accept that engineers have to be on call?
There are plenty of jobs that are not on call. Lots of enterprise positions do not require you to be on call. If you work on a site that requires 24/7 uptime, why is it so hard to accept that engineers have to be on call?
Most Doctors are not on call. Police are not on call. Firemen being on call sounds unbelievable from a union perspective but many volunteer fire departments exist.
On call for a week every 8 weeks is a lot different to a once a year event like a manager getting a call that a break-in happened at a store.
On call for a week every 8 weeks is a lot different to a once a year event like a manager getting a call that a break-in happened at a store.
Firefighters in the US typically work 24hr shifts, then have 48hr off, then repeat[1]. You could say they are 'on call' during that 24hr shift, of course they don't normally fight fires every minute of it (luckily), much of that time is spent doing things in the firehouse like maintenance, eating, etc.
1: https://firefighternow.com/firefighter-shift-schedules-and-w...
1: https://firefighternow.com/firefighter-shift-schedules-and-w...
[deleted]
I really hate on-call, sadly in any serious distributed systems used by many customers/running large scale online services on-call is table stakes :|
It's not like SWEs are paid by hour. If you are on call for a week every 2-3 months (for example), you could just consider the extra 2 work weeks (assuming 8 hour days for a normal week) of part-time work to be part of your paycheck.
Staffing competent people 24x7 is completely impractical in a sufficiently complex system... what you can have, and I have experienced, is some clueless 24x7 first-tier support. These guys don't work on the product so they usually cannot solve any problem that is not completely basic. And if they can't solve a problem, they'd call a component owner (or a random person) at any time, which is worse than someone being on-call. If you have a rotation, you can safely disappear when you are not on, because there's someone competent responsible for dealing with live site.
It's not like SWEs are paid by hour. If you are on call for a week every 2-3 months (for example), you could just consider the extra 2 work weeks (assuming 8 hour days for a normal week) of part-time work to be part of your paycheck.
Staffing competent people 24x7 is completely impractical in a sufficiently complex system... what you can have, and I have experienced, is some clueless 24x7 first-tier support. These guys don't work on the product so they usually cannot solve any problem that is not completely basic. And if they can't solve a problem, they'd call a component owner (or a random person) at any time, which is worse than someone being on-call. If you have a rotation, you can safely disappear when you are not on, because there's someone competent responsible for dealing with live site.
I don't think it's abusive. If you rotate on a team of 8, that should mean you go on-call once every 8 weeks (assuming weekly on-call). This really isn't bad in my opinion. Especially if the systems are decently architected and maintained.
The real problem is that management doesn't like to pay for updates/upgrades/fixes to make the system more stable (tying back to the management/engineering paragraph in the article).
Disclosure: I'm currently on call.
The real problem is that management doesn't like to pay for updates/upgrades/fixes to make the system more stable (tying back to the management/engineering paragraph in the article).
Disclosure: I'm currently on call.
Er, what? On call for an entire week at a time?
Does this actually exist? It's like an... anti-holiday. Insane.
Does this actually exist? It's like an... anti-holiday. Insane.
Our company does that. Each person in the support team has one week on call. The following week they get the Friday off, and they get some extra compensation in addition just for being on call. Overtime if they have to work of course.
Only customers who pay a lot for it have true 24/7, most who pay extra have extended support from IIRC 06-22.
If they can't sort it out, support will try as best they can to work around issues as to not disturb us devs until next morning. I've been woken at most once a year.
Most support folks do say it can be a bit stressful at times, on the other hand they also really enjoy the extended weekend so nobody's been keen to change it.
Part of why it's so useful to do a full week at a time, rather than rotate intra-week, is that whoever's on call needs to have a small purse full of token generators for 2-factor logins, and we typically only get one to share. If they don't have the token they can't help our customer.
Only customers who pay a lot for it have true 24/7, most who pay extra have extended support from IIRC 06-22.
If they can't sort it out, support will try as best they can to work around issues as to not disturb us devs until next morning. I've been woken at most once a year.
Most support folks do say it can be a bit stressful at times, on the other hand they also really enjoy the extended weekend so nobody's been keen to change it.
Part of why it's so useful to do a full week at a time, rather than rotate intra-week, is that whoever's on call needs to have a small purse full of token generators for 2-factor logins, and we typically only get one to share. If they don't have the token they can't help our customer.
I mean it really depends on what the oncall looks like. If your system breaks every 5 minutes or you have other teams escalating non-urgent tasks constantly then 1 week on call seems crazy. But there are many companies (mine included) where on-call basically means: If services go down you have to figure out why. No bug fixing etc. Oncall purely from a infrastructure perspective.
Yes it does exist. I've done plenty of on-call shifts but for paid a minimum of three hours whether I solved the problem in 5 mins or 50 mins. It's great when you're young, starting out, as it's a big pile of money. When you're over thirty, not so much.
Isn't bad and abusive are two ways to describe the same event.
> There is no situation where it’s ok to contact someone to do work outside of their work hours.
This lacks perspective.
Imagine that your system runs banking, healthcare, telecom, etc.
Of course someone needs to be on call 24/7. The best people to hold the pager are the engineers that work on the system.
This lacks perspective.
Imagine that your system runs banking, healthcare, telecom, etc.
Of course someone needs to be on call 24/7. The best people to hold the pager are the engineers that work on the system.
And when they hold the pager, they should get to do nothing else during the day - no feature dev etc. I don't care if there's a deadline..you should have accounted for on-call when planning.
The only thing they should be doing besides responding to incidents is work to stabilize the system to reduce the need to page people.
That's a fair way to make it work imo.
The only thing they should be doing besides responding to incidents is work to stabilize the system to reduce the need to page people.
That's a fair way to make it work imo.
that's a pretty selective quote. you skipped over:
> If an organization thinks their systems should be available 24x7 they should staff people 24x7.
One way to achieve 24x7 coverage without anyone being structurally on-call outside of standard office hours could be "follow the sun" support -- have offices in multiple timezones around the world and run 3 x 8 hour shifts each day of support working their local office hours to provide 24/7 coverage. Still need enough staff at each site to handle rotation for covering weekends and for people to go on holiday and get sick or quit for new jobs etc.
I can understand why many companies would prefer not to pay for that if they can get away with cheaper alternatives, and why they might prefer to understaff and push the burden onto employees, but that's the company's problem, not the problem of the individual worker who trades their life by the hour in exchange for money.
> If an organization thinks their systems should be available 24x7 they should staff people 24x7.
One way to achieve 24x7 coverage without anyone being structurally on-call outside of standard office hours could be "follow the sun" support -- have offices in multiple timezones around the world and run 3 x 8 hour shifts each day of support working their local office hours to provide 24/7 coverage. Still need enough staff at each site to handle rotation for covering weekends and for people to go on holiday and get sick or quit for new jobs etc.
I can understand why many companies would prefer not to pay for that if they can get away with cheaper alternatives, and why they might prefer to understaff and push the burden onto employees, but that's the company's problem, not the problem of the individual worker who trades their life by the hour in exchange for money.
If you're not a global empire with offices all over the world yet, the simpler thing is to pay your engineers well and tell them being oncall is one of the responsibilities.
If you are not a global company being down at 3am doesn't matter either.
Because of course all companies exist in the same time zone as all their clients... Have you ever worked at a SaaS company? Having customers from around the world is literally a thing from day 1, especially if you are not US or (to a lesser extent already) EU based. Try starting a SaaS company in Australia and see how far "doesn't matter if it goes down at 3am" gets you.
Starting a company in Australia and targeting a completely different timezone and not staff support at 3am when traffic is heaviest seems like gambling.
Arguably the importance of availability and time to restore service is less about if the company is global and more about the nature of the services they provide to their customers.
Global ecommerce company letting people buy junk they don't need 24/7 at low low prices? An outage may frustrate customers and lose some sales but they might get most of the customers back in the long run, even if it takes them days to get the site to come back up and stay up.
Bits of the local electricity grid get damaged? Communications for ambulance dispatch go down and stay down? Those outages may directly cause fatalities or greatly increase the risk of fatalities until problems are safely isolated and service can be restored.
Global ecommerce company letting people buy junk they don't need 24/7 at low low prices? An outage may frustrate customers and lose some sales but they might get most of the customers back in the long run, even if it takes them days to get the site to come back up and stay up.
Bits of the local electricity grid get damaged? Communications for ambulance dispatch go down and stay down? Those outages may directly cause fatalities or greatly increase the risk of fatalities until problems are safely isolated and service can be restored.
Staffing folks who can fix these systems 24/7 for the 2-3 times a year something needs to happen doesn't seem like a great idea to me. If the systems are failing left and right continually, or if on-call is used to handle things that can be done from a simple runbook? Sure. But at some point you need an escalation path when its more than that, and sooner or later that escalation path goes to whoever built the thing.
These are usually salaried jobs, so "outside of work hour" is kind of subjective, since you're paid to get the job done.
So then it comes to expectations, and $$$$. Software engineer salaries are not just a result of the supply vs demand (well, everything is supply vs demand, of course, but there's subtleties).
It wasn't that long ago (late 90s, early 2000s) that a 9-5 code monkey job was a low paying job. It makes sense: it's not particularly difficult. But then organizations realized that they could get more bangs for their bucks if the same folks could handle uncertainties, if they would train themselves, if they could architect systems on their own, and yes, if they were responsible for keeping the systems running. That, of course, came at a cost. In addition to growing demands raising wages, you also need to pay a premium to get someone to agree to all that crap. And so they did.
But now as so many people entered the field, there's an expectation that the salaries are the baseline, and that the other responsibilities beyond code monkeying are a premium.
Supply and demand will kick in sooner or later, when folks realize that 6 figure salaries for a 9-5 desk job is a really good deal (we already see it happening, where entry level roles are getting hard to come by).
But in the end, the additional duties, like oncall, really have been baked in the salaries. I really don't think its unreasonable to have someone woken up a handful of times when taking salaries that can go as high as competing with a doctor's... (who's also on call).
Now, that's within reason. My partner worked at AWS for a while, and they'd get paged 5-6 times a night on a bad week. F* that. No amount of money is worth screwing over your health.
These are usually salaried jobs, so "outside of work hour" is kind of subjective, since you're paid to get the job done.
So then it comes to expectations, and $$$$. Software engineer salaries are not just a result of the supply vs demand (well, everything is supply vs demand, of course, but there's subtleties).
It wasn't that long ago (late 90s, early 2000s) that a 9-5 code monkey job was a low paying job. It makes sense: it's not particularly difficult. But then organizations realized that they could get more bangs for their bucks if the same folks could handle uncertainties, if they would train themselves, if they could architect systems on their own, and yes, if they were responsible for keeping the systems running. That, of course, came at a cost. In addition to growing demands raising wages, you also need to pay a premium to get someone to agree to all that crap. And so they did.
But now as so many people entered the field, there's an expectation that the salaries are the baseline, and that the other responsibilities beyond code monkeying are a premium.
Supply and demand will kick in sooner or later, when folks realize that 6 figure salaries for a 9-5 desk job is a really good deal (we already see it happening, where entry level roles are getting hard to come by).
But in the end, the additional duties, like oncall, really have been baked in the salaries. I really don't think its unreasonable to have someone woken up a handful of times when taking salaries that can go as high as competing with a doctor's... (who's also on call).
Now, that's within reason. My partner worked at AWS for a while, and they'd get paged 5-6 times a night on a bad week. F* that. No amount of money is worth screwing over your health.
Industrial companies that have an assembly line go down call people in to fix it at time and a half or double pay.
Something is a little fucked with tech for doing it for free.
Something is a little fucked with tech for doing it for free.
The tech industry is filled with a special kind of person who thinks they are the best at what they do. They get paid a lot (for now) so they make the mistake of thinking they’re at the same level of real professionals like doctors, lawyers, and engineers. The industry encourages this delusion by throwing around essentially made-up feel-good titles like “software engineer”, “senior software engineer”, “staff software engineer”, etc.
When these people begin to doubt themselves and their line of work, they’re told they have “impostor syndrome” and everything is fine.
The reason the industry does all of this is because it discourages their workers from forming unions. They trick their workers into thinking unions will slow them down.
There will come a day when the industry is flooded with enough saps that their high salary is pulled out from under them and they’ll understand the reality of the situation. They’ll be left working for pennies and at odd hours of the day.
When these people begin to doubt themselves and their line of work, they’re told they have “impostor syndrome” and everything is fine.
The reason the industry does all of this is because it discourages their workers from forming unions. They trick their workers into thinking unions will slow them down.
There will come a day when the industry is flooded with enough saps that their high salary is pulled out from under them and they’ll understand the reality of the situation. They’ll be left working for pennies and at odd hours of the day.
That day will literally never come. Software is too hard for average people to get good at in numbers that will displace our salaries.
Covid remote jobs are incapable of lowering USA salaries despite the fears of outsourcing. Your fears of people thinking they are better than they actually are are wrong - and so is the implication that "we will get what we deserve" - well, unless what we deserve is another doubling of compensation for switching jobs every 2 years...
Covid remote jobs are incapable of lowering USA salaries despite the fears of outsourcing. Your fears of people thinking they are better than they actually are are wrong - and so is the implication that "we will get what we deserve" - well, unless what we deserve is another doubling of compensation for switching jobs every 2 years...
And at that point unions will make sense, like in any other low paid, undifferentiated type of work where the worker has a really hard time "just changing jobs" when things get abusive. That is not the situation in IT now.
The sensible time to use bargaining power is before you lose it.
You should also be compensated for the time you spend on-call (which means you can't take on other commitments you otherwise might.) Anything less and it's outright exploitation. "We pay you well" is a non-sequitur.
> "We pay you well" is a non-sequitur.
Not at all. The choice might be between $60k salary plus around $20k for overtime in one career, vs another career with $200k+ salary plus crazy benefits (gym memberships, a variety of mental health services, $20k in fertility treatments (no joke), 3+ months paternity leave, over 10 more things I can't even name they make no sense...)
I don't even use any of the benefits, just the money (and the parental leave) but calling a few on-call nights per month "abuse" or "exploitation" is just ... without perspective. Nobody that does real work has it this cushy - not doctors, not teachers, not construction workers. Maybe aristocratic political appointees or something, but jeez, no field that is accessible to anyone with an old laptop and sufficient motivation. This is as cushy as it gets, for a real job. Enjoy it while it lasts.
Not at all. The choice might be between $60k salary plus around $20k for overtime in one career, vs another career with $200k+ salary plus crazy benefits (gym memberships, a variety of mental health services, $20k in fertility treatments (no joke), 3+ months paternity leave, over 10 more things I can't even name they make no sense...)
I don't even use any of the benefits, just the money (and the parental leave) but calling a few on-call nights per month "abuse" or "exploitation" is just ... without perspective. Nobody that does real work has it this cushy - not doctors, not teachers, not construction workers. Maybe aristocratic political appointees or something, but jeez, no field that is accessible to anyone with an old laptop and sufficient motivation. This is as cushy as it gets, for a real job. Enjoy it while it lasts.
Agreed.
My brother makes about what I do, but he is an MBA and a plant manager with background in logistics.
Sometimes he gets calls and has to go in to work. Hell there has been at least one instance where he had to drop everything and courier a bag of parts on a commercial flight so a line wouldn't stop.
That said, I have worked in 'abusive' on call environments. I.e. we had a system that would break at least once a week between 3am and 6am (like, multiple times in that period). But because the oncall labor to remediate was 'free', fixing the problem was never a priority over the 2 years I worked there.
My brother makes about what I do, but he is an MBA and a plant manager with background in logistics.
Sometimes he gets calls and has to go in to work. Hell there has been at least one instance where he had to drop everything and courier a bag of parts on a commercial flight so a line wouldn't stop.
That said, I have worked in 'abusive' on call environments. I.e. we had a system that would break at least once a week between 3am and 6am (like, multiple times in that period). But because the oncall labor to remediate was 'free', fixing the problem was never a priority over the 2 years I worked there.
[deleted]
“We pay you well” just means “the labor market favors us so much that we can hire a replacement for you to do it.” Charitably, it’s because their pay is above a willingness to work threshold in the market. As a market participant, the supply side of labor typically is too large, not at all organized, and lacks information and any market power to price these things into valuations.
It's the "superhero" complex so many software engineers have. The dopamine hit they get when they resolve some outage or thorny bug or whatever (and the ensuing accolades from their peers) interferes with their ability to reason clearly about their compensation and the circumstances of their employment.
I guess I don’t understand the line you’re drawing here. If they enjoy the circumstances of their work, why should they go out of their way to “reason clearly” about how it’s actually bad? I could understand if there’s a collective action problem, but as many people in this thread have attested, people who want a job without oncall duty can get one.
Remaining neutral on the on-call point, but as a practical/logical matter, things that feel good and make you happy can still be bad for you personally, especially if they're multiplied over the long term. Three or four gin and tonics every night feels great until your doctor reveals that your unexplained weight loss is from cirrhosis, or cancer.
I guess that's pretty fair. I have to admit I've seen people who genuinely enjoy some terrible overloaded schedule right up to the time they burn out.
Yeah the scenario where they're proud of some outage/bug they resolve and it's cool to discuss it. OK. Management should be asking: how do we make sure it never happens again?
Management should be looking to reduce callouts to zero over time.
Management should be looking to reduce callouts to zero over time.
They don't have a union so they hoover up the boss' arbitrary bullshit. Very simple. Most people just comply. They are the worse because they set unreasonable expectations for everyone else. Unions protect you from the other emoloyees' bad deals.
Submit an invoice for a minimum 1 hour charge for any after hours work/calls. Devs need to start billing like lawyers.
The "personal responsibility" narrative has seeped far to far into American life to be tolerable. Back then, we had industrial operations that were running 24 hours/day, things like factory assembly lines, steelmaking, the electric and gas utilities, and they had on-call rotas. But then we also had proper engineering practices, none of this "take ownership" crap, which reduced incidents, and we had unions that were in a position to push back on employer demands.
Instead we have self-indulgent junk like that. Wonder what my uncle would say, he was chief engineer for the gas company back in the day.
Instead we have self-indulgent junk like that. Wonder what my uncle would say, he was chief engineer for the gas company back in the day.
I always managed to get out of doing any on call work by pushing this point. There was a price for being put in the list of numbers that would reach my phone during non work hours and then further charges if I had to answer a call. Was always a pretty tense discussion with managers but they never wanted to pay for it and I never worked for free.
It seems perfectly okay if you were hired at a job where on-call is a stated requirement. Is this a controversial opinion? I’ll admit that I sometimes find myself out of phase with some of the dominant HN views on at-will employment, but it would be useful for me to know what the perceived flaw in my stance is.
No.
If a job is expecting you to be oncall is the part of your job, they are looking for ways to not pay you for the work. (This is not talking about an occasional once a year expectation. ) This is a rotating or permnant basis of where you are expected to be ready to work and they're not paying you.
If a job is expecting you to be oncall is the part of your job, they are looking for ways to not pay you for the work. (This is not talking about an occasional once a year expectation. ) This is a rotating or permnant basis of where you are expected to be ready to work and they're not paying you.
If someone offers you money while at the same time providing a list of things you need to do to get that money, they are paying you to do those things.
You are agreeing for them not to pay you standard pay for 24 hour availability. The company is getting out of paying for that availability.
So if you insist on not doing oncall and they reduce your salary, were you getting paid for oncall or not?
If you are expected to be on call you should calculate that as part as your hourly salary. You can quickly be a person that is working below minimum wage.
So, can this mythical 24x7 support the code you wrote? Did you write documentation that explains every case that could go wrong?
If your team writes code that doesn’t need to be supported by an engineer, then your team won’t get called.
If your team writes code that doesn’t need to be supported by an engineer, then your team won’t get called.
> If an organization thinks their systems should be available 24x7 they should staff people 24x7.
This is hard to do well. Ideally, you need an eng culture where things are well-documented and product development is regularly handed off between multiple time zones. Otherwise, what happens is that you just end up with product development happening in one time zone and an SRE or Ops team getting screwed over by changes they don't understand.
This is hard to do well. Ideally, you need an eng culture where things are well-documented and product development is regularly handed off between multiple time zones. Otherwise, what happens is that you just end up with product development happening in one time zone and an SRE or Ops team getting screwed over by changes they don't understand.
> they should staff people 24x7
Perhaps, if the workload for the overnight team is low, instead of having them sit in an office from 1am to 6am in a dark room staring at a flickering screen you work on alternate solution for staffing.
How about, as a solutution to the problem, we let them work from home and provided they have all the tools at their disposal, upon recieving some kind of notification, attend to the situation.
Perhaps, if the workload for the overnight team is low, instead of having them sit in an office from 1am to 6am in a dark room staring at a flickering screen you work on alternate solution for staffing.
How about, as a solutution to the problem, we let them work from home and provided they have all the tools at their disposal, upon recieving some kind of notification, attend to the situation.
The way I've heard it works in the US seems abusive, but it doesn't need to be. I thought it was regulated in Spain (to give an alternative point of view), but apparently it's only lightly regulated and the pay is left to either the unions to negotiate or to the contract. However, it's regulated that:
- If you do have to work, that's "extraordinary" time paid at 1.75x the normal hourly rate (yes, including for full time employees).
- While you don't need breaks (it's kinda considered a break on itself), you do need at least 12h of rest between shifts, so that on itself makes it basically impossible to be on call the whole time between two work days. It makes it possible to be on call for the weekends though, and between two work days for 4h.
- If you do have to work, that's "extraordinary" time paid at 1.75x the normal hourly rate (yes, including for full time employees).
- While you don't need breaks (it's kinda considered a break on itself), you do need at least 12h of rest between shifts, so that on itself makes it basically impossible to be on call the whole time between two work days. It makes it possible to be on call for the weekends though, and between two work days for 4h.
The way my team handles this in the US and EU is ‘Best Effort,’ where you are provided a company phone and the expectation is that if you’re able to reasonably do so, answer the call. From there, you can triage (is the problem real, how critical is this, is there an alternate method to achieve your goal.)
Each team member has services and/or sites they are responsible for, but our team is flexible enough to work at any site as they’re heavily standardized. If you’re not able or willing to help (or go in if things are REALLY bad), as long as you can get ahold of someone else capable of helping or a supervisor and hand off to continue. If you need to go in, 2x pay that day and vacation time. It’s more if it’s disruptive (2 am friday vs 11 am on a Sunday.)
Out of 47 sites, this has happened one time in the past year where something went horrifically wrong with some Cisco gear configurations. We’re moving even further to get rid of having a company phone, and moving to a tablet that has our comms tools and enough to RDP somewhere. This moves from ‘answer the phone if you’re able to please’ to ‘if you get an urgent Teams notification check it out please if possible.’
Strict on-call is hard, but in our environment most people are willing to help one another out if something is up. This could mean immediate sudden travel, but only if you are willing and capable of doing so. I personally always accept them, because they’re mad fun road trips and you get time off, extra pay, and I like helping my team members when they’re in a pinch. People have lives, I just have a more flexible one :).
Each team member has services and/or sites they are responsible for, but our team is flexible enough to work at any site as they’re heavily standardized. If you’re not able or willing to help (or go in if things are REALLY bad), as long as you can get ahold of someone else capable of helping or a supervisor and hand off to continue. If you need to go in, 2x pay that day and vacation time. It’s more if it’s disruptive (2 am friday vs 11 am on a Sunday.)
Out of 47 sites, this has happened one time in the past year where something went horrifically wrong with some Cisco gear configurations. We’re moving even further to get rid of having a company phone, and moving to a tablet that has our comms tools and enough to RDP somewhere. This moves from ‘answer the phone if you’re able to please’ to ‘if you get an urgent Teams notification check it out please if possible.’
Strict on-call is hard, but in our environment most people are willing to help one another out if something is up. This could mean immediate sudden travel, but only if you are willing and capable of doing so. I personally always accept them, because they’re mad fun road trips and you get time off, extra pay, and I like helping my team members when they’re in a pinch. People have lives, I just have a more flexible one :).
[deleted]
[deleted]
Most jobs I've had there was an implicit expectation that if your shit breaks, you fix it. This resulted in systems which didn't break because, well, that keeps work to working hours.
A few years back I took a job with no mention of on call and then was told there would be a fortnightly 24 hour slot, which later turned into ~weekly as team members left.
I left too.
I have no idea why they felt it reasonable to not mention this before I joined, they effectively just wasted a few months pay on me on a gamble that I had no life outside of work.
They seemed to find the idea that I don't take my laptop to the pub, or up a mountain, or on holiday, some sort of bizarre way of living. Sick system effect? Who knows.
A few years back I took a job with no mention of on call and then was told there would be a fortnightly 24 hour slot, which later turned into ~weekly as team members left.
I left too.
I have no idea why they felt it reasonable to not mention this before I joined, they effectively just wasted a few months pay on me on a gamble that I had no life outside of work.
They seemed to find the idea that I don't take my laptop to the pub, or up a mountain, or on holiday, some sort of bizarre way of living. Sick system effect? Who knows.
> Most jobs I've had there was an implicit expectation that if your shit breaks, you fix it. This resulted in systems which didn't break because, well, that keeps work to working hours.
My experience was similar in my last long-term software role. We were not explicitly on call, but we developed/deployed/supported an app which was critical to a 24×7×365 business process. So we were careful in our testing, code reviews, and deployments.
One time in 7 years I had to answer an out of hours call because of an issue in this app.
I'll take that over semimonthly on-call rotas for suites of apps my team isn't isn't responsible for.
My experience was similar in my last long-term software role. We were not explicitly on call, but we developed/deployed/supported an app which was critical to a 24×7×365 business process. So we were careful in our testing, code reviews, and deployments.
One time in 7 years I had to answer an out of hours call because of an issue in this app.
I'll take that over semimonthly on-call rotas for suites of apps my team isn't isn't responsible for.
Sign of a wider lack of trust between employer and employee. Yet the idea of unionization is still very unpopular.
I don't follow.
Most jobs I've had were high trust, this was the weird exception. I didn't need a union to fix that.
Most jobs I've had were high trust, this was the weird exception. I didn't need a union to fix that.
It's not a rare phenomenon across the industry. Plenty of my colleagues and I have experienced something similar.
> if your shit breaks, you fix it
What if you're drunk?
What if you're drunk?
You think about that when you design it.
I used to work on trading systems which were managed in real time by a rotating team of traders (i.e. mathematically oriented people as opposed to software engineers).
You design the system so that if it breaks there are at least workarounds that anyone can employ without needing to write code. For example, a big stop button, manual adjustments, manual trading, etc.
Maybe it stops printing money overnight and you take an opportunity cost loss. That's fine, post mortem at work, fix it.
In the worst case if you're not available then someone with ownership steps in like a CTO/founder level (who are of course always on call almost by definition, though they generally have the executive power to say - sod this, we'll just leave it down for a while).
I used to work on trading systems which were managed in real time by a rotating team of traders (i.e. mathematically oriented people as opposed to software engineers).
You design the system so that if it breaks there are at least workarounds that anyone can employ without needing to write code. For example, a big stop button, manual adjustments, manual trading, etc.
Maybe it stops printing money overnight and you take an opportunity cost loss. That's fine, post mortem at work, fix it.
In the worst case if you're not available then someone with ownership steps in like a CTO/founder level (who are of course always on call almost by definition, though they generally have the executive power to say - sod this, we'll just leave it down for a while).
Seems like "if your shit breaks, you fix it" = on call 24/7... isn't it better to have a specific shift where you know you are responsible and then you aren't the rest of the time.
No, because in that case you are responsible for other people's systems you have no idea about and there is no incentive to fix them.
I am always on call to secure my own house, so I have a good alarm system, cameras, locks etc which means I hopefully don't have to do much. It doesn't keep me up at night when I'm on holiday because, well, it's overwhelmingly likely that nothing will happen.
If I were on call for the neighbourhood or general area once a week, I'd probably have to physically be there patrolling it because otherwise I have liability for things which are outside of my control.
I am always on call to secure my own house, so I have a good alarm system, cameras, locks etc which means I hopefully don't have to do much. It doesn't keep me up at night when I'm on holiday because, well, it's overwhelmingly likely that nothing will happen.
If I were on call for the neighbourhood or general area once a week, I'd probably have to physically be there patrolling it because otherwise I have liability for things which are outside of my control.
And if somebody ever leaves the company for any reason, who is on call for their stuff?
Projects get reassigned over time regardless.
It's not as if this is some strict 1:1 relationship, we obviously had more than 1 person with knowledge of / working on a codebase.
It would have been an issue if 2-3 working on the same thing left simultaneously, but for more reasons than just on-call.
It's not as if this is some strict 1:1 relationship, we obviously had more than 1 person with knowledge of / working on a codebase.
It would have been an issue if 2-3 working on the same thing left simultaneously, but for more reasons than just on-call.
maybe there is a big difference between my workplace and yours, but for me a lot of stuff never gets improved much after it's written so it's basically impossible to get up to speed until the person who owned it leaves the company and now you're she one fixing their bugs. Onboarding is just really hard on legacy projects.
Personally, I hate coordination and knowledge dissemination. It's better for me if I own my stack and support it 24/7 if that means I don't have to communicate details with other people regularly.
I also gravitate towards teams that work well without a lot of documentation, so that helps in case I'm really unavailable.
I also gravitate towards teams that work well without a lot of documentation, so that helps in case I'm really unavailable.
If any system needs 24/7 support and more than two nines of availability, that's simply not compatible with having a single person "owning the stack" and not documenting/disseminating the details. That's a bus factor of 1, and fails whenever you will get any actual time off offline (e.g. in a plane). So if the uptime of the system and 24/7 is actually important, I would consider that your manager is unacceptably negligent in allowing this to happen.
On the other hand, if 24/7/365 is more like a 'nice to have' option, then sure, such an arrangement will be more cost effective than more people in support; a 99% uptime SLA often can be maintained by one core person.
On the other hand, if 24/7/365 is more like a 'nice to have' option, then sure, such an arrangement will be more cost effective than more people in support; a 99% uptime SLA often can be maintained by one core person.
I mean, two nines is 3 and a half days of downtime. That's not hard to get as long as I don't make a habit of breaking shit in ways that take a long time to find out, the hardware is decent and I don't disappear for days very often. I'd add that the network should be decent too, but if it's not, my stuff should be fine.
A competent person can dig around and fix most anything good enough in three days and that leaves you 12+ hours for other outages.
Three nines is almost 9 hours a year, but planes almost universally have wifi now, even if it's not great. If you have a bad year, with two big incidents when you're on a plane, you might not make your goal.
At four nines, it's about an hour. Good luck with that one. You're kind of screwed either way. Someone with intimitate knowledge of the system can probably handle an incident or three in time, but it's an enormous amount of work to get people up to speed for that and you can't have very many live training sessions because you don't have the outage budget. At the same time, if you page me and I'm at the cinema, I'm not going to fix it in time.
But, the biggest thing is working to make failure as graceful as possible. As much as possible, each component should be able to fail individually, without affecting the rest of the system. And, when there is an outage of a critical dependency, you need a way to push that outage status upstream and a method to manage load when it comes back up. (Those knobs should be stable and can certainly be documented for others).
A competent person can dig around and fix most anything good enough in three days and that leaves you 12+ hours for other outages.
Three nines is almost 9 hours a year, but planes almost universally have wifi now, even if it's not great. If you have a bad year, with two big incidents when you're on a plane, you might not make your goal.
At four nines, it's about an hour. Good luck with that one. You're kind of screwed either way. Someone with intimitate knowledge of the system can probably handle an incident or three in time, but it's an enormous amount of work to get people up to speed for that and you can't have very many live training sessions because you don't have the outage budget. At the same time, if you page me and I'm at the cinema, I'm not going to fix it in time.
But, the biggest thing is working to make failure as graceful as possible. As much as possible, each component should be able to fail individually, without affecting the rest of the system. And, when there is an outage of a critical dependency, you need a way to push that outage status upstream and a method to manage load when it comes back up. (Those knobs should be stable and can certainly be documented for others).
Good thing I was drunk when I made it!
Stop deploying shit then. Also, I am very good at fixing systems even when drunk. Lots of practice.
Reminds me of my manager at a job a few years back. His standard was that any steps an on-call needs to take should be doable while they are drunk. You're waking people up in the middle of the night or they could be in who knows what sort of cognitive state. The runbook needs to be dead simple to follow. Mitigate now, fix it all up tomorrow when someone's awake and coherent.
If it is simple enough to do when drunk, why does it need to be done at all?
Typically for us, when something goes wrong that needs intervention, it's the first time that issue has ever happened and it needs to be debugged and understood. Then we'll put a mitigation in so it hopefully doesn't happen again.
In my last place, we had a runbook more like you describe, but I prefer prioritising bug fixes over features as it means I haven't had an out of hours call in a year!
Typically for us, when something goes wrong that needs intervention, it's the first time that issue has ever happened and it needs to be debugged and understood. Then we'll put a mitigation in so it hopefully doesn't happen again.
In my last place, we had a runbook more like you describe, but I prefer prioritising bug fixes over features as it means I haven't had an out of hours call in a year!
That was the other half of it though. Once it’s simple to mitigate, automate it!
To be fair, this was on a team that had A TON of operational debt. Like you’d get paged 30 times _a day_ when on call. Nobody wanted to move to weekly on call from a daily rotation because it would be a total nightmare, but daily meant the incentive to fix something was low because tomorrow it would be someone else’s problem. Of course, the solution was to get the alerts and underlying software under control, which we did and things got better.
Many of the non automated run books that remained were things like “look at these three graphs, if they’re doing x, escalate to team y”. We were somewhat hamstrung by political considerations that prevented us from simply directly paging those teams, or limitations of our alerting system that couldn’t link alerts (“if this other alarm is already going off, you don’t have to do anything because team y is already being paged for the root cause”).
To be fair, this was on a team that had A TON of operational debt. Like you’d get paged 30 times _a day_ when on call. Nobody wanted to move to weekly on call from a daily rotation because it would be a total nightmare, but daily meant the incentive to fix something was low because tomorrow it would be someone else’s problem. Of course, the solution was to get the alerts and underlying software under control, which we did and things got better.
Many of the non automated run books that remained were things like “look at these three graphs, if they’re doing x, escalate to team y”. We were somewhat hamstrung by political considerations that prevented us from simply directly paging those teams, or limitations of our alerting system that couldn’t link alerts (“if this other alarm is already going off, you don’t have to do anything because team y is already being paged for the root cause”).
If you can't fix your stuff while drunk you have never tried.
> you really want to encourage engineers to move around every 2-3 years
I have to disagree, I can say I honestly don't hit true full stride until ~1-2 yrs of working with a complex system. A big part of this people really don't like know how to make non-complex systems, and there's next to no incentive in it with the 12-18month new project/promotion cycle. (ie, long-termism is rarely the original writer's concern)
In my experience by the time i'm 2-3 years experience with a set of systems I'm many multiples more productive and informed than someone with ~0.5-1 yrs experience on it. And I'm able to tell product/support people exactly how the system behaves in its edges, and how it would hypothetically behave if modified for new features/logic.
So, plz managers consider allowing people to become true masters of a domain too.
I have to disagree, I can say I honestly don't hit true full stride until ~1-2 yrs of working with a complex system. A big part of this people really don't like know how to make non-complex systems, and there's next to no incentive in it with the 12-18month new project/promotion cycle. (ie, long-termism is rarely the original writer's concern)
In my experience by the time i'm 2-3 years experience with a set of systems I'm many multiples more productive and informed than someone with ~0.5-1 yrs experience on it. And I'm able to tell product/support people exactly how the system behaves in its edges, and how it would hypothetically behave if modified for new features/logic.
So, plz managers consider allowing people to become true masters of a domain too.
I find that at the 2-3 year mark too many people have come to know that I will be able to solve their problem, either directly or indirectly. That's when I usually leave, because my day becomes filled with constant "distraction." This distraction may be good for the employer, because problems get solved and things the employer values get done. But it's miserable for me. The part of this career I enjoy is getting that "in the zone" feeling when neck deep in code. If I'm getting yanked in different directions constantly I don't get what I value out of the job anymore.
This was basically my issue too, manager keeps asking why I get so little done some days. I have to explain it's because I'm effectively the productivity of 3-5 ppl on my team and so when you see them delivering fast, it's largely because they came to me with a hard issue, i broke i down for them and all they had to do was write the code.
I was allowed to work on the same problem space at a company for 5 years. I was an absolute master of the domain and it was incredible! I was so productive it was insane. I could plan and execute these totally wild projects that evolved the platform in positive ways. It was a lot of fun.
But... I could feel myself stagnating. I wasn't really learning anything. I left for another company and discovered that, yeah, outside of that area that I had been working on for 5 years I was almost completely incompetent. Didn't feel great. There's something to be said for switching areas every few years, it keeps your skills fresh.
But... I could feel myself stagnating. I wasn't really learning anything. I left for another company and discovered that, yeah, outside of that area that I had been working on for 5 years I was almost completely incompetent. Didn't feel great. There's something to be said for switching areas every few years, it keeps your skills fresh.
My story is so similar, but I’m still in that stagnation phase now. I went from never working on a production service to building, evolving, and ops-ing basically our entire back end over the last five years. I can answer nearly any question about it within a few seconds. And I’m so ready for something new, and for a break.
Leaving this kind of comfort won’t be easy, but I know that I need to find a way.
Leaving this kind of comfort won’t be easy, but I know that I need to find a way.
What's been most painful for me is that I was a few months away from a promotion to being a Principal engineer. I didn't feel like it was deserved since I only really worked on one (very important, very large, very visible) piece of the product. I was getting to the point of pretty extreme burnout and the last thing I wanted was a promotion!
I had thought that a switch to another company to work on new things would "fix" my burnout since it would be such a different problem space. New problems means new learning means more energy!
I couldn't have been more wrong. Leaving the comfort of something I knew to jump into the deep end of an entirely new stack just exacerbated the burnout. I've never wanted to change careers more than I have now.
So I say to you- take a break! I only had two weeks off between jobs which was just enough to start to feel relaxed. If possible, I'd suggest taking at least a month off.
I had thought that a switch to another company to work on new things would "fix" my burnout since it would be such a different problem space. New problems means new learning means more energy!
I couldn't have been more wrong. Leaving the comfort of something I knew to jump into the deep end of an entirely new stack just exacerbated the burnout. I've never wanted to change careers more than I have now.
So I say to you- take a break! I only had two weeks off between jobs which was just enough to start to feel relaxed. If possible, I'd suggest taking at least a month off.
Hey, thanks for the reality check. This is really helpful advice.
I agree. Some degree of moving around between companies is good for a career as it exposes people to different companies’ ways of doing things and helps someone determine what they like (or don’t) about different companies.
But there’s also something great about a company where people genuinely enjoy working for the long term and aren’t just using it as a springboard to the next opportunity.
Having a lot of employees with short time horizons leads to a lot of sacrificing long-term objectives in favor of short-term line items that sound good on a resume. That’s not to say that people can’t accomplish great things in 1-3 years, but when nobody’s time horizon extends beyond that, every goal gets altered to fit within that time frame.
Or worse, people start getting very good at starting ambitious projects, getting them to proof-of-concept phase, and then leaving because they can put it on their resume as an accomplishment. I’ve moved through some companies they were basically a graveyard of half-finished projects they nobody wanted to inherit because they all wanted to start their own thing rather than pick up the hard parts of someone else’s half finished project.
Of course, this requires the company to reward employees appropriately to make it worthwhile to stay. Many companies have figured this out, but many smaller corps underpay as long as possible and count on a steady turnover as part of their compensation strategy. Not great, IMO. Become a workplace that people want to stick with for a long time.
But there’s also something great about a company where people genuinely enjoy working for the long term and aren’t just using it as a springboard to the next opportunity.
Having a lot of employees with short time horizons leads to a lot of sacrificing long-term objectives in favor of short-term line items that sound good on a resume. That’s not to say that people can’t accomplish great things in 1-3 years, but when nobody’s time horizon extends beyond that, every goal gets altered to fit within that time frame.
Or worse, people start getting very good at starting ambitious projects, getting them to proof-of-concept phase, and then leaving because they can put it on their resume as an accomplishment. I’ve moved through some companies they were basically a graveyard of half-finished projects they nobody wanted to inherit because they all wanted to start their own thing rather than pick up the hard parts of someone else’s half finished project.
Of course, this requires the company to reward employees appropriately to make it worthwhile to stay. Many companies have figured this out, but many smaller corps underpay as long as possible and count on a steady turnover as part of their compensation strategy. Not great, IMO. Become a workplace that people want to stick with for a long time.
My argument is that as the tech side of the organization takes on more and more of the operation of the business, it also needs to be taking on more of the rewards.
For example if the expectation is that a engineering team designs, builds and maintains the machine twenty four hours a day that is responsible for the majority (if not all) of the firms profits then the majority of the rewards must flow there.
One the other hand, traditional methods for doing this have fallen by the wayside. With performance overhangs and other chicanery, what was once a lottery ticket is now a lottery ticket that someone will almost certainly try to steal from you once it becomes worth something. Instead of rewarding engineers who keep the lights on, we make them look for another job every 2 years to keep with the market.
People like to say tech is eating the world, tech is doing virtually all the killing but the hangers on seem to do the bulk of the eating.
For example if the expectation is that a engineering team designs, builds and maintains the machine twenty four hours a day that is responsible for the majority (if not all) of the firms profits then the majority of the rewards must flow there.
One the other hand, traditional methods for doing this have fallen by the wayside. With performance overhangs and other chicanery, what was once a lottery ticket is now a lottery ticket that someone will almost certainly try to steal from you once it becomes worth something. Instead of rewarding engineers who keep the lights on, we make them look for another job every 2 years to keep with the market.
People like to say tech is eating the world, tech is doing virtually all the killing but the hangers on seem to do the bulk of the eating.
Ah, capitalism! The company makes surplus profits that can be extracted by shareholders (and executives and employees, although these get labelled as expenses, not profits) if it is able to sell goods or services at a price point much higher than the cost of producing the service.
The cost of keeping the machine running is an operational expense. There is supply and demand in the labour market. If the team keeping the machine running generates $100m of profits for the company and costs $5m in compensation, but it is possible to hire a similar team at market rates to do a comparable job for $5m , then it doesn't make much sense to pay the team more than $5m. The excess value of $95 m generated by the team's activities can be captured as profit.
It's also important not to neglect the contributions of other roles. The machine is only able to produce revenue because it is matched with customers who are a good fit for the service the machine delivers and are willing to pay for it. If there are no customers then the machine and the engineering team that support it produce no revenue. So arguably the sales team that is able to identify potential customers and help them understand if renting access to the machine could help them also provide a large amount of value to the company.
If you take the same high performing engineering team that builds and operates a valuable service in one company, and drop them into a different business context where there are simply much fewer paying customers, or the sales team isn't able to deliver, then the value the engineering team are able to generate in the new situation may be zero or negative. Maybe most of the value generated by the service isn't due to the team who built and maintain it, but the surrounding business context. Leverage is a big thing. The highest paying roles are often in situations with more leverage (in huge orgs or huge projects), not because the work is necessary more skilled or harder than work in other smaller scale situations.
If some of the salespeople and engineers believe they are getting paid disproportionately little compared to the value they create, they could consider banding together and starting their own venture where they would be the owners, not the employees. Much easier said than done, and much easier to stomach if starting in a situation where they have the financial safety net and career connections so they'll be okay if the business fails.
The cost of keeping the machine running is an operational expense. There is supply and demand in the labour market. If the team keeping the machine running generates $100m of profits for the company and costs $5m in compensation, but it is possible to hire a similar team at market rates to do a comparable job for $5m , then it doesn't make much sense to pay the team more than $5m. The excess value of $95 m generated by the team's activities can be captured as profit.
It's also important not to neglect the contributions of other roles. The machine is only able to produce revenue because it is matched with customers who are a good fit for the service the machine delivers and are willing to pay for it. If there are no customers then the machine and the engineering team that support it produce no revenue. So arguably the sales team that is able to identify potential customers and help them understand if renting access to the machine could help them also provide a large amount of value to the company.
If you take the same high performing engineering team that builds and operates a valuable service in one company, and drop them into a different business context where there are simply much fewer paying customers, or the sales team isn't able to deliver, then the value the engineering team are able to generate in the new situation may be zero or negative. Maybe most of the value generated by the service isn't due to the team who built and maintain it, but the surrounding business context. Leverage is a big thing. The highest paying roles are often in situations with more leverage (in huge orgs or huge projects), not because the work is necessary more skilled or harder than work in other smaller scale situations.
If some of the salespeople and engineers believe they are getting paid disproportionately little compared to the value they create, they could consider banding together and starting their own venture where they would be the owners, not the employees. Much easier said than done, and much easier to stomach if starting in a situation where they have the financial safety net and career connections so they'll be okay if the business fails.
Hell no, you're not a monster for not wanting to be on call. You're normal. Being on call is fundamentally not that fun.
You're also totally normal if you respond to an incentive structure of offering senior level and above employees the perk of not having to carry a pager. Some companies use this to hire senior talent because, hey, it works.
As a side note: there's a shitload of negative solidarity in this thread. That's the attitude of: "If I must eat shit, everyone else must eat some too". I think this article (while a little flawed) makes a good pivot towards positive solidarity, and asking the question: "Is there a way to achieve this goal where everyone eats less shit?"
You're also totally normal if you respond to an incentive structure of offering senior level and above employees the perk of not having to carry a pager. Some companies use this to hire senior talent because, hey, it works.
As a side note: there's a shitload of negative solidarity in this thread. That's the attitude of: "If I must eat shit, everyone else must eat some too". I think this article (while a little flawed) makes a good pivot towards positive solidarity, and asking the question: "Is there a way to achieve this goal where everyone eats less shit?"
When I worked in IT we had a mantra: Don't be a superhero. This was at a massive company of 70k+ employees. Don't stay after hours to fix things. Don't take on more issues than you can handle. Don't agree to unforeseeable deadlines.
Backlogs represent a staffing failure. Customers are waiting 2 days for an email response? That's a staffing failure. Working harder to cover up a backlog leads to staffing issues never being addressed, and only passes that burden on to the other IT staff.
Backlogs represent a staffing failure. Customers are waiting 2 days for an email response? That's a staffing failure. Working harder to cover up a backlog leads to staffing issues never being addressed, and only passes that burden on to the other IT staff.
The best way to keep people from eating shit on-call is to make technology that is failure-resistant. Most software today is not designed to resist failure, and most operations are not designed to resist failure. But it could get better.
For example: a trend in recent years was static site generators. You can still find the internet littered with websites that return only errors due to SQL servers breaking long ago. But those static sites, as long as they don't depend on some other precarious tech, will keep working as long as someone's paying the bills. No security patches needed, no logs filling up disks. The servers could still go down, but it won't be your code's fault, so you shouldn't get a call.
The majority of failure risk today comes down to security and mutable state. Both of them are easier when the tech is read-only. For security, you can reduce your attack surface until no reasonable attack would work. For mutable state, you have a couple options. First, all changes to both code and data should be versioned and immutable with a working rollback mechanism. Second, never allow a change without verification, and rollback should happen immediately if verification fails. Third, do not depend on technology whose state can or does mutate regularly. Fourth, do not depend on technology that will stop operating once resource limits are exceeded. And fifth, never build something if you can buy it. (The latter has to do with investment in robustness. If you have to build an entire car, you will never have enough time or money or expertise to design a better brake system/suspension system/muffler/etc as someone who specializes in manufacturing just that one part)
Another positive solidarity take would be for everyone to stop taking jobs where 24/7 is a requirement. How do you do this? Before signing a contract, require that they put in language that says you will not be required to work outside of normal business hours. If you're a developer, have it say explicitly that you will only be required to maintain application code and not maintain servers. If they balk, don't take the job. Have a lawyer look it over. This doesn't stop the company from firing you in at-will employment states if you pull out the contract one day, so it's better to just walk if they challenge putting it in writing. Some industries just do not expect their users to be 24/7.
And a final note: if you are a developer, please don't just wing your app's design in terms of how it will work in production. Don't let the ops person take care of running it for you, and also don't try to figure it out by yourself. Put your brain together with the brain of somebody who specializes in system architecture and operations - at the start of your project, not the end. Design your app along with this other brain, to work in tandem with a very robust system architecture. And then as you build your app, ask the other brain to review it periodically and give you feedback. In terms of car analogies, this is like asking a mechanic for feedback on the parts you are designing. They've seen what fails when and why, what's painful to maintain and what isn't, what's expensive and what's not. I promise you this will result in fewer calls.
For example: a trend in recent years was static site generators. You can still find the internet littered with websites that return only errors due to SQL servers breaking long ago. But those static sites, as long as they don't depend on some other precarious tech, will keep working as long as someone's paying the bills. No security patches needed, no logs filling up disks. The servers could still go down, but it won't be your code's fault, so you shouldn't get a call.
The majority of failure risk today comes down to security and mutable state. Both of them are easier when the tech is read-only. For security, you can reduce your attack surface until no reasonable attack would work. For mutable state, you have a couple options. First, all changes to both code and data should be versioned and immutable with a working rollback mechanism. Second, never allow a change without verification, and rollback should happen immediately if verification fails. Third, do not depend on technology whose state can or does mutate regularly. Fourth, do not depend on technology that will stop operating once resource limits are exceeded. And fifth, never build something if you can buy it. (The latter has to do with investment in robustness. If you have to build an entire car, you will never have enough time or money or expertise to design a better brake system/suspension system/muffler/etc as someone who specializes in manufacturing just that one part)
Another positive solidarity take would be for everyone to stop taking jobs where 24/7 is a requirement. How do you do this? Before signing a contract, require that they put in language that says you will not be required to work outside of normal business hours. If you're a developer, have it say explicitly that you will only be required to maintain application code and not maintain servers. If they balk, don't take the job. Have a lawyer look it over. This doesn't stop the company from firing you in at-will employment states if you pull out the contract one day, so it's better to just walk if they challenge putting it in writing. Some industries just do not expect their users to be 24/7.
And a final note: if you are a developer, please don't just wing your app's design in terms of how it will work in production. Don't let the ops person take care of running it for you, and also don't try to figure it out by yourself. Put your brain together with the brain of somebody who specializes in system architecture and operations - at the start of your project, not the end. Design your app along with this other brain, to work in tandem with a very robust system architecture. And then as you build your app, ask the other brain to review it periodically and give you feedback. In terms of car analogies, this is like asking a mechanic for feedback on the parts you are designing. They've seen what fails when and why, what's painful to maintain and what isn't, what's expensive and what's not. I promise you this will result in fewer calls.
Author lists off group of people who should not get pager duty and I think it is unfair to the rest of the team that individuals who fall into the groups outlined are able to escape the duties and responsibilities of the job.
If the job posting or interviewing was upfront about the role will have on-call then the interviewee should make the decision to not apply or accept the offer if they're unable to fulfill the responsibilities.
If the job posting or interviewing was upfront about the role will have on-call then the interviewee should make the decision to not apply or accept the offer if they're unable to fulfill the responsibilities.
You think it's unfair for people who have young children to not be expected to also be on call 24/7? The author even specifies that they're talking about kids too young to sleep through the night. It's unfair for people with sleep disorders to not have to worry about getting called in at 2 am? In the anxiety case, it even sounds like the employee really wanted to be on call, but maybe needs to see a specialist to handle their reaction to stress.
If people don't deserve a little slack due to personal circumstances, what happens when you find yourself in a situation where you can't pull your weight? I feel like people who think that not being able to do 110% of your job at all times makes you unqualified are in for a rude awakening when they find themselves in a similar situation.
If people don't deserve a little slack due to personal circumstances, what happens when you find yourself in a situation where you can't pull your weight? I feel like people who think that not being able to do 110% of your job at all times makes you unqualified are in for a rude awakening when they find themselves in a similar situation.
It's unfair for people who have young children to not be expected to also be on call 24/7 and be rewarded in the same way as people who don't, and are.
There is no reward for OnCall work...except for a lack of respect. The OnCall people have always been viewed as less. I'm not doing anything on call unless our department head is on the bridge the entire time. OnCall is a pain that is inflicted during working hours.
> You think it's unfair for people who have young children to not be expected to also be on call 24/7?
That's dicey and depends on if you consider folks having children as them doing something they wanted for themselves, vs them doing society a favor.
If the former, it's like reducing work responsibilities of someone because they're also taking piano lessons on the side and its cutting in their free time. It's their choice and they are doing it for themselves.
If the later, then yeah, you'd have to cut them a break, in the same way we often do for folks with jury duty.
But having young children isn't, like, a health condition someone's born with.
The sleep disorder one is a bit trickier, since no one willing has sleep disorders. At the same time, it's not really your team mate's problems. I have some health issues that make me less productive at work. I don't get the roles, promotions, and compensation of the folks who can do more than I can. It sucks, but at the end of the day, companies hire you to do something. It's not a social service.
That's dicey and depends on if you consider folks having children as them doing something they wanted for themselves, vs them doing society a favor.
If the former, it's like reducing work responsibilities of someone because they're also taking piano lessons on the side and its cutting in their free time. It's their choice and they are doing it for themselves.
If the later, then yeah, you'd have to cut them a break, in the same way we often do for folks with jury duty.
But having young children isn't, like, a health condition someone's born with.
The sleep disorder one is a bit trickier, since no one willing has sleep disorders. At the same time, it's not really your team mate's problems. I have some health issues that make me less productive at work. I don't get the roles, promotions, and compensation of the folks who can do more than I can. It sucks, but at the end of the day, companies hire you to do something. It's not a social service.
Many programming jobs require on-call and overtime when they shouldn’t. I wouldn’t be able to have a job if that requirement was truly valid. It’s usually not.
I can’t do on-call for medical reasons, nor can I work overtime. Neither of these are considered to be an essential job function for programmers, regardless of job descriptions, under ADA. A company must justify that job requirement. They can’t.
By the way, the “fairness” reason is explicitly excluded by law. It’s the entire reason ADA exists.
I can’t do on-call for medical reasons, nor can I work overtime. Neither of these are considered to be an essential job function for programmers, regardless of job descriptions, under ADA. A company must justify that job requirement. They can’t.
By the way, the “fairness” reason is explicitly excluded by law. It’s the entire reason ADA exists.
My understanding is that they must offer a reasonable accommodation. For on call, I imagine that takes usually just takes the form of assigning some different responsibilities. Doesn’t seem too burdensome.
I worked with someone who had a sleep disorder. She would do twice as many on call shifts but only from like 8am to 8pm during the day. Someone else would take the night shift component, although that usually rotated too.
It wasn’t a bad trade because most of the on call work happened during the day. The night shifter only rarely got woken up to deal with something, so on the whole they dealt with less on call.
I worked with someone who had a sleep disorder. She would do twice as many on call shifts but only from like 8am to 8pm during the day. Someone else would take the night shift component, although that usually rotated too.
It wasn’t a bad trade because most of the on call work happened during the day. The night shifter only rarely got woken up to deal with something, so on the whole they dealt with less on call.
[deleted]
That attitude leads directly to ageism, sexism and other discriminatory practices, and has second order effects on society which are contrary to the long term interests of capitalism - a stable consumer base with disposable income.
Fairness isn't a straightforward concept, but your conception is more naive and - I'd guess - self-serving than most. How do you feel about progressive income taxes? Under a simplistic model of fairness, flat rates seem fair, but in reality the marginal ability to pay increases, and perceived pain of paying decreases, with increased income and wealth.
Now expand that concept from the monetary domain to the time domain, and consider how fair it is for people who are particularly time poor - for non-selfish reasons - to need to spend more of their time on the job. Affordances can be made for inclusiveness and they can be fair.
Fairness isn't a straightforward concept, but your conception is more naive and - I'd guess - self-serving than most. How do you feel about progressive income taxes? Under a simplistic model of fairness, flat rates seem fair, but in reality the marginal ability to pay increases, and perceived pain of paying decreases, with increased income and wealth.
Now expand that concept from the monetary domain to the time domain, and consider how fair it is for people who are particularly time poor - for non-selfish reasons - to need to spend more of their time on the job. Affordances can be made for inclusiveness and they can be fair.
Fairness, as used by the parent, means “if someone else can do something, I should be able to too.”
The other person’s situation is never a factor. They see someone getting things for free and they think they should have them for free too.
The other person’s situation is never a factor. They see someone getting things for free and they think they should have them for free too.
Do you feel the same way about maternity and paternity leave?
I used to not mind oncall as much, before managerialism and corporate attitudes took over tech. It was fun (I’d also stressful) to feel in the trenches so to speak, while being treated as someone mildly crazy for taking on the responsibility but also not held to the same silly and stupid bureaucratic attitudes that afflicted the rest of the company.
Now I’m supposed to talk to businessspeak talk and walk the promo game walk while also supposed to answer the phone at one am and stay up all night saving the business from melting down.
Yes I know I’m ranting. The industry has turned into a miserable corporate grind like finance or something, only with the added obligation of someone has to mop up to slop in the middle of the night and that someone is me.
Now I’m supposed to talk to businessspeak talk and walk the promo game walk while also supposed to answer the phone at one am and stay up all night saving the business from melting down.
Yes I know I’m ranting. The industry has turned into a miserable corporate grind like finance or something, only with the added obligation of someone has to mop up to slop in the middle of the night and that someone is me.
[deleted]
> It is engineering’s job to own their software
I'll accept this responsibility so long as it comes with the corresponding authority.
Now tell me, do I really own the software?
I'll accept this responsibility so long as it comes with the corresponding authority.
Now tell me, do I really own the software?
Ahh the ownership argument, dashed upon the rocks.
I have denied your request for a new feature because I have ownership over the project and decided we should work on X. Oh I don’t actually have any power but you think I’m an idiot who can’t see that? Cool
Of course, but you don't own the business, so the software still needs to meet the businesses goals. This isn't the gotcha you seem to think it is?
I was on call for 5 years at an old job (non IT). Getting paged and yanked out of bed at all hours messes with you in so many ways. Being sleep deprived you turn to high sugar drinks and foods to keep you awake. The sleep deprivation and bad diets causes you to put on weight. Going to bed with the idea you could get paged does terrible things for your anxiety. On top of that all the sleep deprivation changes your personality.
You become highly irritable and it really brings down your mood. Eventually I couldn't deal with it after 5 years and I pretty much walked off the job. I didn't realize the toll it was taking on me and what being on call was doing to me until I was no longer on call.
You become highly irritable and it really brings down your mood. Eventually I couldn't deal with it after 5 years and I pretty much walked off the job. I didn't realize the toll it was taking on me and what being on call was doing to me until I was no longer on call.
I see a lot of commenters saying it is part of the job and to be expected... that's fine, but in my own career what I observed was that the reason an on call rotation was even necessary was that the companies knew things would fail regularly, because they refused to invest in good practices up front. If a company wants to do everything it can to not turn on call time into a shit show and they want to fairly compensate for a person being available nights and weekends, then cool- we can all be team players. But, too often they want to squeeze you into doing the work of three people if they can get away with it.
I think the challenge that organizations run into with on call is that the costs of it are hidden and delayed - reduced productivity and lower retention. I think that because of this, organizations do not invest in keeping their systems in order.
The question is, what do you do about it? At some point things do have to escalate to an engineer, and I don't think it's practical to have a 24/7 engineering team staffed for every service (although having 24/7 non-eng support is very reasonable).
The best solution I've seen is having error budgets and on call compensation commensurate with required response times. Error budgets since they balance maintaining and expanding the service, compensation to respect people's time.
What have you seen that works?
The question is, what do you do about it? At some point things do have to escalate to an engineer, and I don't think it's practical to have a 24/7 engineering team staffed for every service (although having 24/7 non-eng support is very reasonable).
The best solution I've seen is having error budgets and on call compensation commensurate with required response times. Error budgets since they balance maintaining and expanding the service, compensation to respect people's time.
What have you seen that works?
If your business requires 24x7 coverage, you should pay for 24x7 coverage and not expect the same people working 8x5 are the ones who will do it "because it has to be done".
I once worked for a tech company that said they had follow the sun rotation and i was dumb enough to take the job only to find out that the other countries had strict labor laws and follow the sun didn't mean the US folks would have their nights and weekends, but that the US folks were doing off hour support for the rest of the world because we don't have labor laws that protect our workers rights. I literally had to keep a seat warm from 8-5 my time to do the bulk of my work from 5-9pm and on weekends to "work around the customers" and they failed miserably at making tech work... (fin tech.. where they through money and soul sucking work at every problem)
I once worked for a tech company that said they had follow the sun rotation and i was dumb enough to take the job only to find out that the other countries had strict labor laws and follow the sun didn't mean the US folks would have their nights and weekends, but that the US folks were doing off hour support for the rest of the world because we don't have labor laws that protect our workers rights. I literally had to keep a seat warm from 8-5 my time to do the bulk of my work from 5-9pm and on weekends to "work around the customers" and they failed miserably at making tech work... (fin tech.. where they through money and soul sucking work at every problem)
Shift work has been a thing for decades. I would imagine hiring some percentage of your devs for the night shift wouldn’t be that hard as many people enjoy that lifestyle. Then you already cover the majority of the 24 hours with normal work
Additionally there are a lot of devs who are from East Asia but started working in the US after undergrad or grad school. I'm sure there are many of them who would love to return to their home countries and work during the day time (for example, Vietnam timezone is currently a simple 12 hour difference from NYC tz). Even with a salary reduction they'd still come out ahead through lower taxes & the lower cost of living.
Additionally you could still hire devs from western backgrounds. While studying abroad my cookies have led to heavy advertising from turing.com, which seems to hire overseas US devs to work remote for US companies. There's plenty of American devs in Chiang Mai working on their stealth phase startup.
edit: It seems turing.com is wasting their ad budget, because they have a 4 hour overlap with US TZ requirement. Not sure why they're advertising to IP addresses coming from Asia.
Additionally you could still hire devs from western backgrounds. While studying abroad my cookies have led to heavy advertising from turing.com, which seems to hire overseas US devs to work remote for US companies. There's plenty of American devs in Chiang Mai working on their stealth phase startup.
edit: It seems turing.com is wasting their ad budget, because they have a 4 hour overlap with US TZ requirement. Not sure why they're advertising to IP addresses coming from Asia.
idk, here are some ideas for on call must haves based on other talk about on call things:
- normal rotation has 5-8 people
- on call lasts for one week
- some institutional trigger (like an SLO) exists for changing team priorities when pages happen too often
- the culture around on call is empowering, not heroic/dysfunctional
- people can still carry on with normal life while on call (just keep your laptop in a backpack and don't get drunk)
- option for next day off after picking up a page (no questions asked)
- frequent pages (more than a couple per quarter) trigger discussions about how to improve code quality or fix bug hot spots
- during on call shift you can get a break from normal sprint work (do rewarding things that add value for the team)
- there's no fear about being on call since problems are rare and everyone knows how the software systems work
- if you're stuck while debugging on call there are more senior people you can page too to get help
- work on app monitoring/debugging a little bit of the year to make debugging and triage easier when things do come up
- ops team catches the page before your team to provide context and repro
- reverting or deploying fixes isn't stressful because the deployment is stable and fully automated
> In a healthy engineering organization, there are no gaps in coverage. Every critical component is owned by a TEAM, not a person.
I just joined a large organisation that practices this, and it's driving me nuts. Because no individual is taking ownership, nothing happens. There's no accountability, no drive to improve anything. We ask questions on Slack about "can someone deal with this problem, please?" and nothing happens. Rather than an alarm going off at 2am and the owner dealing with it (again), the alarm goes off and nothing happens, no-one fixes it. Stuff stays broken until it gets a ticket and assigned to a sprint (and even then, people self-assign to tickets, and there's no consequences to not finishing all the tickets on a sprint, so it's still relying on someone to be vaguely interested enough to pick it up and deal with it).
I moved from being tech co-founder of a startup (where everything is my responsibility and it's all on me to fix it) to this. It's doing my head in. I think I actually prefer being "on-call" for code I built and have complete control over.
I just joined a large organisation that practices this, and it's driving me nuts. Because no individual is taking ownership, nothing happens. There's no accountability, no drive to improve anything. We ask questions on Slack about "can someone deal with this problem, please?" and nothing happens. Rather than an alarm going off at 2am and the owner dealing with it (again), the alarm goes off and nothing happens, no-one fixes it. Stuff stays broken until it gets a ticket and assigned to a sprint (and even then, people self-assign to tickets, and there's no consequences to not finishing all the tickets on a sprint, so it's still relying on someone to be vaguely interested enough to pick it up and deal with it).
I moved from being tech co-founder of a startup (where everything is my responsibility and it's all on me to fix it) to this. It's doing my head in. I think I actually prefer being "on-call" for code I built and have complete control over.
> I moved from being tech co-founder of a startup (where everything is my responsibility and it's all on me to fix it) to this
likely because the startup you co-founded would benefit hugely if you worked to fix stuff, which means you personally would benefit since you're a co-founder. In an employee relationship, this benefit doesn't flow on to the employee, and thus everyone is incentivized to do the least possible that they can get away with.
It doesn't gel with your personality, and that is why it's doing your head in.
likely because the startup you co-founded would benefit hugely if you worked to fix stuff, which means you personally would benefit since you're a co-founder. In an employee relationship, this benefit doesn't flow on to the employee, and thus everyone is incentivized to do the least possible that they can get away with.
It doesn't gel with your personality, and that is why it's doing your head in.
Your team dynamic feels broken, and your manager isn't managing well. On-call is a shared responsibility, and so is ensuring that on-call isn't awful. That's one of the main arguments in this blog post.
There's lots of ways to fix this, but one of the more popular approaches is to make the on-call responsible for fixing this kind of stuff during normal working hours, and for that person to not have sprint-work during that period of time. The goal is to ensure the next person's on-call is better than yours, which leads to a continuous improvement mentality, with an end-goal of never being paged after-hours, and for all pages to be actionable.
There's lots of ways to fix this, but one of the more popular approaches is to make the on-call responsible for fixing this kind of stuff during normal working hours, and for that person to not have sprint-work during that period of time. The goal is to ensure the next person's on-call is better than yours, which leads to a continuous improvement mentality, with an end-goal of never being paged after-hours, and for all pages to be actionable.
Try taking some individual ownership and see what sort of recognition or reward you get.
I've been in organizations like the one you describe and the lack of initiative is often a side effect of hostility to autonomy.
I've been in organizations like the one you describe and the lack of initiative is often a side effect of hostility to autonomy.
Thanks, I'll try that and see what happens
There does seem to be a culture of "keeping your head down"
There does seem to be a culture of "keeping your head down"
I ask every team I interview with whether they have any sort of on call. A “yes” answer terminates the interview, unless quickly followed by “but we can make an exception”. Call me a monster if you wish, but my time is mine and my family’s.
There's a salutary effect on developer conscientiousness to put them on call for problems with their own code on production. If a missing nil check means that you get dragged out of bed at peak REM sleep, you check for missing nils harder. You test more. You design your whole project around the requirement of getting uninterrupted sleep.
For years I was in this position for a trading company that had pre-market downloads at 4am my time. Bad data and my own bad code woke me up a lot ... which contributed a lot to making that code wake-me-up proof for further years.
Of course you should get paid for that time, and there's a diminishing return to such character building exercises. But eating your own dog food is worthwhile particularly in the middle of the night.
For years I was in this position for a trading company that had pre-market downloads at 4am my time. Bad data and my own bad code woke me up a lot ... which contributed a lot to making that code wake-me-up proof for further years.
Of course you should get paid for that time, and there's a diminishing return to such character building exercises. But eating your own dog food is worthwhile particularly in the middle of the night.
This assumes your company is okay with spending time to write reliable systems.
[deleted]
> My best idea is ‘pay people more,’ but that doesn’t sound so compelling as a pitch.
I would not accept a position that paid for on-call, even if they paid more. That just legitimizes encroaching on personal time and likely will make it happen more often. Pay me instead in equivalent PTO (or greater) so that there is a stronger forcing function to reduce pages.
I would not accept a position that paid for on-call, even if they paid more. That just legitimizes encroaching on personal time and likely will make it happen more often. Pay me instead in equivalent PTO (or greater) so that there is a stronger forcing function to reduce pages.
At google you could choose either 1/3-2/3 of your hourly pay (base) for each hour of being oncall outside of 9-5 or equivalent number of hours of additional pto. Best oncall comp structure I’ve seen ever since. Somehow people like to parrot only employee unfriendly practices from big corps…
Really liked the article. One thing I'd add (and that I wrote about before here: https://jmmv.dev/2021/07/principal-engineers-oncall.html) is that high-level engineers should be on-call. These engineers (along with management) are the people that can most easily make a difference in making the rotation better: and not because of their technical skills, but because if they "suffer through the pain", they may be able to set priorities to fix things.
As some other people have alluded here, though, in some organizations engineers seem to "escape" on-call rotations the more senior they get, which is unfortunate...
As some other people have alluded here, though, in some organizations engineers seem to "escape" on-call rotations the more senior they get, which is unfortunate...
Get the VPs and Directors on call and see how quickly we’ll change priorities lol
In many orgs, the pager escalation path is to VPs and Directors.
At some point, there has to be someone to push the right button at 3AM. That can be done by shift work, it can also be handled by on call. In general, a job with on-call requirements should disclose that up front, and compensation should be better than the same position without the on-call requirement. For small teams, on-call may be the only way to do it because there isn't going to be a night shift.
This problem highlights one advantage of having a distributed remote team. In principle, there may not need be a midnight shift.
I and another person are taking over an old batch processing system at work. The system itself is fairly stable, but interacts with some other systems which keep falling over or changing things without adequate notification. Currently there are (mostly false) alarms almost every day, usually for stupid reasons like the network falling over for a few minutes or disk use going above a fairly arbitrary level. So when the subject of weekend on-call came up we just went hard "no", and we've been discussing solutions to that ever since (adjusting cron scheduling, making alarms more forgiving, adjusting customer expectations, improving automation, etc).
Importantly, we can do this because we live somewhere with employee protection, and our contracts do not mention being on call or working weekends. If you don't have that you're pretty much SOL.
Importantly, we can do this because we live somewhere with employee protection, and our contracts do not mention being on call or working weekends. If you don't have that you're pretty much SOL.
You probably know this already, but getting rid of those false alarms is absolutely critical. Instead of alerting on disk use, alert on user-facing metrics when possible (unless it’s disk space going below a certain threshold, since that’s one of the worst things that can happen).
Problem is, the only user facing metric is basically "Did you get the last week's updates?", which currently is shipped on Friday and the user usually notices on Monday.
I can’t even be on-call if I wanted to. ADA requires a company justify any essential job functions. Justifying a programmer needing to be on call is difficult because it is a secondary role.
Programmers aren’t hired to do on-call. The company just decides they can skip hiring anyone for those roles by reusing “existing assets”. Same with single sources of knowledge. They don’t invest in having sufficient capacity.
Basically, companies hire people who can do multiple jobs as cheaply as possible and with has as little support as possible. Which is not sustainable.
ADA can really highlight dysfunctional systems.
For anyone saying a small company can’t afford this, ADA doesn’t apply to companies under 15 employers or that can demonstrate undo-hardship given their resources. (By the way, trying to hire employees as contractors does not work.)
Programmers aren’t hired to do on-call. The company just decides they can skip hiring anyone for those roles by reusing “existing assets”. Same with single sources of knowledge. They don’t invest in having sufficient capacity.
Basically, companies hire people who can do multiple jobs as cheaply as possible and with has as little support as possible. Which is not sustainable.
ADA can really highlight dysfunctional systems.
For anyone saying a small company can’t afford this, ADA doesn’t apply to companies under 15 employers or that can demonstrate undo-hardship given their resources. (By the way, trying to hire employees as contractors does not work.)
Troubleshooting and mitigating an outage always take priority over anything else. In that sense on call is my job, and I could wake my entire team or ask them to drop everything else if I really needed to.
But if I understand the design and code well enough to do that, why wouldn’t I also work on features and fixes in between outages? Why divide the roles when you have to choose among the same small group of people for either?
But if I understand the design and code well enough to do that, why wouldn’t I also work on features and fixes in between outages? Why divide the roles when you have to choose among the same small group of people for either?
Extremely important article. I have been a part of some hell rotations and it is not sustainable at all.
> First, I’d like to say that pager duty isn’t something we should treat like chronic pain or diabetes, where you just constantly manage symptoms and tend to flare-ups day and night. Being paged out of hours is as serious as a fucking heart attack. It should be RARE and taken SERIOUSLY. Resources should be mustered, product cycles should be reassigned, until the problem is fixed.
Having experienced both long-term chronic pain and long-term chronic on-call for some grumpy services, I find this comparison ... off-putting. On-call does not feel like it's even remotely close to as difficult.
Having experienced both long-term chronic pain and long-term chronic on-call for some grumpy services, I find this comparison ... off-putting. On-call does not feel like it's even remotely close to as difficult.
Having a young kid is an edge-case while on-call is the norm? Software Engineering has jumped the shark and is now commodotized, exploited labor. Laptops and smartphones don't turn engineers into ER physicians.
Why do so many websites and services need to run 24/7?
Put another way, if a non-internet business wanted to operate 24/7, they would have significant expense and logistical problems (to the extent that very few do so).
If you’re a PaaS/IaaS provider, ok, you’re going to have to run 24x7. If you’re anything else, maybe you should have a serious discussion about what it’s worth to you to operate around the clock.
Of course it’s weird having web sites that are unavailable 24x7 but it’s not unheard of - some government web sites like clerks of court, inspections and GIS already are “business hours only”.
Put another way, if a non-internet business wanted to operate 24/7, they would have significant expense and logistical problems (to the extent that very few do so).
If you’re a PaaS/IaaS provider, ok, you’re going to have to run 24x7. If you’re anything else, maybe you should have a serious discussion about what it’s worth to you to operate around the clock.
Of course it’s weird having web sites that are unavailable 24x7 but it’s not unheard of - some government web sites like clerks of court, inspections and GIS already are “business hours only”.
I think the situation is pretty clear. You either have a significant stake of your empoloyer's success upside or not. This you see by the equity amount given to you.
Employers have zero loyality to you as an employee and care the same amount about your career progression so in the end if you're not in for five-six (seven?) figure equity stake you're effectively a contractor / freelancer.
By being an employee you're taking unlimited liability for long tail risk of your employer that otherwise would be priced into the freelance or service contract at a very high amount.
Employers have zero loyality to you as an employee and care the same amount about your career progression so in the end if you're not in for five-six (seven?) figure equity stake you're effectively a contractor / freelancer.
By being an employee you're taking unlimited liability for long tail risk of your employer that otherwise would be priced into the freelance or service contract at a very high amount.
So I've seen personally been involved in this at both Google and Facebook. They have very different approaches. One works reasonably well. One... doesn't.
At Google, when you develop a service, you as a software team are responsible for maintaining it. This includes being oncall. But, depending on how critical the service is, you'll get bonus pay for this oncall time. This can amount to >$10k a year. And your oncall may not be that noisy. That is something and (IMHO) significant. Generally you'll be oncall for a week but this can vary.
For sufficiently high profile services, oncall may end up being owned by an SRE team. That's not really something that can be thrown over the fence by SWE teams. SRE teams have to accept that ownership. To do this, it requires meeting standards like having an oncall runbook, a sufficiently long history (~6 months), adequate metrics and so on. At that point, SWEs will still be second level support for something SREs may need help resolving. You'll still get paid for this.
SREs don't generally have week long oncalls. For the highest profile services, SRE support is global, meaning you'll have team members in 3 time zones such that whoever is oncall is in their normal working hours. So you might have an 8 hour oncall every week or something.
At Facebook, generally as a software team you'll be responsible for that service forever. Some key services may be fully or partially supported by PEs (Production Engineers). PEs are less common than Google SREs.
You don't get oncall pay at Facebook. It's simply expected. Depending on your management, you may be expected to do that oncall and everything else you're supposed to be doing.
So here's why Google's system is better than Facebook's:
1. Oncalls, in my experience, tend to be a lot less noisy at Google. Services tend to be much more mature and stable at Google than Facebook;
2. Teams are generally larger at Google. This means that not everyone needs to be oncall. This is good. There is an optimum size for oncalls. IMHO it is about 8-12. Fewer than 8 and people are oncall too much. More than 12 and you lose skills by not using them often enough.
3. The pay is really important to (2) because it means that those who are oncall at least get compensated for it. At a minimum this sends the message that oncall is important and rewarded. It also means those oncall are less likely to resent those not oncall, which might well be the case if you're not compensated for the extra work;
4. There are a ton of orphan projects at Facebook (IME). By this I mean some product team (in particular) will develop something, ship it and then... forget about it. Onwership of source code trees and projects tends to be fairly strictly enforced at Google. These orphan projects tend to create a number of support issues and tasks that are a drain on oncall as there is no one to pass those tasks on to.
5. Culturally, Google seems to acknowledge and respect oncall work load more than Facebook does. For example, tasks at Facebook have SLAs and I saw plenty of games played to avoid dealing with tasks. Examples: changing priority to increase or remove the SLA, silently closing tasks, passing tasks back months later to the reporter asking "is this still an issue?", etc.
So my point is that not wanting to be oncall is understandable but for many engineers, it's going to be part of your job. If so, you need to make sure your organization does it well. There's a world of difference between an oncall you don't get paid for that's 40 hours of work vs one where you get 2 tasks a week and you're getting paid for possibly having to answer an alert.
Hating oncall is indicative of a bad engineering culture IME. Things like prioritizing shipping above all else, not rewarding fixing things, unreasonable management driven deadlines and so on. Oncall is the canary in the coal mine for all of that.
At Google, when you develop a service, you as a software team are responsible for maintaining it. This includes being oncall. But, depending on how critical the service is, you'll get bonus pay for this oncall time. This can amount to >$10k a year. And your oncall may not be that noisy. That is something and (IMHO) significant. Generally you'll be oncall for a week but this can vary.
For sufficiently high profile services, oncall may end up being owned by an SRE team. That's not really something that can be thrown over the fence by SWE teams. SRE teams have to accept that ownership. To do this, it requires meeting standards like having an oncall runbook, a sufficiently long history (~6 months), adequate metrics and so on. At that point, SWEs will still be second level support for something SREs may need help resolving. You'll still get paid for this.
SREs don't generally have week long oncalls. For the highest profile services, SRE support is global, meaning you'll have team members in 3 time zones such that whoever is oncall is in their normal working hours. So you might have an 8 hour oncall every week or something.
At Facebook, generally as a software team you'll be responsible for that service forever. Some key services may be fully or partially supported by PEs (Production Engineers). PEs are less common than Google SREs.
You don't get oncall pay at Facebook. It's simply expected. Depending on your management, you may be expected to do that oncall and everything else you're supposed to be doing.
So here's why Google's system is better than Facebook's:
1. Oncalls, in my experience, tend to be a lot less noisy at Google. Services tend to be much more mature and stable at Google than Facebook;
2. Teams are generally larger at Google. This means that not everyone needs to be oncall. This is good. There is an optimum size for oncalls. IMHO it is about 8-12. Fewer than 8 and people are oncall too much. More than 12 and you lose skills by not using them often enough.
3. The pay is really important to (2) because it means that those who are oncall at least get compensated for it. At a minimum this sends the message that oncall is important and rewarded. It also means those oncall are less likely to resent those not oncall, which might well be the case if you're not compensated for the extra work;
4. There are a ton of orphan projects at Facebook (IME). By this I mean some product team (in particular) will develop something, ship it and then... forget about it. Onwership of source code trees and projects tends to be fairly strictly enforced at Google. These orphan projects tend to create a number of support issues and tasks that are a drain on oncall as there is no one to pass those tasks on to.
5. Culturally, Google seems to acknowledge and respect oncall work load more than Facebook does. For example, tasks at Facebook have SLAs and I saw plenty of games played to avoid dealing with tasks. Examples: changing priority to increase or remove the SLA, silently closing tasks, passing tasks back months later to the reporter asking "is this still an issue?", etc.
So my point is that not wanting to be oncall is understandable but for many engineers, it's going to be part of your job. If so, you need to make sure your organization does it well. There's a world of difference between an oncall you don't get paid for that's 40 hours of work vs one where you get 2 tasks a week and you're getting paid for possibly having to answer an alert.
Hating oncall is indicative of a bad engineering culture IME. Things like prioritizing shipping above all else, not rewarding fixing things, unreasonable management driven deadlines and so on. Oncall is the canary in the coal mine for all of that.
Do you have kids? hobbies? friends? Do you like to go fishing on weekends? Do you like to go out on a date on weeknights? Do you like going to visit family on Christmas and Thanksgiving and leaving your laptop at home? It's great explaining to your 11 year old that you can't go to their recital because your on call isn't it?
It boggles my mind that you suggest people lose skills by not using them often enough... these on call rotations aren't skills.
What's weird, is that we default to look at the goodness or badness of something purely on "skills" and "money".
Yes.. yes.. we've all read the SRE book and understand Googles approach to flexibility - but... it doesn't have to be that way.
Ask yourself.. what's next? What are you looking to do after 25 years of being on call? Will you still enjoy it?
I'll never have those nights and weekends back that I spent being on call for products and services that don't exist anymore. Sure, I made some money - but it didn't have to be at such a great cost.
It boggles my mind that you suggest people lose skills by not using them often enough... these on call rotations aren't skills.
What's weird, is that we default to look at the goodness or badness of something purely on "skills" and "money".
Yes.. yes.. we've all read the SRE book and understand Googles approach to flexibility - but... it doesn't have to be that way.
Ask yourself.. what's next? What are you looking to do after 25 years of being on call? Will you still enjoy it?
I'll never have those nights and weekends back that I spent being on call for products and services that don't exist anymore. Sure, I made some money - but it didn't have to be at such a great cost.
At least at Google, oncall shifts aren't set in stone. If you need to swap with a coworker (even for a few hours), you can. Devs should not be holding 15-minute (much less 5-minute) pagers; if your system is really that critical, you should be working to onboard an SRE team.
> these on call rotations aren't skills.
Troubleshooting live production issues is a skill.
Triaging incidents to prioritize actions to minimize user harm is a skill.
Assessing tradeoffs of mitigation strategies is a skill.
The very tangible scenario of being woken up in the middle of the night builds empathy with your devops / SRE organization (which can manifest itself in designing more resilient systems, engaging in less risky behavior, improving documentation for the next oncaller, etc).
Oncall isn't for everyone, and it's certainly not worthwhile if you don't take away any lessons from it. But devs are ultimately the closest ones to their code, and they should take advantage of the opportunity to understand how it behaves in production.
> Ask yourself.. what's next? What are you looking to do after 25 years of being on call? Will you still enjoy it?
I've had bad weeks of oncall as well as quiet weeks. If it ever gets to a point where bad weeks are much more frequent, and nobody's doing anything about it, then yeah, I won't want to do it anymore. But as long as it's still providing value beyond humans providing blood / sweat / tears, and as long as we're actually trying to make things better, I don't mind holding a pager a few weeks a year.
> these on call rotations aren't skills.
Troubleshooting live production issues is a skill.
Triaging incidents to prioritize actions to minimize user harm is a skill.
Assessing tradeoffs of mitigation strategies is a skill.
The very tangible scenario of being woken up in the middle of the night builds empathy with your devops / SRE organization (which can manifest itself in designing more resilient systems, engaging in less risky behavior, improving documentation for the next oncaller, etc).
Oncall isn't for everyone, and it's certainly not worthwhile if you don't take away any lessons from it. But devs are ultimately the closest ones to their code, and they should take advantage of the opportunity to understand how it behaves in production.
> Ask yourself.. what's next? What are you looking to do after 25 years of being on call? Will you still enjoy it?
I've had bad weeks of oncall as well as quiet weeks. If it ever gets to a point where bad weeks are much more frequent, and nobody's doing anything about it, then yeah, I won't want to do it anymore. But as long as it's still providing value beyond humans providing blood / sweat / tears, and as long as we're actually trying to make things better, I don't mind holding a pager a few weeks a year.
I can troubleshoot during business hours.
I can triag during business hours.
I can asset tradeoffs and mitigation strategies during business hours.
Being woken up in the middle of the night never builds empathy and if you pay enough attention to the industry, we're leadning more towards trauma, depression, anxiety and self-diagnosing ADHD.
BTW, me being anti-on call is not anti-resiliency... in fact, I don't think on-call is resilient at all with how most organizations operate since the resilience isn't adaptive capacity, but "do your day job and be on call" - no matter HOW much we talk about "your on call week is the week your on call and you focus on that" - it never happens. Never ever.
I can triag during business hours.
I can asset tradeoffs and mitigation strategies during business hours.
Being woken up in the middle of the night never builds empathy and if you pay enough attention to the industry, we're leadning more towards trauma, depression, anxiety and self-diagnosing ADHD.
BTW, me being anti-on call is not anti-resiliency... in fact, I don't think on-call is resilient at all with how most organizations operate since the resilience isn't adaptive capacity, but "do your day job and be on call" - no matter HOW much we talk about "your on call week is the week your on call and you focus on that" - it never happens. Never ever.
I get that any one of us can do all of these things during normal business hours (assuming that said situations arise then). But "can" isn't the same as "will". There are plenty of things I'll never get around to if I can't explicitly set aside time for oncall.
> Being woken up in the middle of the night never builds empathy and if you pay enough attention to the industry, we're leadning more towards trauma, depression, anxiety and self-diagnosing ADHD.
Being woken up in the middle of the night leaves me grumpy and motivates me the following work day to figure out how I can avoid getting woken up again for the same issue.
This is somewhat like dogfooding the oncall experience. If I'm trying to troubleshoot something but the logs are unhelpful or the runbook is too vague for my sleep-deprived brain to figure out, I can fix that for next time. If it takes me half an hour to answer a simple question, it's worth considering how to improve our monitoring and dashboards. If I can't follow my own documented process, how can I expect a team who didn't build these systems to do so?
As I said, I don't think oncall is for everyone. Oncall as a hidden job requirement is especially problematic, especially if your team / company doesn't expect (and/or mandate!) that your regular project work will be on the back burner during your shifts. Ideally oncall should be opt-in, but there's a balancing act with not having enough people to avoid rapid burnout. Google limits how much you can earn per oncall compensation (for SWEs, 2 weeks per quarter) -- if you're regularly exceeding that limit, then your oncall rotation isn't healthy.
There's a lot of horror stories in these comments about oncall rotations, and quite frankly I wouldn't want to join one of those either.
...
Another program Google has that's like oncall on steroids: Mission Control [0]. It's an exchange program where SWEs embed themselves in SRE teams, up to and including taking on their pager. It's a fantastic opportunity for gaining a deeper understanding / appreciation of production, but it can potentially be an even greater imposition. If you're an unattached 20-something, you might welcome the opportunity to live in Sydney for 6 months, but if you've got kids, that's not reasonable. (Of course, with the pandemic, relocation to a different country isn't really a thing at this time.)
[0] https://cloud.google.com/blog/products/gcp/adventures-in-sre...
> Being woken up in the middle of the night never builds empathy and if you pay enough attention to the industry, we're leadning more towards trauma, depression, anxiety and self-diagnosing ADHD.
Being woken up in the middle of the night leaves me grumpy and motivates me the following work day to figure out how I can avoid getting woken up again for the same issue.
This is somewhat like dogfooding the oncall experience. If I'm trying to troubleshoot something but the logs are unhelpful or the runbook is too vague for my sleep-deprived brain to figure out, I can fix that for next time. If it takes me half an hour to answer a simple question, it's worth considering how to improve our monitoring and dashboards. If I can't follow my own documented process, how can I expect a team who didn't build these systems to do so?
As I said, I don't think oncall is for everyone. Oncall as a hidden job requirement is especially problematic, especially if your team / company doesn't expect (and/or mandate!) that your regular project work will be on the back burner during your shifts. Ideally oncall should be opt-in, but there's a balancing act with not having enough people to avoid rapid burnout. Google limits how much you can earn per oncall compensation (for SWEs, 2 weeks per quarter) -- if you're regularly exceeding that limit, then your oncall rotation isn't healthy.
There's a lot of horror stories in these comments about oncall rotations, and quite frankly I wouldn't want to join one of those either.
...
Another program Google has that's like oncall on steroids: Mission Control [0]. It's an exchange program where SWEs embed themselves in SRE teams, up to and including taking on their pager. It's a fantastic opportunity for gaining a deeper understanding / appreciation of production, but it can potentially be an even greater imposition. If you're an unattached 20-something, you might welcome the opportunity to live in Sydney for 6 months, but if you've got kids, that's not reasonable. (Of course, with the pandemic, relocation to a different country isn't really a thing at this time.)
[0] https://cloud.google.com/blog/products/gcp/adventures-in-sre...
You can't set aside business hours if you don't set aside weekends and nights?
Professionalism, natural empathy, and performance evaluations motivate people enough in my experience.
Team exchanges are great. Why would it have to be 6 months abroad though?
Professionalism, natural empathy, and performance evaluations motivate people enough in my experience.
Team exchanges are great. Why would it have to be 6 months abroad though?
> Do you have kids? hobbies? friends? Do you like to go fishing on weekends?
Do you like having a job that gives you money to pay for all those things as well as the basic ability to, you know, live? How far do you want to take this? Is working 40 hours a week taking time away from your children, friends or hobbies? Of course it is.
But nobody owes you a living. A job is you trading your time and your skills to someone else in a way that provides that person value. There are going to be parts of that job you like and probably parts you don't.
If you really want to avoid oncall then fine, do it. That will likely limit your career opportunities because some jobs will require it or your coworkers may be rewarded (monetarily and careerwise) because they're doing something you're not. There's nothing wrong with that choice but at the same time you don't get to complain because others benefit from doing something you won't.
> but... it doesn't have to be that way.
For 24/7 production services then who exactly supports those services?
> I'll never have those nights and weekends back that I spent being on call for products and services that don't exist anymore.
There's a pretty good chance that nothing you worked on will exist in 25 years. What's your point?
Do you like having a job that gives you money to pay for all those things as well as the basic ability to, you know, live? How far do you want to take this? Is working 40 hours a week taking time away from your children, friends or hobbies? Of course it is.
But nobody owes you a living. A job is you trading your time and your skills to someone else in a way that provides that person value. There are going to be parts of that job you like and probably parts you don't.
If you really want to avoid oncall then fine, do it. That will likely limit your career opportunities because some jobs will require it or your coworkers may be rewarded (monetarily and careerwise) because they're doing something you're not. There's nothing wrong with that choice but at the same time you don't get to complain because others benefit from doing something you won't.
> but... it doesn't have to be that way.
For 24/7 production services then who exactly supports those services?
> I'll never have those nights and weekends back that I spent being on call for products and services that don't exist anymore.
There's a pretty good chance that nothing you worked on will exist in 25 years. What's your point?
The issue seems to be about unreasonable work hours. On-call time is work time - perhaps not "full" work time, but definitely not free time or rest time. If you're on call for 24 hours of Saturday, that's fine, but then work is taking up 24 hours more than your usual week, and deserves a major change in compensation or extra rest time (e.g. the following monday off) as it's a big change from a 40 hour or 50 hour week. If your team has allocated every engineer to an on-call day every week, that's the equivalent of expecting all of your engineers to devote 60+ hour weeks every week to their job - and IMHO that's not okay; some of your engineers may be fine with that (and would deserve what essentially is overtime compensation), but having that as a requirement for everyone is not reasonable in non-toxic workplaces, if there's no good place in your team for people who will only work 40-ish hours, then that would be pretty much discriminatory on multiple groups. Having a "week of on-call" is equivalent to a continuous 168-hour shift, which is simply not done anywhere else and should not be a thing.
If you want 24/7 production services, then you need to map shifts of support. For continuous work you just run 3x8 or 2x12 hour shifts; but outside of IT industry where you need "spotty monitoring", 24 hours on / 48 hours off is a commonly used system - but it's not a 24 hour shift on top of another full time job.
If you want 24/7 production services, then you need to map shifts of support. For continuous work you just run 3x8 or 2x12 hour shifts; but outside of IT industry where you need "spotty monitoring", 24 hours on / 48 hours off is a commonly used system - but it's not a 24 hour shift on top of another full time job.
Google SRE here.
Point of clarification on "losing skills" by not being oncall enough.
Google designs its SRE teams to scale sublinearly to the service, which means we're often responsible for entire portfolios, not just a single service.
It's common for individuals to be SMEs on a subset of the portfolio, as the focus of their coding projects.
Obviously the rest of the portfolio is also undergoing continuous change, and so to remain a broadly effective oncaller across the entire portfolio, you need regular production exposure to the parts you interact with less frequently.
Those are the specific bits that get rusty with disuse.
Point of clarification on "losing skills" by not being oncall enough.
Google designs its SRE teams to scale sublinearly to the service, which means we're often responsible for entire portfolios, not just a single service.
It's common for individuals to be SMEs on a subset of the portfolio, as the focus of their coding projects.
Obviously the rest of the portfolio is also undergoing continuous change, and so to remain a broadly effective oncaller across the entire portfolio, you need regular production exposure to the parts you interact with less frequently.
Those are the specific bits that get rusty with disuse.
I'm confused, the post you're replying to says that the Google model means you don't have nights OnCall.
> . these on call rotations aren't skills.
Incident response/management is absolutely a skill.
> . these on call rotations aren't skills.
Incident response/management is absolutely a skill.
You don't have to be on call to have that skill.
BTW, the google model works for them... i'm being more in general since the issue was facebook vs google vs everyone else that I was replying to.
BTW, the google model works for them... i'm being more in general since the issue was facebook vs google vs everyone else that I was replying to.
Incident response requires an incident responder. That is, a person who is alerted when things break.
That person (or persons) is defacto on call. If you are never responsible for responding to incidents, your ability to respond to incidents will atrophy.
That person (or persons) is defacto on call. If you are never responsible for responding to incidents, your ability to respond to incidents will atrophy.
I would advise such a person works during their normal 8-5 and that "on call" is just a form of work intake for a normal job - not an exception that happens after hours or on weekends when most people refer to "on call".
Ok, but then what do you do on weekends?
Like you still need people, who have the skills to do the oncall work, to man the pager during off-work hours.
Hiring a weekend person doesn't work (they don't have the skills since they're not really part of the team, + bus factor). Forcing people to work weekends doesn't work. Some sort of rotation seems the least worst option.
Like you still need people, who have the skills to do the oncall work, to man the pager during off-work hours.
Hiring a weekend person doesn't work (they don't have the skills since they're not really part of the team, + bus factor). Forcing people to work weekends doesn't work. Some sort of rotation seems the least worst option.
According to that post, "You don't have nights OnCall" applies only to SREs covering the systems of the highest profile.
Which is not even all SREs
Anyways SREs are software developers as well.
The distinction SWE/SRE came up to be in times of SOX compliance; nowadays there are also some oncall-related distinctions (that SREs are more likely to be in such multiple-timezone-office teams, for instance, could be true).
Which is not even all SREs
Anyways SREs are software developers as well.
The distinction SWE/SRE came up to be in times of SOX compliance; nowadays there are also some oncall-related distinctions (that SREs are more likely to be in such multiple-timezone-office teams, for instance, could be true).
Considering how much of Facebook's production engineering old-timers came from Google SRE, I am surprised to hear it is very different there. Perhaps it indicates the lack of a strong leader like Treynor setting the tone for the organization.
Strong sre culture not super compatible with their early “move fast and break things”
> For example, tasks at Facebook have SLAs and I saw plenty of games played to avoid dealing with tasks. Examples: changing priority to increase or remove the SLA, silently closing tasks, passing tasks back months later to the reporter asking "is this still an issue?", etc.
What kind of tasks are you referring to in the context of on call here? Tasks to remedy the root cause of repeated alarms paging folks?
What kind of tasks are you referring to in the context of on call here? Tasks to remedy the root cause of repeated alarms paging folks?
Task is a general all-encompassing term here as well as the internal task tracking tool everyone uses (called "Tasks").
Tasks could be filed by users of your service or external reports that get routed to your oncall or an alarm that fires because a metric moves outside of a "normal" range or whatever. Alarms ("pages") will come to you and need to be ACKed. Often they'll also generate a task. That task may require separate action or not. Many times those will just get closed or merged to existing tasks.
Tasks could be filed by users of your service or external reports that get routed to your oncall or an alarm that fires because a metric moves outside of a "normal" range or whatever. Alarms ("pages") will come to you and need to be ACKed. Often they'll also generate a task. That task may require separate action or not. Many times those will just get closed or merged to existing tasks.
I have a question. Is this for all environments (dev, qa, staging or whatever the equivalent is at google/fb) or just production?
SREs are (with a few exceptions) only responsible for production (or more precisely, things serving prod traffic, which may be multiple environments). Dev teams are usually responsible for keeping qa and ci green.
Me neither, but I'd like to keep getting paid to work on 24/7 availability systems, so I guess I'll put up with it, like all the other parts of my job I don't want to do.
I don't often want to be on-call, but when I do, it goes like this
My team managed to get paid for oncall that we had set up with a number of false positives, click-to-fix alerts, "override if slightly above threshold", rules for not paging outside of work hours, and whatnot that makes it bearable.
That extra compensation is at 1/3 of hourly rate, so effectively it is like paying for working during weekends (1/3 * 24h = 8h) and some smaller bonus for weekdays.
Perhaps it might make up for the pandemic inflation
My team managed to get paid for oncall that we had set up with a number of false positives, click-to-fix alerts, "override if slightly above threshold", rules for not paging outside of work hours, and whatnot that makes it bearable.
That extra compensation is at 1/3 of hourly rate, so effectively it is like paying for working during weekends (1/3 * 24h = 8h) and some smaller bonus for weekdays.
Perhaps it might make up for the pandemic inflation
We need federal legislation that bans super strong Service level agreements in cloud tech. Amazon has the best cloud entirely because of their hellish on-call powered by their absurd SLAs. I'm sure this is one of the most important components behind why there is so much oncall.
I'd love to see amazon and bezos get punished instead of rewarded just once for being slave drivers. Please pass federal legislation against "999" or other extremely absurd SLAs.
I'd love to see amazon and bezos get punished instead of rewarded just once for being slave drivers. Please pass federal legislation against "999" or other extremely absurd SLAs.
You can’t walk in here and compare slavery to working at a FAANG. Demonstrates either supreme hubris or supreme ignorance.
If you don’t like it, fucking quit? Slaves can’t do that
If you don’t like it, fucking quit? Slaves can’t do that
This model of oncall (what I currently operate off) is less abusive than how DevOps teams were run at my last company.
I would get called on days off, evenings, weekends. As late as 1am. I once brought my laptop to a Cubs game, because I knew I would get called (and I did). It was mostly because Devs broke the build, and their managers refused to have them learn how to triage that themselves because it would be extra responsibility.
I would get called on days off, evenings, weekends. As late as 1am. I once brought my laptop to a Cubs game, because I knew I would get called (and I did). It was mostly because Devs broke the build, and their managers refused to have them learn how to triage that themselves because it would be extra responsibility.
My team is distributed across the world. We ensure that we have multiple SME’s for the most critical systems in each region. No one has to wake up at 3 AM.
I used to be on call, and now I'm not anymore (but I changed jobs in between).
I'm not against being on-call but in my opinion if I get called tonight then tomorrow morning I want to be able to start a process that prevents the problem from happening again, and I want to be able to put EVERYTHING in discussion. I don't want to hear anything about development or operations, that thing must be root caused and it must whoever can implement a fix (either me, my team or the development team, either on our side or on customer side) must drop everything they're doing and implement a correction of error.
TL;DR:
If I can't fix stuff and prevent the problem from happening again then I don't want be on call.
I'm not against being on-call but in my opinion if I get called tonight then tomorrow morning I want to be able to start a process that prevents the problem from happening again, and I want to be able to put EVERYTHING in discussion. I don't want to hear anything about development or operations, that thing must be root caused and it must whoever can implement a fix (either me, my team or the development team, either on our side or on customer side) must drop everything they're doing and implement a correction of error.
TL;DR:
If I can't fix stuff and prevent the problem from happening again then I don't want be on call.
[deleted]
I am reading an article and comments and see that typical on-call schedule is weekly. That's really brutal. We have a daily rotation, 3 ops and 5 devs, additionaly paid about 1/2 daily wage no matter if it's weekend or not. We can swap our days if someone has some plans. Overall, being a part of on-call team is really sought after.
On call rotation = Company is unwilling to invest in global support of said product. If they're willing to cut corners with their employees precious time off just imagine what other foul business practices are happening behinds the scenes.
[deleted]
> Being paged out of hours is as serious as a fucking heart attack.
As someone who is oncall regularly and had close ones getting heart attacks I find this offensive. But hey, at the same time I understand the need for hyperbole to grab attention, and generally I think people get offended too easily.
Other than that my feelings about oncall are complicated. Most everyone hates it as engineers, but as customers we expect systems to be online all the time and recover quickly if they fail. Managed properly, oncall is a necessity for many products that doesn't need to be too disruptive.
As someone who is oncall regularly and had close ones getting heart attacks I find this offensive. But hey, at the same time I understand the need for hyperbole to grab attention, and generally I think people get offended too easily.
Other than that my feelings about oncall are complicated. Most everyone hates it as engineers, but as customers we expect systems to be online all the time and recover quickly if they fail. Managed properly, oncall is a necessity for many products that doesn't need to be too disruptive.
[deleted]
On call is a dysfunctional company culture attacking its own workforce in a battle of attrition. Its the equivalent of rheumatic fever to a company process. If you need it, your reliability engineering is failing and the process that should mend it, is instead busy fixing symptoms.
Such a company is already dieing and will do once the team "on call" is used up, burned out and gone.It will be replaced in the market by something more vital, still capable to value its workforce.
Such a company is already dieing and will do once the team "on call" is used up, burned out and gone.It will be replaced in the market by something more vital, still capable to value its workforce.
Are you saying, every fortune 100 tech company's reliability engineers are failing to do their jobs?
MAANG all have on-call individuals even with the fact they're across all timezones and have been growing steady.
MAANG all have on-call individuals even with the fact they're across all timezones and have been growing steady.
He's saying we're all suckers... and sometimes, it takes a long time to find out how true that really may be.
There are different ways of handling on-call and the practices at the companies you mention are very different, as is the on-call load.
There is a world of difference between someone voluntarily accepting on-call as part of their duties in exchange for higher pay or a 4 day work week, and on-call being forced on everyone.
There is a world of difference between someone voluntarily accepting on-call as part of their duties in exchange for higher pay or a 4 day work week, and on-call being forced on everyone.
Not an SWE, but how common is on call duty, and are there no companies that pay spot bonuses per issue?
It’s hard to quantify exactly how common it is. Common enough that you’d need to deliberately search for jobs that don’t have it, but not so common that you won’t find any.
I’ve never heard of spot bonuses per issue, although I imagine it would create a perverse incentive to have noisy alerting that files a lot of issues. Flat bonuses per oncall shift exist (and I think they’re a great idea) but they’re relatively rare.
I’ve never heard of spot bonuses per issue, although I imagine it would create a perverse incentive to have noisy alerting that files a lot of issues. Flat bonuses per oncall shift exist (and I think they’re a great idea) but they’re relatively rare.
It is common in internet software companies, less common in other kinds of software companies. Bigger companies tend to have multiple levels of compensation for different oncall rotations.
I'm sure you could pay me enough to be on call again, but so far nobody's offered.
[deleted]
Never again. I’m out.
Nothing wrong being on call as long as you are being paid to adequately. I always refuse being on-call if I am not being paid full hours plus extra for outside 9 to 5 for the entire duration regardless if anyone called.
Someone else is likely willing to be on call, though.
What some people struggle to realize is how easily the responsibility of an individual can scale at certain companies, without their compensation improving in the slightest. If you are working on-call and not making big bucks your company is likely milking you dry.