Ask HN: How often are you on call?
Please share a few words on what size company you work for, and what size team you’re apart of. If you’d like to share, mention how often you’re paged or activated!
45 comments
Never, and I don't consider any job that demands this. Learn what work-life separation means.
I'm at the complete opposite of the spectrum there, on-call 24/7/365.
Then again partially own the company and enjoy my colleagues. If I go dark for a few days answering non-urgent queries no one questions it because they know I wake up at 3am and put out digital fires within short timeframes.
Life and balance are just words in the end, you can clock off at exactly 5pm to the nanosecond and still not find that holy grail ever. It is what you make it.
That's clearly not a sustainable answer for the world - professions ranging from doctors to priests to soldier to sailor to CEO (can't ignore a crisis if it happens on Saturday morning) to lawmaker all require working the occasional unscheduled hour. And that's not even counting jobs like pilot or convenience store clerk or paramedic that require at least some people to work on what the world considers holidays (even if it's scheduled as part of their 40 hours a week).
Is there a meaning of "work-life separation" that encompasses these professions too, or shall we say the world relies on some people not having it?
Is there a meaning of "work-life separation" that encompasses these professions too, or shall we say the world relies on some people not having it?
If someone is dying, go ahead and page me!
Otherwise, companies need to hire night shifts if they want someone covering while the devs are asleep. If you want full uptime, then hire for it! Your employees shouldn't give up their sleep and weekends for a measly 0.003% of your company.
Also, I would strongly support extra pay for low-wage workers working holidays, weekends, or night shifts.
Otherwise, companies need to hire night shifts if they want someone covering while the devs are asleep. If you want full uptime, then hire for it! Your employees shouldn't give up their sleep and weekends for a measly 0.003% of your company.
Also, I would strongly support extra pay for low-wage workers working holidays, weekends, or night shifts.
It's pretty naive to think you can just hire random people to step in and support complex business systems part-time.
Sure, you can have a global ops team that can deal with basic system issues like "this machine ran out of disk space", but not "we're seeing a spike in 500s in our payment system because something that got deployed earlier today broke a workflow". And saying "well just roll back!" isn't a solution either. Maybe you can't roll back. Maybe that won't fix it because data is now corrupt. Maybe it wasn't even a deploy to that system that broke things, but some feature flag in some distant system. No one except the team itself can diagnose these issues appropriately.
Sure, you can have a global ops team that can deal with basic system issues like "this machine ran out of disk space", but not "we're seeing a spike in 500s in our payment system because something that got deployed earlier today broke a workflow". And saying "well just roll back!" isn't a solution either. Maybe you can't roll back. Maybe that won't fix it because data is now corrupt. Maybe it wasn't even a deploy to that system that broke things, but some feature flag in some distant system. No one except the team itself can diagnose these issues appropriately.
> or shall we say the world relies on some people not having [...]?
It does rely on that. Supply and demand works that way. Eventually the offered price is enough to change some minds. I'll change mine at about $3k/hour-on-call.
It does rely on that. Supply and demand works that way. Eventually the offered price is enough to change some minds. I'll change mine at about $3k/hour-on-call.
I’ll do it for $100/h on call. Sweet gig if you do 40h a month on call that’s a sweet $4000 a month and hopefully most times you won’t even be called.
Sounds like you’re being a facetious but there are plenty of companies that do pay you if you do end up getting paged. Not for being scheduled on call. Plenty of companies also do 12 hour shifts during the day if you’re on call and have SREs handle the rest for the most part.
There are companies that compensate you for all on call hours.
Being paid only if you are paged is not appropriate compensation for interrupting your personal life: having your laptop, reliable internet, cellular coverage, being available on x minutes notice.
Being paid only if you are paged is not appropriate compensation for interrupting your personal life: having your laptop, reliable internet, cellular coverage, being available on x minutes notice.
Working N hours in unusual schedule is different from working N hours in usual schedule and then being on call.
With the former, you get some compensation, for example you may work on Saturday but have a free Monday. Depending on your priorities, it may actually be a good deal -- there are things you can do more conveniently when you have a free Monday.
With the latter, it's "when you're at work, you're at work, and when you're at home, there is still a tiny voice in your head remininding you that your boss could call you at literally any moment". This may limit the ways you spend your free time, for example you can't go to places where it would be impossible or very inconvenient to get quickly online when called. And generally, you just can't make promises even about how you spend your free time.
For a sufficiently large company, it should be possible to find a few people willing to take an unusual schedule, and then the company could be covered 24/7, with every employee able to turn off their work phone the moment they leave. It's just... more convenient for the managers to do it the other way round.
With the former, you get some compensation, for example you may work on Saturday but have a free Monday. Depending on your priorities, it may actually be a good deal -- there are things you can do more conveniently when you have a free Monday.
With the latter, it's "when you're at work, you're at work, and when you're at home, there is still a tiny voice in your head remininding you that your boss could call you at literally any moment". This may limit the ways you spend your free time, for example you can't go to places where it would be impossible or very inconvenient to get quickly online when called. And generally, you just can't make promises even about how you spend your free time.
For a sufficiently large company, it should be possible to find a few people willing to take an unusual schedule, and then the company could be covered 24/7, with every employee able to turn off their work phone the moment they leave. It's just... more convenient for the managers to do it the other way round.
Seems unreasonable to say never. Like what a sibling commenter said, it seems like you’ll miss out on potential earnings in your career by being this inflexible.
Just like I miss out on some earnings by not being a contract killer. Some things aren't worth it. That is a personal choice. And this is mine.
One week out of two, for a company size of 20.
Company the size of about 10.
I am the only person on call 24/7.
I am the only person on call 24/7.
once every 5 weeks, FAANG
It depends on how many people we have on the team! Right now it’s every three weeks. A while ago it was every 6 weeks.
I used to get paged dozens of times a week. Now it’s either once a week, or a few dozen times a week (usually this is a blip that can just be ignored because something went wrong and then fixed itself). Typically there’s one small actionable item every week I’m on call.
The company is 500 people, my team varies in size a lot as people quit and we try and hire replacements. It’s been as small as 3 and as big as 12.
I used to get paged dozens of times a week. Now it’s either once a week, or a few dozen times a week (usually this is a blip that can just be ignored because something went wrong and then fixed itself). Typically there’s one small actionable item every week I’m on call.
The company is 500 people, my team varies in size a lot as people quit and we try and hire replacements. It’s been as small as 3 and as big as 12.
I’d also be interested if people could add if being on call is (a) voluntary, or (b) required on their team.
Also, it’d be good to know if on call comes with additional compensation?
Also, it’d be good to know if on call comes with additional compensation?
[deleted]
Large cross-site team. We do 12x4/12x3 rotations (Mon-Thu, Fri-Sun) so everyone stays fresh.
I've heard others are trying one-day rotations to make scheduling holidays fairer and more flexible for people who have commitments on certain days.
I've heard others are trying one-day rotations to make scheduling holidays fairer and more flexible for people who have commitments on certain days.
Small startup, tech team of about 10 (2 SREs + 7 devs + 1 biz, total company size is 20, fully remote distributed across 3 continents), roughly 8 hours twice a week and only during my local timezone business hours, then the same 8 hours in one weekend day every two weeks, also roughly. We have lots of automated checks and alerts and only react to critical ones paging us via Slack (most of the alerts that page us are known, some are flaky some are false positives we can't ignore).
It was kind of forced on everybody (prematurely IMHO) but respecting people's timezones and business hours was key to make it acceptable and you can take the next day off no-questions-asked if you were on call on a weekend day even if you didn't have to act on anything. It's far from ideal considering how we still react to the critical alerts and what in them usually triggers actual on-call support but I've had worse in the past so it's alright... for now.
It was kind of forced on everybody (prematurely IMHO) but respecting people's timezones and business hours was key to make it acceptable and you can take the next day off no-questions-asked if you were on call on a weekend day even if you didn't have to act on anything. It's far from ideal considering how we still react to the critical alerts and what in them usually triggers actual on-call support but I've had worse in the past so it's alright... for now.
Currently contracting for a ~10 person company.
In the past I've mostly worked at small companies and have never officially been on call. I've fixed the odd out of hours issue here and there without there being a proper arrangement.
The systems I work on (trading / betting software) are such that by the time you're through testing it's very rare to hit a game breaking issue because there's a very large incentive to not like, lose the bank.
Dependencies in the critical path are kept to a minimum, etc.
Usually problems are more 'soft', like an API being turned off or changed or something.
In the past I've mostly worked at small companies and have never officially been on call. I've fixed the odd out of hours issue here and there without there being a proper arrangement.
The systems I work on (trading / betting software) are such that by the time you're through testing it's very rare to hit a game breaking issue because there's a very large incentive to not like, lose the bank.
Dependencies in the critical path are kept to a minimum, etc.
Usually problems are more 'soft', like an API being turned off or changed or something.
I'm on call 24/7. It's fairly normal to wake up at 2 am because there's a pressing situation and a client is in need. That said, I actively chose this situation for myself and find it to be pretty rewarding. Of course, it requires some planning and coordination, so my team divvied up our responsibilities accordingly to ensure that there are always resources available for taking meetings on a normal schedule.
It's probably worth noting, we work with large brands and marketing agencies from all around the world and are often called upon to 'save the day' once someone's plans have fallen through. As a result, time is usually the one thing our clients don't have enough of and being able to solve their problems ASAP is what allows our clients to get back to normal operation. Given that, it's easy for me to pick up the phone at 2 am because I know that whoever is calling me is going to be having a worse time than I am.
It's probably worth noting, we work with large brands and marketing agencies from all around the world and are often called upon to 'save the day' once someone's plans have fallen through. As a result, time is usually the one thing our clients don't have enough of and being able to solve their problems ASAP is what allows our clients to get back to normal operation. Given that, it's easy for me to pick up the phone at 2 am because I know that whoever is calling me is going to be having a worse time than I am.
Large company. Team of 12. Do very specialized work.
On call: Never and never have been in 25+ years across multiple large tech companies.
Seriously considered looking into a job at Facebook where this would have been required and it was definitely one of the considerations that made that position very unappealing.
On call: Never and never have been in 25+ years across multiple large tech companies.
Seriously considered looking into a job at Facebook where this would have been required and it was definitely one of the considerations that made that position very unappealing.
I just started working for the government, and I got yelled at for sending an email on Saturday. The union contract says we're paid for overtime if we work more than 40 hours, but in turn, we never are supposed to work overtime unless maybe the world is ending.
I used to work for a company that had a departmental phone and we would rotate to a different person each week to be on call. However, waking people was essentially eliminated by adding people on third shift (including me) and in India.
In both cases, I was on a team of about 5-10 in an agency or division of a couple or three hundred, within a much larger organization.
I used to work for a company that had a departmental phone and we would rotate to a different person each week to be on call. However, waking people was essentially eliminated by adding people on third shift (including me) and in India.
In both cases, I was on a team of about 5-10 in an agency or division of a couple or three hundred, within a much larger organization.
Never ever.
Current company has 400 employees my team is around seven but we'll have to grow very fast very shortly.
Current company has 400 employees my team is around seven but we'll have to grow very fast very shortly.
I feel I’m in the sweet spot. Work for a company based across 5 time zones. Our team of 12 has optional out of hours support and people have to volunteer. But it is only for Sev 1 issues where the entire app/service is down.
Since it is paid, we end up taking turns for the extra money. In the year I’ve worked there, haven’t been pinged even once.
Since it is paid, we end up taking turns for the extra money. In the year I’ve worked there, haven’t been pinged even once.
tl;dr One week out of four.
I’m an SRE on a team of 4 (incl manager). The company is about 140 people. Each of my team members participates in on-call (incl manager) and is on for a week at a time. We’ve dialed things in such that we might get paged once or twice during that week and usually the pages wind up being minor/non-issues. It’s really not bad, in my opinion.
I’m an SRE on a team of 4 (incl manager). The company is about 140 people. Each of my team members participates in on-call (incl manager) and is on for a week at a time. We’ve dialed things in such that we might get paged once or twice during that week and usually the pages wind up being minor/non-issues. It’s really not bad, in my opinion.
Presently, one week in five. Average shift has one overnight page and one other out of hours page (with high variance).
I am compensated for the oncall, but not a meaningful amount.
I am compensated for the oncall, but not a meaningful amount.
Work for FAANG, team size is 6-8~. I'm oncall 24/7 roughly one week out of 6. Usually paged 1-2 times per shift except during 'peak' periods but we usually subdivide the rotation for those.
My worst shift I was paged during off hours twice, but was only woken one of those times.
My worst shift I was paged during off hours twice, but was only woken one of those times.
My tier 3 team is on 1-week point-Dev rotations. Roughly 1wk in 9 on primary, same again on secondary. They take about 2-3 calls per primary+secondary. I’m on Tier 4 in mgmt roster, and on call every other week. Take escalations and fall thrus every 3-4mths. Tier 4 also plays duty manager during events such as product launches.
Largeish company (engineering staff around 1000, I think?), and my team owns the OS platform (packaging, config management), hardware purchasing, and the installer, as well as our private cloud, which hosts a good chunk of services (notably including dev machines, notably excluding the most money-making parts of the business). A separate team is first-line sysadmin+SRE for most of the company, so we don't respond unless you can't install machines or if every machine is broken, or if the private cloud (as infrastructure, not a specific VM on it) is acting up.
Private cloud was originally a separate team, and we had about 5 people on a weekly rotation, where you'd do support and 24/7 on-call for a week. Now that we've merged, it's about a dozen people on the on-call rotation, though we still have a subteam of about 5 for the weekly rotation of who's paying attention to private cloud support tickets during working hours.
Thanks to some work that we did a bit after I joined, we got the off-hours paging to be much less frequent - in particular we no longer page if any hypervisor is down, only if either a whole availability zone is down, or a hypervisor hosting personal dev machines is down and it's business hours. (Otherwise, higher-level services ought to have failover between multiple VMs, and devs working odd hours can use backup machines.) Usually that second condition gets noticed by the support person (in Jira) before it escalates to the on-call person. So it's rare to have anyone actually paged, and it usually means something unexpected is on fire. So, any member of the team is as good as any other at figuring out what broke and whether it can wait until morning, which is why we were able to throw all of the combined team into the rotation without training them all in everything.
There are still improvements I want to do, but the expected number of pages rounds to zero (it looks like we've had 6 off-hours pages this year) so it's certainly manageable.
Private cloud was originally a separate team, and we had about 5 people on a weekly rotation, where you'd do support and 24/7 on-call for a week. Now that we've merged, it's about a dozen people on the on-call rotation, though we still have a subteam of about 5 for the weekly rotation of who's paying attention to private cloud support tickets during working hours.
Thanks to some work that we did a bit after I joined, we got the off-hours paging to be much less frequent - in particular we no longer page if any hypervisor is down, only if either a whole availability zone is down, or a hypervisor hosting personal dev machines is down and it's business hours. (Otherwise, higher-level services ought to have failover between multiple VMs, and devs working odd hours can use backup machines.) Usually that second condition gets noticed by the support person (in Jira) before it escalates to the on-call person. So it's rare to have anyone actually paged, and it usually means something unexpected is on fire. So, any member of the team is as good as any other at figuring out what broke and whether it can wait until morning, which is why we were able to throw all of the combined team into the rotation without training them all in everything.
There are still improvements I want to do, but the expected number of pages rounds to zero (it looks like we've had 6 off-hours pages this year) so it's certainly manageable.
Technically 24/7, except for clustered stuff that hardly goes down. More than a year since the last time; I think. Thousands of employees, team of 7 or so
Quite a lot. 30% of the day on average.
Small bootstrappes company working on SaaS. Im the lead and has 2 programmers on the team, so Im the one usually on call, 24/7.
Large university, team of 8. All out-of-hours work is on a “best effort” basis only, i.e. I’ll do what I can (which may be nothing) if emailed.
I run a bootstrapped SAAS business with total team size of 12. I am technically always on call if shit hits the fan. Not in the traditional sense though. If server goes down (has happened like 2 times in past 5 years), I get a call no matter what time of the night it is.
SRE at Google. We are oncall for 12h then the other site takes over for the other 12h, so no night shifts. You do that for 7 days in a row once every 5 week-ish. Some team have slight variations on that.
We get paged around once per 12h on average but our service is fairly young. More mature services tend to page less I think.
We get paged around once per 12h on average but our service is fairly young. More mature services tend to page less I think.
What is the on-call compensation?
Last I knew it was 33.3% time credited for on call hours with 30 minute response requirement and 66.6% time credited for on call hours with 5 minute response time. And it could be taken as additional holiday or as cash.
Last I knew it was 33.3% time credited for on call hours with 30 minute response requirement and 66.6% time credited for on call hours with 5 minute response time. And it could be taken as additional holiday or as cash.
I work at a 100-person startup. We currently do 1 week secondary followed by 1 week primary. This comes up every 2.5 months. On-call is 8am-8pm with any overnight pages going to execs.
I'll not that we originally had people doing daily shifts and it was a huge pain for those who were religiously observant. It was much harder to get someone to cover a Saturday or Sunday shift since nobody wants to cover those. Will send much better
do you have any staff in other timezones to get better night coverage?
I own a marketing agency. I check my email often just on case but I rarely have to hop on my laptop after hours. I'll probably pen an after hours short email a few times a week though.