Ask HN: I can’t do oncall rotations. Am I done as a SE?
15 comments
It’s not that I want you to do on-call, I want you to be responsible for the work you do and feel the pain when it doesn’t work so you _fix_ the root causes of incidents (or at least automate mitigations).
The biggest threat to my company was the developers not fixing the problems they created. Yes, culture problems and “feature factory”, but this is part of how you fix those. (Of course you also need management support for above, super important to look for.)
There shouldn’t be on-call burnout with the above because you have agency to improve the problems. Matched with the situation continually improving and compensation for incidents. (Realistically the target is always moving and emergent behaviours will happen but this should still lie in the range of being reasonable, given a sufficiently large team among other things).
The biggest threat to my company was the developers not fixing the problems they created. Yes, culture problems and “feature factory”, but this is part of how you fix those. (Of course you also need management support for above, super important to look for.)
There shouldn’t be on-call burnout with the above because you have agency to improve the problems. Matched with the situation continually improving and compensation for incidents. (Realistically the target is always moving and emergent behaviours will happen but this should still lie in the range of being reasonable, given a sufficiently large team among other things).
> There shouldn’t be on-call burnout with the above because you have agency to improve the problems.
I've never seen a situation where management felt people needed to "feel the pain" and gave true agency to the developers. Lip service agency like, "You can put the issue into the backlog." But never true agency permitting the developers to reprioritize to fix the fundamental architectural issues causing the bulk of problems (often by developers who'd left the company or advanced into management and were no longer on call).
I've never seen a situation where management felt people needed to "feel the pain" and gave true agency to the developers. Lip service agency like, "You can put the issue into the backlog." But never true agency permitting the developers to reprioritize to fix the fundamental architectural issues causing the bulk of problems (often by developers who'd left the company or advanced into management and were no longer on call).
what would true agency actually look like? would the team "owning" the system get to capture the bulk of revenue or value the system produces -- so they could decide if they wanted to stack on another feature that might improve revenue by blah percent versus spend the quarter trying to stabilise the system to reduce the number of false-positive alerts or avoidable outages.
I already answered your question, but to reiterate:
If the belief is that developers should be on-call (as the OP I responded to suggested) so that they "feel the pain" and address the issues, then they actually need agency to address the issues.
If the problems are in someone else's code (a purchased system, another department), then they have no agency to address the problem beyond entering tickets.
If the problems are in code that they are responsible for, but are not permitted to change (it's too low in the priority queue), then they have no agency.
If the problems are in the structure of the system and they aren't permitted to change it (or, again, it's too low in the priority queue), then they have no agency.
They have to be permitted to restructure, recode, and reprioritize everything to have agency. Otherwise the "feel the pain" approach is just management fucking around not wanting to hire enough staff to handle the on-call rotations, but covering it up by claiming they have a "good" reason but failing at follow through.
As a real-world example of this, a place I worked at did the on-call rotation for devs, but had a problem with a message queue. The devs weren't permitted to change its configuration (tune it). They weren't permitted to purchase new hardware to distribute the load. They weren't (it was a purchased system) permitted to change the code to the message queue system. They just got calls every night saying that the queue was full and had to address it. Every single night. There were no fixes because they weren't empowered to fix it. Naturally, a lot of people left rather quickly because they got tired of management's bullshit "feel the pain but don't fix the problem" attitude.
And turnover is another issue. Even if the problem were caused by those developers (and not a component they couldn't control), turnover, which happens in every company, means that at some point the devs getting the calls won't even be the ones who made the system. They're just the poor schmucks who hired on thinking they were going to be software developers, but who got saddled with an extra job they weren't actually trained for.
If the belief is that developers should be on-call (as the OP I responded to suggested) so that they "feel the pain" and address the issues, then they actually need agency to address the issues.
If the problems are in someone else's code (a purchased system, another department), then they have no agency to address the problem beyond entering tickets.
If the problems are in code that they are responsible for, but are not permitted to change (it's too low in the priority queue), then they have no agency.
If the problems are in the structure of the system and they aren't permitted to change it (or, again, it's too low in the priority queue), then they have no agency.
They have to be permitted to restructure, recode, and reprioritize everything to have agency. Otherwise the "feel the pain" approach is just management fucking around not wanting to hire enough staff to handle the on-call rotations, but covering it up by claiming they have a "good" reason but failing at follow through.
As a real-world example of this, a place I worked at did the on-call rotation for devs, but had a problem with a message queue. The devs weren't permitted to change its configuration (tune it). They weren't permitted to purchase new hardware to distribute the load. They weren't (it was a purchased system) permitted to change the code to the message queue system. They just got calls every night saying that the queue was full and had to address it. Every single night. There were no fixes because they weren't empowered to fix it. Naturally, a lot of people left rather quickly because they got tired of management's bullshit "feel the pain but don't fix the problem" attitude.
And turnover is another issue. Even if the problem were caused by those developers (and not a component they couldn't control), turnover, which happens in every company, means that at some point the devs getting the calls won't even be the ones who made the system. They're just the poor schmucks who hired on thinking they were going to be software developers, but who got saddled with an extra job they weren't actually trained for.
If you can't remove yourself from the oncall list...
...then you already have no agency.
...then you already have no agency.
In one of my first jobs at Fortune 20 company, they had on-call escalation procedure, longer you take to fix an issue, more and more people get called. It was something like on-call developer > team lead > whole team and manager > director > senior director > VP... Usually, after an hour, we had to get manager and whole team online. Manager would be freaking out and driving us hard to fix outage with whatever means necessary and not let it escalate further. They would make it sound like our whole team will suffer if director got on the call. Of course, they said that we can create issues and fix them and of course, there were always more important issues.
My team quickly and informally learned if you want to take care of technical debt, let the higher ups feel the pain. Even when nginx/some process died and just need to restart, we would wait and really try to let those calls escalate to director. And anytime director had to join bridge, those issues would become highest priorities and fixed permanently.
It also became a game where manager would keep asking to do simple things to fix and we would be stalling or looking at unimportant things. Pretending like slow internet connection, dropped calls, locking ourselves out from the box, etc.
And in our team we were all PS3 gamers, so we would get on GTA4, and start having fun. A few times we even yelled at each other for messing up in the game while unmuted.
These days depending on company's culture I may fix issue quickly or I may let it escalate to right level where we will get support for fixing the issue for good.
My team quickly and informally learned if you want to take care of technical debt, let the higher ups feel the pain. Even when nginx/some process died and just need to restart, we would wait and really try to let those calls escalate to director. And anytime director had to join bridge, those issues would become highest priorities and fixed permanently.
It also became a game where manager would keep asking to do simple things to fix and we would be stalling or looking at unimportant things. Pretending like slow internet connection, dropped calls, locking ourselves out from the box, etc.
And in our team we were all PS3 gamers, so we would get on GTA4, and start having fun. A few times we even yelled at each other for messing up in the game while unmuted.
These days depending on company's culture I may fix issue quickly or I may let it escalate to right level where we will get support for fixing the issue for good.
> feel the pain when it doesn’t work
If I have to FEEL THE PAIN when software I work on contains bugs, I need to be paid a whole of lot more than what you are offering.
If I have to FEEL THE PAIN when software I work on contains bugs, I need to be paid a whole of lot more than what you are offering.
Some options:
- contracting/consulting - quite often no on-call duty as it is tricky to bill,
- working on software for brick&mortar businesses, such who usually overlap with your work hours,
- working in R&D area where you don't touch prod, where there is a dedicated team who does the integration, deployment and prod support - often quite popular in big organizations with long release cycles etc.
- contracting/consulting - quite often no on-call duty as it is tricky to bill,
- working on software for brick&mortar businesses, such who usually overlap with your work hours,
- working in R&D area where you don't touch prod, where there is a dedicated team who does the integration, deployment and prod support - often quite popular in big organizations with long release cycles etc.
I think it’s fine to set boundaries. I joined an organization and 6 months in was invited to a meeting (with 30 other devs) to discuss our onboarding to prod support on the weekends and such. I politely refused and held firm. I’ve done enough of that that I never want to do it again.
It’s still an employees market out there. Find something that fits.
It’s still an employees market out there. Find something that fits.
Situations like these are where recruiters can actually be really helpful. When looking for work with specific qualities I usually open up LinkedIn and start responding to recruiters thanking them for reaching out and explaining the type of work I'm looking for.
There are definitely senior dev jobs with no oncall. I think a lot of developers feel the same way you do. Perfectly reasonable stipulation.
Chances are you've just run across a few in a row that do oncall. When you get on a streak with something like that it can feel like it's every job.
There are definitely senior dev jobs with no oncall. I think a lot of developers feel the same way you do. Perfectly reasonable stipulation.
Chances are you've just run across a few in a row that do oncall. When you get on a streak with something like that it can feel like it's every job.
> Unfortunately, companies these days require senior software engineers to be oncall from time to time because, well, their culture says so, or something like that...
What type of companies are you applying at (industry, size, etc.)? What is your geographic location?
There are tons of companies in the world that employ SEs and I know plenty of SEs who are not required to do on-call rotations. In fact, I've never heard of this being common.
I suspect your experience might be a function of what type of companies you're applying to and perhaps where they're located.
What type of companies are you applying at (industry, size, etc.)? What is your geographic location?
There are tons of companies in the world that employ SEs and I know plenty of SEs who are not required to do on-call rotations. In fact, I've never heard of this being common.
I suspect your experience might be a function of what type of companies you're applying to and perhaps where they're located.
Same here. I’ve been a software engineer for 20+ years and I’ve never been on call. I’ve put out fires and fixed urgent bugs, of course.
A company that forces on-call on its devs is a huge red-flag.
Good companies incentivize for it and don't need to force it upon anyone.
Good companies incentivize for it and don't need to force it upon anyone.
Agree. I would go further and say: good companies hire SREs to be the ones on-call (just like good companies hire QA engineers for testing, and designers for UI/UX topics).
[deleted]
- play their game: do countless interviews (phone, hangouts, pair programming, you name it) - excel in each part of the interview - once they know I’m a good fit for their company, I ask more than they usually pay for whatever the role I am applying for. Sometimes I ask for more vacation days or other similar perks
Now, I have to say that my tactic has worked out so far so good. Recently, though, I have been hitting a wall: oncall rotations. I reject any kind of work besides the usual 9-5; it’s nonnegotiable for me. I don’t need more money (it’s not that I’m earning a lot either), but even if I would needed it, I would prefer to work “harder” (smarter?) during the usual 9-5 hours.
The stress of potential incidents occurring during the night or on weekends gives me anxiety and no amount of money can fix that. Unfortunately, companies these days require senior software engineers to be oncall from time to time because, well, their culture says so, or something like that (“In this company every team owns their products from conception to deployment and maintenance. We expect from engineers to have a business mindset, to be proficient using a variety of tech stacks, and to be there to keep the software up and running in case of unexpected failure”).
Is this a trend or I just got bad luck in my recent findings?
Edit: sector is web development.