Tell HN: I salute everyone on call/working support through the holidays
Thank you for keeping systems available and safe. I've been there many times in the past, including having to fly at the last minute to a non-internet-connected data center in NJ to babysit an emergency production bug fix that took the entire holiday to create, install, verify, and monitor.
206 comments
Always be kind, and say it’s your fault.
If you don’t do it for the sake of the person you are asking for help, do it because it works better. That’s the most practical advice [0] ever given by Hans Rosling [1], the Fact master himself:
> In fact, I have the secret to how to get the best help immediately from any customer service, like the phone company or the bank or anything. I have the best line, it always works. You want to know what it is? When I call, I say, “Hello. I am Hans Rosling and I have made a mistake.” People immediately want to help you when you put it this way. You get much more when you don’t offend people.
[0]: Unless you are in charge of a developing country’s budget and have to decide between education and healthcare.
[1]: https://blog.ted.com/qa_with_hans_ro_1/
If you don’t do it for the sake of the person you are asking for help, do it because it works better. That’s the most practical advice [0] ever given by Hans Rosling [1], the Fact master himself:
> In fact, I have the secret to how to get the best help immediately from any customer service, like the phone company or the bank or anything. I have the best line, it always works. You want to know what it is? When I call, I say, “Hello. I am Hans Rosling and I have made a mistake.” People immediately want to help you when you put it this way. You get much more when you don’t offend people.
[0]: Unless you are in charge of a developing country’s budget and have to decide between education and healthcare.
[1]: https://blog.ted.com/qa_with_hans_ro_1/
> Always be kind, and say it’s your fault.
I do this with internal teams at work. I've found approaching other teams with issues with their library/framework in a "this could be our mistake" manner really helps in keeping them from getting defensive and stonewalling.
I do this with internal teams at work. I've found approaching other teams with issues with their library/framework in a "this could be our mistake" manner really helps in keeping them from getting defensive and stonewalling.
Saying, "I'm sorry; I've made a mistake" is the killer disarming technique for even the most emotional conflict. Not sure if it's our pride or fear of liability but western culture is very hesitant to say "sorry" - other than the fake one "I'm sorry if anyone interpreted my actions|remarks|words as ..." - that doesn't count.
I know this is a tangent, but if anything, western culture is better about this. We're not an "honor" culture, and we don't warp family relationships to "save face". People are not even allowed to apologize in some other cultures because it brings shame on their family/group.
To give a really grotesque example, there is an country run by autocrats that carelessly unleashed a plague on the world because they prioritized their self image over taking descisive action to prevent an epidemic.
To give a really grotesque example, there is an country run by autocrats that carelessly unleashed a plague on the world because they prioritized their self image over taking descisive action to prevent an epidemic.
There are plenty of examples of honor culture within western culture, such as the Antebellum South. Dueling was huge amongst the aristocratic cultures of Europe and US (see Alexander Hamilton). It is certainly making a comeback today. In the US specifically certain people were against mask mandates to the point where they actually banned people wearing masks during the height of the COVID-19 pandemic. Masks became a point of contention where some people needed to defend the family/group from wearing them at any cost.
Do NOT do this too much, though. I did this in my last relationship and it turned into resentment from me, while she walked all over me, knowing I would say sorry for everything. It wasn’t her fault, I should’ve protected my values more, but this is a slippery slope. However, my next partner in my opinion should take this and say something like “no, this is not your fault. Let’s work on it”.
It could go both ways. I can never change though; saying sorry is too simple for me
It could go both ways. I can never change though; saying sorry is too simple for me
> she walked all over me
> It wasn’t her fault
Sir, I don't know you at all, but coming from personal experience, if you often get walked over, maybe it's time to stop taking shape of a door mat. Maybe it _was_ her fault, and you deserve better.
> It wasn’t her fault
Sir, I don't know you at all, but coming from personal experience, if you often get walked over, maybe it's time to stop taking shape of a door mat. Maybe it _was_ her fault, and you deserve better.
> maybe it's time to stop taking shape of a door mat
I'm sorry, I know this is a serious discussion, but this made me laugh. I need to use this in the future.
I'm sorry, I know this is a serious discussion, but this made me laugh. I need to use this in the future.
I would agree on this from experience, even though I would agree with the initial statement as well. The advice applies for some parts in life but not all social contexts.
Admitting weakness gets exploited very easily by people.
Me: Sorry, I forgot about some details about the stuff I was working on last month.
Him: Your memory is so bad. I can literally remember all the details of the last 6 months. Try to become more like me.
Me: Sorry, I forgot about some details about the stuff I was working on last month.
Him: Your memory is so bad. I can literally remember all the details of the last 6 months. Try to become more like me.
> but western culture is very hesitant to say "sorry"
This is very wrong. Eastern/Asian culture with its obsession with saving face is the culture where saying sorry is anathema.
This is very wrong. Eastern/Asian culture with its obsession with saving face is the culture where saying sorry is anathema.
Just don’t generalise about massive groups of people? “Eastern” and “western” culture are not monoblocs.
"culture" itself is a stupid generalization
> Eastern/Asian culture with its obsession with saving face is the culture where saying sorry is anathema
Ever been to Japan?
Ever been to Japan?
It's also very Japanese to say "sorry" and accepting blame I believe. That might be what they mean with by "Asian culture", but it's certainly doesn't apply to all of Asia.
Years ago we started to get basic introduction into Chinese culture, so that we could better navigate situations with a Chinese customer. This helped to better understand requirements and defuse certain situations. I've NEVER experience Chinese or Indian companies go to the same length, instead they will frequently attempt to bullshit their way out of situations or be offended that you believe that their product/service might in some way be at fault and they sure as hell won't apologize for it, under any circumstances.
Years ago we started to get basic introduction into Chinese culture, so that we could better navigate situations with a Chinese customer. This helped to better understand requirements and defuse certain situations. I've NEVER experience Chinese or Indian companies go to the same length, instead they will frequently attempt to bullshit their way out of situations or be offended that you believe that their product/service might in some way be at fault and they sure as hell won't apologize for it, under any circumstances.
Absolutely. The same is true for bug reports: if you always approach any bug report with the possibility that it may be your mistake, 1) you avoid annoying someone if it is your mistake, and might actually get helpful advice, and 2) you're more likely to get a cordial reception for real bugs.
You don't have to be excessively self-effacing about it, just avoid presenting things as though the project you're reporting it to being at fault is the only possible conclusion.
You don't have to be excessively self-effacing about it, just avoid presenting things as though the project you're reporting it to being at fault is the only possible conclusion.
It's also important to consider how what you view as a bug might not be one from the point of view of the person treating your issue. It is so, so infuriating to receive a "bug" report asking you to "fix" something that is in fact a feature request for something that is not implemented yet.
Even if you get an error from the software, consider that you might not be using it as intended or setup properly.
Even if you get an error from the software, consider that you might not be using it as intended or setup properly.
I do something similar. Hey, I'm pretty sure I'm doing something wrong. Can you help me figure it out?
Then be grateful for the help, because it truly isn't granted or a given that people have to drop everything and figure things out for you, even if you work together. And even if the mistake was actually theirs. Gratitude is huge.
Then be grateful for the help, because it truly isn't granted or a given that people have to drop everything and figure things out for you, even if you work together. And even if the mistake was actually theirs. Gratitude is huge.
Whatever happened to honesty?
Sure, be kind, but don’t bullshit people. My personal and professional tolerance for bullshit is very low these days because there’s so much of it.
Sure, be kind, but don’t bullshit people. My personal and professional tolerance for bullshit is very low these days because there’s so much of it.
Say "I might have made a mistake" as there's always a non zero probability even if small that you actually have even if you believe 100% that you haven't.
Yeah because a bunch of people will read this and instead of realising this means their state of mind should be one of humbleness and respect for who is going to help you, and that's why you lead with "hey I made a mistake", they'll still feel like other people messed up and are dumb and an obstacle to whatever they need to do but will add "I made a mistake" as a first sentence "life hack". Usually you can tell if someone means it based on how they write or say the rest of what they need though.
You'd probably get the same response approaching them with a "we're trying to do this but couldn't make it work, could you help us ?" kind of pitch.
Except if you're actually convinced it could be your mistake, getting that tone will feel like getting played like some small kid. Most people will help you anyway and be professional, of course.
That reminds me of detective Columbo. It's cute and all when it's supposed to be done to strangers. Imagine Columbo coming to you every week with that same convoluted spiel.
Except if you're actually convinced it could be your mistake, getting that tone will feel like getting played like some small kid. Most people will help you anyway and be professional, of course.
That reminds me of detective Columbo. It's cute and all when it's supposed to be done to strangers. Imagine Columbo coming to you every week with that same convoluted spiel.
I think this works great sometimes, but I've been in plenty of situations where "I messed up" leads to being railroaded into the "boilerplate tech support" discussion tree. "Did you try doing X", "Did you confirm that Y is Z", "Make sure your VPN is working, try installing a new config", and finally "That's strange, double check all your parameters, there may be a typo in there somewhere"
My dad once took apart his motorcycle to change out a cylinder seal. He put it all back together and it wouldn't start. The mechanic at the shop heard the story, and then checked one thing: indeed my dad had forgotten to refuel it. That tech support tree is always worth checking.
Oh, man. Flashback to the time I painstakingly rebuilt the carburetor on my scooter, then spent way too long trying to figure out why it wouldn’t start.
The kill switch was set to “off.”
The kill switch was set to “off.”
He took apart a motorcycle because the gas tank was empty?
no, he forgot to re-fill the tank after disassembling.
That's cute that you think people working the bottom rung have any sort of autonomy and aren't monitored constantly for compliance with scripts, decision trees, policies, and procedures.
I'm going to try that's but will need more information about Hans Rosling to get through the identity verification ...
Or just watch his TED Talks, back when they were actually good:
https://youtu.be/hVimVzgtD6w https://www.ted.com/playlists/474/the_best_hans_rosling_talk...
https://youtu.be/hVimVzgtD6w https://www.ted.com/playlists/474/the_best_hans_rosling_talk...
You're joking, of course, but I recommend reading Factfulness if you want to learn more about the late Mr Rosling.
Blame doesn't really help solve or prevent problems. Root cause, blameless analysis with awareness and new tests and mitigation does. Also, career-wise, you won't become an IC9 by admitting to making a lot of mistakes. It's best to just solve them as fast as possible.
I find laying of blame to be the most egregious waste of time, for work as well as personal issues. People who insist on it are, by and large, not people you want to spend time or money with.
Agree. Honesty, transparency, professionalism.
Practice those first, then work on inter personal soft skills.
Don’t bullshit people by lying to them about “whose fault it is”.
Practice those first, then work on inter personal soft skills.
Don’t bullshit people by lying to them about “whose fault it is”.
I like to append ", unless I'm missing something?" in a similar manner, might be useful in situations where 'made a mistake' doesn't actually make sense :)
One habit I picked up from a previous job was that if something is negative then it's always "I", if it's good, it's the team, "we". So "I failed to meet our deadline", "We successfully delivered the project on time and within budget".
It basically helps move things forward, blame has been allocated, how do we move on from here, but this only works if you have a health work environment.
This wasn't a company policy or anything like that, it was just how we talked and supported each other.
It basically helps move things forward, blame has been allocated, how do we move on from here, but this only works if you have a health work environment.
This wasn't a company policy or anything like that, it was just how we talked and supported each other.
Did you find it hard to sell/justify your own promotions when you were asked to self-review ? In the past I've found management quick to dilute my accomplisments as "Team" and pin my flaws as "Me"(maerf0x0). Been passed over for plenty of promotions in my day despite delivering value that just ticks away and does its job year after year, and rarely needs "firefighting" ...
You can ask customer service for help politely and constructively, without disingenuously (or passive-aggressively) stating that the problem is your fault.
If you want, you can acknowledge how you tried to fix it and failed (if that's accurate). But don't say that the problem is your fault unless it is.
(There are situations in which taking blame for a situation not necessarily yours might be a convention, but mistakes of vendors when talking with the vendor aren't one of them, IMHO. For example, you might take a little heat for colleagues, when appropriate, and all the CEO you're talking with needs to hear right then is, "Sorry, I don't have that for you yet; let me get that to you later today." Not "I've been pestering Bob since Monday for the dependency." Then you can go tell Bob that you two really need to solve this in the next couple hours. And if there's a larger problem, like Bob has been overextended by a family problem, or tasking has been unclear since a tentative pivot, then work it with management in the appropriate vertices of the org chart.)
If you want, you can acknowledge how you tried to fix it and failed (if that's accurate). But don't say that the problem is your fault unless it is.
(There are situations in which taking blame for a situation not necessarily yours might be a convention, but mistakes of vendors when talking with the vendor aren't one of them, IMHO. For example, you might take a little heat for colleagues, when appropriate, and all the CEO you're talking with needs to hear right then is, "Sorry, I don't have that for you yet; let me get that to you later today." Not "I've been pestering Bob since Monday for the dependency." Then you can go tell Bob that you two really need to solve this in the next couple hours. And if there's a larger problem, like Bob has been overextended by a family problem, or tasking has been unclear since a tentative pivot, then work it with management in the appropriate vertices of the org chart.)
I'm going to be a bit pedantic and say fault and mistake are not equal in the message they convey. Hans says he has 'made a mistake' which is not the same as saying 'I'm at fault.' In the end it might be proven that you are at fault but until then all you know is that something went wrong, isn't working, etc.
I think that’s the kind of pedantry that is relevant in that case and in designing your product and customer support tree.
Friends of mine hearing him would say, He never says what the mistake is precisely, but there’s always the option that it was booking a flight with that airline.
Friends of mine hearing him would say, He never says what the mistake is precisely, but there’s always the option that it was booking a flight with that airline.
Hey thank you so much for linking the Hans Rosling interview, he seemed like a really great guy -- it led me to buying his book just now!!
I mean, that sounds very holistic in theory, but in practice just doesn't work out.
A few days ago I suddenly had my french press for coffee suddenly shatter and almost blast hot coffee over my upper body.
How am I supposed to start that call with "Hi, I'm jorvi and I made a mistake"..?
It's not like that is a unique situation either. And you can guarantee that if you tell customer service "I made a mistake", and it is clear they delivered a broken service / product (but often want to duck responsibilities), there is no way in hell they will not take the freebie you just gave them by admitting fault.
A few days ago I suddenly had my french press for coffee suddenly shatter and almost blast hot coffee over my upper body.
How am I supposed to start that call with "Hi, I'm jorvi and I made a mistake"..?
It's not like that is a unique situation either. And you can guarantee that if you tell customer service "I made a mistake", and it is clear they delivered a broken service / product (but often want to duck responsibilities), there is no way in hell they will not take the freebie you just gave them by admitting fault.
"Hi, I'm jorvi and seems I made a mistake or something, my french press just exploded in front of me! Can you help me figure out what went wrong?" is one way of putting it.
[deleted]
I agree. If a part failed or broke unexpectantly it makes no sense to say it was `your fault` or `you made a mistake`. The goal is the solve the issue at hand and following the representative's instructions to either fix it or prove the item is defective is the easiest way to get there. If it was a mistake on your end you learned something new but I don't see any additional benefit by saying you made a mistake when you don't know what went wrong.
[deleted]
"Hi Hans Rosling..."
ticks the 'potential liability' box
"How can I help you?"
ticks the 'potential liability' box
"How can I help you?"
"I've Made a Huge Mistake."
https://www.youtube.com/watch?v=GwQW3KW3DCc
https://www.youtube.com/watch?v=GwQW3KW3DCc
this is what I usually do (to skip the blame game and get to working on the solution), but REALLY be careful about WHO you tell that you made a mistake
that dale carnegie book
caminante(1)
Yeah, we discourage production changes starting first or second december week, and start freezing changes third december week until it's frozen solid fourth december week until second week of january.
December tends to be hell for our customers, so stability should be a priority there.
And honestly, no one wants to work on holidays. So lets just wrap everything starting in december, maybe use the third week for some unnoticed issues and then just lay down the tools. Use that time for documentation, or shorter days, quite frankly.
That way we minimize the on-call situations occuring. Let's hope it goes well for the engineer this year as well. We have a streak to keep.
December tends to be hell for our customers, so stability should be a priority there.
And honestly, no one wants to work on holidays. So lets just wrap everything starting in december, maybe use the third week for some unnoticed issues and then just lay down the tools. Use that time for documentation, or shorter days, quite frankly.
That way we minimize the on-call situations occuring. Let's hope it goes well for the engineer this year as well. We have a streak to keep.
The place I work for pushed v2 of their software, a full rewrite (nothing from the old system, not even databases) by a new team, into production this week for several customers. Mostly they did it so they could say they met their made up 2023 KPIs for the v2 rewrite. There was no good reason to push it out now other than that, and there were several reasons not to, such as it wasn’t well tested and it’s fucking December 20th. Anyways, I’m not really on call so I can’t complain much, but my poor coworkers have to support this over the holidays now.
Ugh. Several years ago I spent an entire Christmas vacation, including all day Christmas Day, putting out fires because a team couldn't be bothered to do five minutes of cursory load testing. As a consequence, multiple production systems went down under load.
Later, after we regrouped after a month of this brutality, they wandered around the office bragging like they'd hung the fucking moon after they fixed the crippling, obvious design issue they'd released. I confronted the dev lead with the fact that they would have seen this after 30s of load testing and he just laughed, I think he literally said "LOL". A giant middle finger, that's what Ops got from Dev for Christmas that year.
Here's to the people who KTLO. My people.
Later, after we regrouped after a month of this brutality, they wandered around the office bragging like they'd hung the fucking moon after they fixed the crippling, obvious design issue they'd released. I confronted the dev lead with the fact that they would have seen this after 30s of load testing and he just laughed, I think he literally said "LOL". A giant middle finger, that's what Ops got from Dev for Christmas that year.
Here's to the people who KTLO. My people.
I've been is similar situation before. They wanted to release right on Christmas. But luckily enough, instead of releasing version full of bugs, managers come up with excuse: release postponed for one month due to some new vulnerability in third party library project used.
What a brilliant move! Christmas's was saved, everyone eligible received their bonuses.
What a brilliant move! Christmas's was saved, everyone eligible received their bonuses.
[deleted]
We don't all do dev ops here 8)
My little firm have just lifted and shifted a customer's hardware from someone else's computer room (data centre is too grand) and plopped it down in ours. Downtime was roughly six hours which includes two hours driving, unracking, loading, unloading and racking.
Then there was a flurry of network knitting ... oh they've tagged the bloody VLAN instead of untagging it on what are effectively access ports and don't need to be trunks or hybrid. lol, lose 20 mins. I wasn't allowed to look at the "source" switch's config and might (emogi: looking up and whistling) have assumed a few things ...
We did spend quite a long time trying to work out what the customer might have failed to tell us because we hadn't asked the right questions.
... so I plug my laptop into the NIC in question on the Hyper-V box and run up Wireshark ... fuck (dot 1Q tag) ... run back upstairs to my PC and reconfigure the port to hybrid with tagged VLAN 100 instead of access on VLAN 100. A better solution would be a trunk with PVID on the naughty VLAN and tagged v100. I chose the former to make it stand out.
The naughty VLAN thing is similar to a discard VLAN but the traffic is not discarded but instead gets logged. We should never see traffic on the naughty VLAN. If we do its a miss-configuration or something nasty.
As well as that, we have customers for whom Chrimbo is anything up to 50% of annual turnover. Their systems tend to be treated in the same way as yours.
My little firm have just lifted and shifted a customer's hardware from someone else's computer room (data centre is too grand) and plopped it down in ours. Downtime was roughly six hours which includes two hours driving, unracking, loading, unloading and racking.
Then there was a flurry of network knitting ... oh they've tagged the bloody VLAN instead of untagging it on what are effectively access ports and don't need to be trunks or hybrid. lol, lose 20 mins. I wasn't allowed to look at the "source" switch's config and might (emogi: looking up and whistling) have assumed a few things ...
We did spend quite a long time trying to work out what the customer might have failed to tell us because we hadn't asked the right questions.
... so I plug my laptop into the NIC in question on the Hyper-V box and run up Wireshark ... fuck (dot 1Q tag) ... run back upstairs to my PC and reconfigure the port to hybrid with tagged VLAN 100 instead of access on VLAN 100. A better solution would be a trunk with PVID on the naughty VLAN and tagged v100. I chose the former to make it stand out.
The naughty VLAN thing is similar to a discard VLAN but the traffic is not discarded but instead gets logged. We should never see traffic on the naughty VLAN. If we do its a miss-configuration or something nasty.
As well as that, we have customers for whom Chrimbo is anything up to 50% of annual turnover. Their systems tend to be treated in the same way as yours.
Holiday oncalls are a fun tradeoff. On one hand, no one should be making any changes (and if they do, they'll have some explaining to do), so it's more likely to be calm. On the other, traffic patterns are weird, and it's time off where you'd rather not be tethered to your phone. What's universally bad is being oncall when the code freeze ends or the week leading up to the freeze.
Major Major Lesson learned in the trenches.
Business offices cut their chilled water supply back to minimums (or nothing) over holiday weekends & breaks.
If you're running a server closet, even if you have a dedicated Liebert HVAC, when the chilled water cuts.. you will overheat.
I learned this over the course of three consecutive Thanksgivings.
Business offices cut their chilled water supply back to minimums (or nothing) over holiday weekends & breaks.
If you're running a server closet, even if you have a dedicated Liebert HVAC, when the chilled water cuts.. you will overheat.
I learned this over the course of three consecutive Thanksgivings.
> And honestly, no one wants to work on holidays.
Actually I bet some people like it (I know I do). It's not that crazy to want to dodge the whole mad rush and take lots of time off later in the year when it's actually nice outside. Summer vacation beats winter vacation, so if you have to take days off in the winter there's pressure to try and get somewhere warm where the days are longer. Besides. The "office" is quiet, even if you're a telecommuter, so it's easy to get things done. If you're not touching production, that's fine, there's usually all kinds of fun or quality-of-life projects around tech debt, tooling, whatever. Lots of important work is actually easier to do during a change-freeze or other downtime.
Actually I bet some people like it (I know I do). It's not that crazy to want to dodge the whole mad rush and take lots of time off later in the year when it's actually nice outside. Summer vacation beats winter vacation, so if you have to take days off in the winter there's pressure to try and get somewhere warm where the days are longer. Besides. The "office" is quiet, even if you're a telecommuter, so it's easy to get things done. If you're not touching production, that's fine, there's usually all kinds of fun or quality-of-life projects around tech debt, tooling, whatever. Lots of important work is actually easier to do during a change-freeze or other downtime.
I liked it as well when I was single. Do you have a spouse and/or children?
Our customer demands changes for December 1 and for January 1, which sounds like a terrible idea. Fortunately, for legal reasons we don't handle deployment but they do, so it's up to them to decide when to put our changes in production.
I completely acknowledge it's utopian but isn't it a better goal to target continuous stability, or at least semi trusted process for when things inevitably break?
It's a similar concept to not deploying on Fridays. If you're afraid to introduce changes due to some arbitrary timing, perhaps it's worth focusing on the source of that uncertainty.
It's a similar concept to not deploying on Fridays. If you're afraid to introduce changes due to some arbitrary timing, perhaps it's worth focusing on the source of that uncertainty.
It's not either/or. The observation that freezes and no Friday deployments capitalize on is that the single most likely cause of production incidents is production changes.
We always should target better stability, but no matter how good your system and incident response are, if your goal is to minimize customer disruption during a certain time window, or avoid dealing with incidents on weekends, minimizing production changes is the simplest and most effective measure
We always should target better stability, but no matter how good your system and incident response are, if your goal is to minimize customer disruption during a certain time window, or avoid dealing with incidents on weekends, minimizing production changes is the simplest and most effective measure
I agree. The flipside though is a freeze also invokes a scenario that can lead to a premature release. Any blanking window forces a decision between deferring or rushing work, neither of which are ideal.
How do you avoid a sort of thundering herd problem when the freeze is over?
Technically we don't... But at least everyone's had a nice vacation so we're ready to deal with it after we're back.
I think that’s a great policy as it’s clearly intended to help people when they need it, and get people to unplug when it’s valued by their loved ones.
_However_ (that part is probably best bookmarked until Jan 2nd), it also betrays that your system is brittle and can be broken by a bad commit. Don’t do it because you want people to grind until Dec 24th at 6 pm. Do it because it’s great the rest of the year, too. I’d recommend you look into (or ask me about) feature flags, alerting, and automated roll-backs.
The short version is: there’s a meta-system on top of your release process that can tell (if you are using roll-back not features flags): - commits until xyzsdf are fine; - roll-outs starting from commit abcdef have a 2% error rate, 80% on Android; - revert to xyzsdf, send a message (low-priority, email) to the DevOps on call and the author of abcdef that it happened; - for all commits after abcdef: if there no conflicts with xyzsdf, re-try to roll them out; - if there is a conflict because they were on top or abcdef, send a message (low-priority email) to the authors that there is a conflict.
There are more sophisticated versions that can do things like, if you use feature flags, flagging Android users to use the previous version. Another way to do this is to scale who has access to abcdef gradually: say 1% every hour, and revert if you detect issues.
All those seem daunting to teams that haven’t worked like this before, but it my experience, they love it very fast.
_However_ (that part is probably best bookmarked until Jan 2nd), it also betrays that your system is brittle and can be broken by a bad commit. Don’t do it because you want people to grind until Dec 24th at 6 pm. Do it because it’s great the rest of the year, too. I’d recommend you look into (or ask me about) feature flags, alerting, and automated roll-backs.
The short version is: there’s a meta-system on top of your release process that can tell (if you are using roll-back not features flags): - commits until xyzsdf are fine; - roll-outs starting from commit abcdef have a 2% error rate, 80% on Android; - revert to xyzsdf, send a message (low-priority, email) to the DevOps on call and the author of abcdef that it happened; - for all commits after abcdef: if there no conflicts with xyzsdf, re-try to roll them out; - if there is a conflict because they were on top or abcdef, send a message (low-priority email) to the authors that there is a conflict.
There are more sophisticated versions that can do things like, if you use feature flags, flagging Android users to use the previous version. Another way to do this is to scale who has access to abcdef gradually: say 1% every hour, and revert if you detect issues.
All those seem daunting to teams that haven’t worked like this before, but it my experience, they love it very fast.
We use these systems liberally on other times of the year and no one notices, usually. If they do, downtime and interruption budgets handle this.
/However/, let me counter with the point: Just one of our customer has 8000 FTEs working with our system. During hell-time (aka, December and Christmas shopping and shipping), each of those dudes spends their shift taking customer calls lasting 2-4 minutes, which in turn require a few requests into our systems.
Due to the stress of their customers^2 (because it's Christmas and holidays and such), if an agent of a customer is unable to access our systems, they cannot handle the use case of the customer^2 and that will piss of the customer of the customer.
So if we push a bad change during this time, we're going to piss of hundreds of customers^2 per minute for that one customer alone. Even with a fast automatic rollback, that's a long time during hell-time. And they have people who know how to yell at vendors in nasty ways who don't like that.
I enjoy moving software fast and enabling moving software quickly, but customer focus and customer orientation means to understand when to move slow as well.
And hey, if that means more quiet holidays for the hard working operators on my team, who's gonna complain?
/However/, let me counter with the point: Just one of our customer has 8000 FTEs working with our system. During hell-time (aka, December and Christmas shopping and shipping), each of those dudes spends their shift taking customer calls lasting 2-4 minutes, which in turn require a few requests into our systems.
Due to the stress of their customers^2 (because it's Christmas and holidays and such), if an agent of a customer is unable to access our systems, they cannot handle the use case of the customer^2 and that will piss of the customer of the customer.
So if we push a bad change during this time, we're going to piss of hundreds of customers^2 per minute for that one customer alone. Even with a fast automatic rollback, that's a long time during hell-time. And they have people who know how to yell at vendors in nasty ways who don't like that.
I enjoy moving software fast and enabling moving software quickly, but customer focus and customer orientation means to understand when to move slow as well.
And hey, if that means more quiet holidays for the hard working operators on my team, who's gonna complain?
You are a lot more ahead than most companies.
I’ve worked for too many places where the Christmas break was because of a lack of tooling. I’m glad you are two steps ahead.
I’ve worked for too many places where the Christmas break was because of a lack of tooling. I’m glad you are two steps ahead.
As the person before mentioned, partial rollouts with separate monitoring would help with that and might be an improvement the other 11 month..
But we are doing the same thing, 2 weeks around Christmas there is please take holidays if you can period where we do not merge any non priority one tickets.. which has not happened yet.
But we are doing the same thing, 2 weeks around Christmas there is please take holidays if you can period where we do not merge any non priority one tickets.. which has not happened yet.
How do you detect errors like this?
What is an error? Is a business logic bug going to be picked up by this process automatically, or is some manual steps involved?
Ie a point of sale app releases an update that automatically halves the amount to charge, but displays the full amount to the merchant in the UI. Unit tests pass (because an engineer made a human mistake). Backend calls are correctly used, no errors thrown, simply the wrong amount is used.
How would this be automatically detected and reverted?
Would anyone writing point of sale software want to risk this over one of the biggest trading periods of the year?
What is an error? Is a business logic bug going to be picked up by this process automatically, or is some manual steps involved?
Ie a point of sale app releases an update that automatically halves the amount to charge, but displays the full amount to the merchant in the UI. Unit tests pass (because an engineer made a human mistake). Backend calls are correctly used, no errors thrown, simply the wrong amount is used.
How would this be automatically detected and reverted?
Would anyone writing point of sale software want to risk this over one of the biggest trading periods of the year?
As you point out, it really depends on what is an error. Most of the companies I know of have a Holiday freeze are video games, casual ones, even. Changes are minor fixes and optimization—glitches that a player likely won’t notice, but you want to detect them early to avoid losing your ability to detect more.
Back-end tools are different, and I definitely see reasons other than bugs to not change business logic this month.
Back-end tools are different, and I definitely see reasons other than bugs to not change business logic this month.
Yeah, that model may work for many public facing apps, but probably less so for enterprise systems that are heavy in business logic.
> it also betrays that your system is brittle and can be broken by a bad commit.
Correct. So's yours. So's everyone. You might not know what the bad commit is, you might've fixed a bunch of the other bad commits, but even Google gets taken down by bad commits. Your system is brittle and can be broken by a bad commit.
Correct. So's yours. So's everyone. You might not know what the bad commit is, you might've fixed a bunch of the other bad commits, but even Google gets taken down by bad commits. Your system is brittle and can be broken by a bad commit.
Yes, absolutely thanks to all who keep our world running when no one is looking. To keep the yule log on Youtube, to keep our christmas tree lights on, to keep a fresh glass of water from the tap, warm natural gas to keep the freezing cold outside etc. Thank you for keeping society ticking away :)
I’ll give a shout out too to everyone in the military monitoring warning systems and maintaining stance to protect us from being killed while we’re with our families.
Hundreds of thousands are fighting in the trenches right now to protect their homes, to protect their families, to protect democracy while politicians went on holiday break without approval of military aid for allied army.
[deleted]
[deleted]
You ever read something and it's just like... yeah, I'm absolutely done with interacting with this userbase forever?
Yeah
Yeah
I'm curious why you say that? I'm taking it as my comment makes you not want to interact with HN?
It was more what I perceived as a weirdly gleeful insincerity in your comment combined with a _lot_ of similarly weirdly gleeful insincerity in the big comment thread about the SMTP smuggling attack, which had a lot of 'GRRRREEEEAATT WORK BY THESE GUYS' type of comments
Just too much discordanism seemed to surface all at once and it was exhausting.
I know this is 'hacker' news but I always perceived that to mean 'hacker' as in the hacker ethos, not the pejorative 'yeah break shit lol' type of hacker. Sometimes it seems to be all the latter.
Just too much discordanism seemed to surface all at once and it was exhausting.
I know this is 'hacker' news but I always perceived that to mean 'hacker' as in the hacker ethos, not the pejorative 'yeah break shit lol' type of hacker. Sometimes it seems to be all the latter.
as someone who's been part of the community for years, I'd say your perceived version of Hacker is the correct usage here.
Let's not confuse on-call firefighters or a water facility staff with the on-call admins that maintain money-making machines monetizing attention of billions. The latter is a net negative on society.
Not a fan of the YouTube Yule Log, I see.
That firefighter is probably using YouTube or scrolling through Instagram to unwind while they’re stuck at the station waiting for a call. Just because someone works in entertainment or ads doesn’t mean that the economic puzzle piece they represent isn’t valuable to society.
Imagine instead they were making a meal for the whole fire hall, working out, or talking to each other …
[deleted]
[deleted]
Yeah, how dare Netflix provide entertainment on-demand and for cheaper than the other entertainment companies?
I am currently viewing this on ethically sourced rfc1149 (birds gave consent via a scientifically proven “brain electrode interface”), manually decoding packets using an abacus made out of various animal droppings foraged on the forest floor. If I can’t view your content this way, it should not be on the internet.
I get nervous every New Year’s Eve due to date/time issues. I work on emergency 911 software. In our system each time a 911 Call is created, we create an incident number in the format YYYY-NNNNNNNNNNNN where N is an incrementing number. I was oncall a few years ago when a date time bug was introduced that resulting in numbers being created prematurely by a few hours. As each hour passed more customers in a different time zone called in to report the issue. I was the only person working and was getting hammered with cases.
It sounds like an easy isssue to correct, but downstream systems that consume those numbers had already processed them and associated reports and other records with the incidents. I spent the next few months sorting out that mess and helping work with partners to clear out data.
It sounds like an easy isssue to correct, but downstream systems that consume those numbers had already processed them and associated reports and other records with the incidents. I spent the next few months sorting out that mess and helping work with partners to clear out data.
On call sucks so badly. At this point of my life, I firmly believe that there's not enough amount of money that can compensate the mental suffering it implies. Even more if the company you work for has this mentality of "deal with it" without making improvements, which was my case in the last period I did on call and what made the camel's back to break for me. Nowadays I simply refuse it. For those who are still on the trenches, stay strong, never resígnate yourself to just "deal with it" and thank you.
Quote From a classic:
You might be under the impression that what makes you qualified for various positions in software development is primarily your technical acumen and ability to work with other technically-capable engineers.
You’d be wrong.
While a certain minimum of capability is required to do your day-to-day work, what your value really consists of is in grinding yourself against the piercing pincers of elusive bugs and razor-wire bundles of bullshit code until something resembling progress is made. You are not a problem-solver, you are a problem-endurer.
https://web.archive.org/web/20160317234837/https://medium.co... -> Point 4
You might be under the impression that what makes you qualified for various positions in software development is primarily your technical acumen and ability to work with other technically-capable engineers.
You’d be wrong.
While a certain minimum of capability is required to do your day-to-day work, what your value really consists of is in grinding yourself against the piercing pincers of elusive bugs and razor-wire bundles of bullshit code until something resembling progress is made. You are not a problem-solver, you are a problem-endurer.
https://web.archive.org/web/20160317234837/https://medium.co... -> Point 4
To me, the worst part of being on call is the stress _after_ my shift ends. I understand that it's a necessary part of the job to fix issues that occur during my shift, so I don't really mind it, but it gives me long term issues. I feel anxious whenever I don't have my phone on me, or when I'm far enough into the wilderness to lose my cell signal. Late night when I don't expect to be getting messages from anyone, a random notification can sometimes give me an immediate stomach-drop panic response.
Unfortunately I feel like I lucked into this role and if I left I wouldn't be able to find anything anywhere near as good.
Unfortunately I feel like I lucked into this role and if I left I wouldn't be able to find anything anywhere near as good.
And I am not even sure whether you are talking about just day-time on-call or the 24 hours on-call for at least 1 full week to two week stretches or a simple 12 hours on-call you are talking about? In India the Indian managers (and American managers are just fine with it) have made an environment of this barbaric practice of 24x7 on-call handled by just one person.
In fact, even when there are US/western counterparts these subhumans projects that they will make sure Indian engineers are on-call even during American daytime. This has been happening at my workplace. They employ all tactics - from fear, intimidation, to try to sweat talk engineers into it with shit like, "Oh, we own it, right? So it's our responsibly to support even when it's night".
With that environment it becomes extremely difficult and a pressurised situation for someone like me who simply refuse to even sign up on something like PagerDuty and make it clear that my phone remains silenced and out of my bedroom between 10pm-7am and it really does.
I agree with you - there is no amount of money that can put on on-call, definitely not on a night shift on-call.
In fact, even when there are US/western counterparts these subhumans projects that they will make sure Indian engineers are on-call even during American daytime. This has been happening at my workplace. They employ all tactics - from fear, intimidation, to try to sweat talk engineers into it with shit like, "Oh, we own it, right? So it's our responsibly to support even when it's night".
With that environment it becomes extremely difficult and a pressurised situation for someone like me who simply refuse to even sign up on something like PagerDuty and make it clear that my phone remains silenced and out of my bedroom between 10pm-7am and it really does.
I agree with you - there is no amount of money that can put on on-call, definitely not on a night shift on-call.
> have made an environment of this barbaric practice of 24x7 on-call handled by just one person.
If it makes you feel any better this is very common in small to mid-sized US tech companies as well. In every team I've been on that had an oncall rotation it was a full week 24/7 per person, that rotated among team members. Even at Google we were on call for our own service overnight and didn't have SRE / other time zone oncalls.
But the number of pages and other work varied significantly between teams. The worst was risk at Square in 2016, where we routinely got paged 40+ times a week (mostly noise) and when real incidents were most likely on Saturday morning. The best was Instant Apps at Google where we got a ~$5k bonus for each week of overnight oncall and almost never got a single page.
If it makes you feel any better this is very common in small to mid-sized US tech companies as well. In every team I've been on that had an oncall rotation it was a full week 24/7 per person, that rotated among team members. Even at Google we were on call for our own service overnight and didn't have SRE / other time zone oncalls.
But the number of pages and other work varied significantly between teams. The worst was risk at Square in 2016, where we routinely got paged 40+ times a week (mostly noise) and when real incidents were most likely on Saturday morning. The best was Instant Apps at Google where we got a ~$5k bonus for each week of overnight oncall and almost never got a single page.
Why would that made me feel better, tdeck? It doesn't make me feel better.
Besides it’s different what you mentioned about where you are from and what I experience and see as the norm where I am from.
Besides it’s different what you mentioned about where you are from and what I experience and see as the norm where I am from.
With the last (and only) job that required me to be on call I quit the day before I was scheduled. I've always refused to do it. Devs have no business doing it.
I appreciate setting boundaries but I don't really understand this attitude. Frequently on call issues are caused by problems with the application logic, therefore solving them requires an understanding of the code. It's not usually my experience that oncall issues are a simple case of force-restarting something or provisioning more boxes, although that can happen from time to time.
A system that can get itself into a non-functioning state and that can't be supported by an operator or dedicated support person is fundamentally broken and should not be in production. In my view devs should never have access to production, under any circumstances, ever.
This is an artifact of devs (and others) not knowing what they're doing, and just hacking and hoping for the best. It's really not that hard to develop a system that is reliable and supportable in a basic way. Understanding the code shouldn't be a requirement, but understanding the system should, and that's a requirement of support personnel. Put another way, the functional model of the system has to be at a higher level than the code.
This is an artifact of devs (and others) not knowing what they're doing, and just hacking and hoping for the best. It's really not that hard to develop a system that is reliable and supportable in a basic way. Understanding the code shouldn't be a requirement, but understanding the system should, and that's a requirement of support personnel. Put another way, the functional model of the system has to be at a higher level than the code.
I'd argue that a software system that can be supported by a dedicated operator who isn't a developer is fundamentally broken. Any response protocol that can be handled by someone without familiarity with the system's internal workings can fundamentally be automated. Scaling hardware? Restarting boxes? Ignoring and silencing an alert? Draining traffic to a bad host? These are all fairly simple actions that could theoretically be automated, and have been automated at many companies. There shouldn't be a need for a person who can only do things like this.
On the other hand, there will be production issues caused by a complex interaction within the system that arises in an unforeseen edge case. These issues frequently require a code change which requires the ability to understand the codebase. In that case, the system is broken, but it's not "fundamentally" broken, it's broken for a particular edge case. Unfortunately, we may not have the luxury of waiting until 10 AM PST to start looking into the problem and coming up with a fix for it.
On the other hand, there will be production issues caused by a complex interaction within the system that arises in an unforeseen edge case. These issues frequently require a code change which requires the ability to understand the codebase. In that case, the system is broken, but it's not "fundamentally" broken, it's broken for a particular edge case. Unfortunately, we may not have the luxury of waiting until 10 AM PST to start looking into the problem and coming up with a fix for it.
> Devs have no business doing it.
Agree but, I have to say that, as a DevOps, it was infuriating to me to have to deal with developers without any care for the quality of what they were delivering. Sometimes for pressure from someone higher in the chain, other times, for pure laziness and/or incompetence. I remember coming in the morning after a hell of a night on the on-call, reporting the issues to the Devs in charge and being answered something along the lines of "fixing that is not the priority right now" and my replying on anger with "If it was your damn phone the one ringing during the whole night I'm pretty sure you would make it a priority".
Agree but, I have to say that, as a DevOps, it was infuriating to me to have to deal with developers without any care for the quality of what they were delivering. Sometimes for pressure from someone higher in the chain, other times, for pure laziness and/or incompetence. I remember coming in the morning after a hell of a night on the on-call, reporting the issues to the Devs in charge and being answered something along the lines of "fixing that is not the priority right now" and my replying on anger with "If it was your damn phone the one ringing during the whole night I'm pretty sure you would make it a priority".
There should be some sort of trigger whereby over a certain threshold of problems the devs have to perform the support role. It's unacceptable to deliver a shitty system and rely on support to avert disaster or user revolt, there has to be some sort of incentive to counter this.
Meanwhile a huge number of us (non-religious? introverted kernel compiling cave dwellers?) treat this period no differently than any other week in the year. I'll be here keepin the servers runnin :horns:
It's actually my favorite time of the year. Everyone is gone, it is quiet, and I can get shit done.
It's actually my favorite time of the year. Everyone is gone, it is quiet, and I can get shit done.
Holidays are special because they’re special, both the winter solstice festival (rebranded for christianity) and the spring equinox one (same deal) can be treated differently for cultural variety by the non-observant.
I’m a militant proselytizing atheist raised by a jew and I still have a tree with pretty lights, give presents, and drink and eat some things I only drink/eat once per year (never make homemade eggnog if you ever want to enjoy it guilt free again, you’re basically drinking a megacalorie of heavy cream, yum). It’s fun to celebrate the generic concept of “holiday” - a time that is different from other times.
You’re allowed to feel nice about peppermint candy (and/or chocolate gelt, I go for both) at the end of December without bringing the supernatural into the equation. :)
\m/
I’m a militant proselytizing atheist raised by a jew and I still have a tree with pretty lights, give presents, and drink and eat some things I only drink/eat once per year (never make homemade eggnog if you ever want to enjoy it guilt free again, you’re basically drinking a megacalorie of heavy cream, yum). It’s fun to celebrate the generic concept of “holiday” - a time that is different from other times.
You’re allowed to feel nice about peppermint candy (and/or chocolate gelt, I go for both) at the end of December without bringing the supernatural into the equation. :)
\m/
> never make homemade eggnog if you ever want to enjoy it guilt free again, you’re basically drinking a megacalorie of heavy cream, yum
Solid advice. I literally put on 5 lbs. while refining my non-alcoholic eggnog recipe.
One thing I noticed is that ice cream, crème brûlée, and non-alcoholic eggnog are all just variations on the same recipe. A glass of egg nog is pretty similar to a glass of melted ice cream.
Solid advice. I literally put on 5 lbs. while refining my non-alcoholic eggnog recipe.
One thing I noticed is that ice cream, crème brûlée, and non-alcoholic eggnog are all just variations on the same recipe. A glass of egg nog is pretty similar to a glass of melted ice cream.
Oh ya same I love the smell of evergreen wreaths and trees and enjoy partaking in festive activities. A 4K cracklin’ Yule log goes a long way too.
Certainly doesn't have to be religious. I think you probably have all your family living near each other or don't like them. To me, this is the only time of year outside of major events like weddings, funerals, and graduations that I can be reasonably assured my parents, all my sisters, all of my nieces and nephews, and at least a few aunts, uncles, and cousins will all be in the same place at the same time. It's both nice and convenient to be able to travel to one place and see all of them together. It's the kind of thing that can only realistically happen at a coordinated national level, and if took a religious holiday to give the country an excuse to give us all a holiday at the same time, I'm fine with that even if I don't practice the religion.
> I think you probably have all your family living near each other or don't like them
Bingo!
Bingo!
Just like San Francisco during Burning Man!
Build A 300-Mile Wall Around SF During Burning Man:
https://web.archive.org/web/20190213021206/https://megagogo....
>A community effort to construct a 300-mile wall in one week and prevent Burning Man attendees from returning to the Bay Area.
>About This Project
>We want to help Burning Man attendees continue their favorite week of the year, and allow them to keep experiencing the genuine community and deep connections they can only feel while at Burning Man. To do this, we will build a 300-mile wall around the entire Bay Area during Burning Man.
>For the rest of us, what’s normally our favorite week of the year… lasts forever!
Build A 300-Mile Wall Around SF During Burning Man:
https://web.archive.org/web/20190213021206/https://megagogo....
>A community effort to construct a 300-mile wall in one week and prevent Burning Man attendees from returning to the Bay Area.
>About This Project
>We want to help Burning Man attendees continue their favorite week of the year, and allow them to keep experiencing the genuine community and deep connections they can only feel while at Burning Man. To do this, we will build a 300-mile wall around the entire Bay Area during Burning Man.
>For the rest of us, what’s normally our favorite week of the year… lasts forever!
When I did shift work I always volunteered for the holidays because I don't care about holidays.
I’m surprised more people don’t know this.
I love end of year because nobody’s pushing anything or needs help.
Then you take vacation in January when the floodgates open.
I love end of year because nobody’s pushing anything or needs help.
Then you take vacation in January when the floodgates open.
People know it but they're eager beavers.
The right way: slow way the fuck down to the point you're practically on vacation during the holidays without taking vacation. Take your vacation during other times in the year.
The right way: slow way the fuck down to the point you're practically on vacation during the holidays without taking vacation. Take your vacation during other times in the year.
I do partake in Christmas stuff now because my family appreciates it. But I used to love the quiet around Christmas. One year also I just completely forgot my own birthday. It was a good day (I started editing OpenStreetMap on that day).
You may have something special to do this holiday then :)
https://news.ycombinator.com/item?id=38727987
https://news.ycombinator.com/item?id=38727987
100% - as a fairly staunch atheist though I do enjoy the small pause in reality to catch up on life admin, volunteer, or even just sit in the sun and drink beer.
[deleted]
My D-I-L is a nurse and will be working the coming weekend. That's her "price" for getting Thanksgiving (in the US) off. We'll schedule as much as possible around her work hours and make sure there's food left for her when she gets home.
Many thanks to all of the health care workers who take care of us over the holidays. (Along with all of the others, of course.)
Many thanks to all of the health care workers who take care of us over the holidays. (Along with all of the others, of course.)
My mum always worked on most of the public holidays because we never really cared about them but it did mean everyone owed her favors (she was a physician and would be on call, at the hospital etc) so she could get someone to cover for when whenever she wanted.
Growing up ignoring holidays is mostly great (fly on xmas and everybody feels sorry for you, even though they are the ones working on xmas). But it causes relationship problems bc even when you genuinely try to participate you’re “doing it wrong”.
Growing up ignoring holidays is mostly great (fly on xmas and everybody feels sorry for you, even though they are the ones working on xmas). But it causes relationship problems bc even when you genuinely try to participate you’re “doing it wrong”.
Usually I choose to work during normal holiday season too, and then go on vacation in mid January when crowds are smaller.
Having a family that accepts rescheduling Holidays helps. We've celebrated Thanksgiving, New Year and Christmas on different days before.
Having a family that accepts rescheduling Holidays helps. We've celebrated Thanksgiving, New Year and Christmas on different days before.
And for all those of you trying to hold the world on your shoulders: don't be a hero. If you don't let things fail (that aren't your responsibility), nobody will notice it's at risk of failing, and thus will keep letting you hold it up by yourself.
Is it a US thing to push updates right before holidays and force people to be on call?
European companies I've been working on are planing releases till the end of November, first week of December max. All project plans are baked like there is one week in December.
While US company tried to force every contractors to work everyday and some weekends too. Are you Indian? You don't celebrate Christmas, you'll be on call. Are you Eastern Slavic? You celebrate Orthodox Christmas in January. You'll be on call.
European companies I've been working on are planing releases till the end of November, first week of December max. All project plans are baked like there is one week in December.
While US company tried to force every contractors to work everyday and some weekends too. Are you Indian? You don't celebrate Christmas, you'll be on call. Are you Eastern Slavic? You celebrate Orthodox Christmas in January. You'll be on call.
Not all breakage happens because of new updates. Sometimes the servers go down, sometimes a bug that already existed gets triggered by something, sometimes there's extra volume over the holidays and the load is too much. Full agreement that you shouldn't push new code in late December, but that won't stop the need for some folks to be on call.
[deleted]
We discourage changes starting early/mid november and completely freeze deploys a week before thanksgiving and christmas, but there's still so much that can go wrong. The other commenter summarized it well, but in addition there's also systems that require continuous automatic updates as part of their functionality, and sometimes those automatic updates can start to go haywire. For some systems, the holidays are the most important part of the year, and it's extra crucial for people to be monitoring and ready to respond at a moment's notice.
There are also systems that aren’t entirely stable due to resource leaks, and when you stop deploying you might find you need to restart them occasionally.
> Is it a US thing to push updates right before holidays and force people to be on call?
Yes. And usually they get it done via their off-shore employees in third world countries like India working during their night time.
> Are you Indian? You don't celebrate Christmas, you'll be on call.
Not celebrating Christmas isn't a point. The point usually is - you will be without a job probably if you are not agreeing to be on-call 24x7.
Yes. And usually they get it done via their off-shore employees in third world countries like India working during their night time.
> Are you Indian? You don't celebrate Christmas, you'll be on call.
Not celebrating Christmas isn't a point. The point usually is - you will be without a job probably if you are not agreeing to be on-call 24x7.
SRE teams are Oncall 24/7/365
My uncle used to work for the City. Every holiday season, if it snowed, he would be called away to plow the roads.
Here's to those out there plowing the roads so we can get there safe!!
Here's to those out there plowing the roads so we can get there safe!!
FYI Israelis are not on holiday - our holidays are on whole different dates. Hire Israelis and experience no down time while working with Silicon Valley level talent
Just hope stuff doesn’t break on sabbath, they ain’t touching no computer that day :)
Most Israelis in tech are not religious and take on call duties on Saturday (Sabbath) just like Christians take on call duties on Sundays.
I see they also have no sense of humour.
They'd need a goy on-call to flick the switch.
As I wrote in another comment, most Israelis in tech are not religious and take on call duties during Saturdays and religious holidays
Then why was it a common response after the recent attack that the poor military and police response was due to it being a holiday?
[deleted]
Lolol...
Every Saturday they will convince you that they have a list..... When it actually just a calendar matrix..and steal your land
Every Saturday they will convince you that they have a list..... When it actually just a calendar matrix..and steal your land
True story
Of course, then you're complicit in funding the genocide of Palestinians. But might be worth it.
I announced a downtime for a smallish GPU Cluster starting from christmas eve just a few hours ago. It is just the perfect time to schedule a day or two of downtime for a system like that. And if IPMI doesn't fail me, I can get a lot of things done without leaving the comfort of my home. I scheduled this without pressure from my boss. It was a totally voluntary decision... While being raised as a Christian, this time of the year is for me more about solstice then about the Christian clelbration. A time to enjoy the comfort of a heated home. A time to celebrate that the days are going to be longer from now on again. A time to reflect on the past year. And all of this is easily done while having a few terminals open and waiting for remote stuff to complete...
Holidays are excellent times for hackers to take advantage. It’s not just Christmas or other Western holidays, either.
Extend this principle to any holiday/world conflict/anniversary of conflict made into holiday/calendar new year and then adjust your time of attack.
protip: US companies with offshore groups are usually underfunded, understaffed, and underskilled. Time to see if that disaster recovery environment works!
Happy holidays to those who encounter system stress tests. Can’t spell salary without some elements of slavery…
protip: US companies with offshore groups are usually underfunded, understaffed, and underskilled. Time to see if that disaster recovery environment works!
Happy holidays to those who encounter system stress tests. Can’t spell salary without some elements of slavery…
Some people just wake up in the morning wanting to fight. If you're providing support, you might just be the first person they find on a given day.
It's crazy to want to fight someone who is trying to help you, but lots of people end up in crazed states when hammering away at projects behind their computer terminals.
Taking a step back even further, I think that requiring any form of software support or correspondence with software makers is a completely alien concept. I have done it under 5 times in more than 30 years of using computer software and writing documentation.
No doubt that countless people encounter show stopping software issues for which they cannot find alternative tools, workarounds, or workflows, but I find that it's almost always better to just go shopping at these junctures, rather than enter into some kind of chain of correspondence hoping that someone out there in the ether might possibly one day fix your problem.
It's crazy to want to fight someone who is trying to help you, but lots of people end up in crazed states when hammering away at projects behind their computer terminals.
Taking a step back even further, I think that requiring any form of software support or correspondence with software makers is a completely alien concept. I have done it under 5 times in more than 30 years of using computer software and writing documentation.
No doubt that countless people encounter show stopping software issues for which they cannot find alternative tools, workarounds, or workflows, but I find that it's almost always better to just go shopping at these junctures, rather than enter into some kind of chain of correspondence hoping that someone out there in the ether might possibly one day fix your problem.
In a similar vein, I'm grateful for the people who maintain the foundational pieces of our digital world that often go unnoticed like date & time systems.
I worked four years in the military, and three of those I had evening and night shifts during the holidays.
Absolutely nothing happened, no activity whatsoever - just babysitting systems deep inside a bunker. Closest I got to new year's eve was watching the fireworks through CCTV.
The last year I was on call, which was miles better, but those years definitely cemented my will to get a job with normal work hours.
Absolutely nothing happened, no activity whatsoever - just babysitting systems deep inside a bunker. Closest I got to new year's eve was watching the fireworks through CCTV.
The last year I was on call, which was miles better, but those years definitely cemented my will to get a job with normal work hours.
Just barely started my current job too recently to be in the on-call rotation yet. Lucked out! Props to those keeping the wheels turning.
I remember playing Unreal for 3 days solid while working an IT service desk in the late '90s. Compaq Deskpro in software mode later upgraded to a Matrox G200 which upended my world. Working Christmas and New Year was a bonus as far as I was concerned. Dodged the family drama and got in plenty of gaming. Also tidied the office...
I'm not sure if there is something in the water this year, but this week, Dec 18th to Dec 21st (only a partial week), has been our busiest week all time already.
Sweating over here trying to make it through the week and praying that it slows at least for the first half of next week.
Sweating over here trying to make it through the week and praying that it slows at least for the first half of next week.
Thank you indeed! While I have fond memories of working holiday pager support early in my career, especially before marriage and kids to cover for those with families, I’m very grateful for those able to cover for all of us now! Cheers to you all
On call till 31st, so please don't hit refresh too much this days ;-)
Me too, but they pay a few bucks an hour to carry the phone so at least that adds up.
Ultimate trick is to have a diverse team. Someone that doesn’t care about Christmas but absolutely needs some random day off in March (cool with us!). Someone that celebrates new years some other time.
Ultimate trick is to have a diverse team. Someone that doesn’t care about Christmas but absolutely needs some random day off in March (cool with us!). Someone that celebrates new years some other time.
Speaking of diversity, if you don’t do Christmas dinner I strongly recommend ordering takeout from a Chinese place on the 25th! There’s lots of happy photos of Chinese chefs and Jewish customers doing a Christmas fist-bump.
Just remember this time of year is often peak vulnerability time. When attackers exploit that teams are at reduced strength and off guard. Slower response times to investigate and fix issues etc.
parent company got hit by cyberattack yesterday. salute to all IT and InfoSec colleagues working around the clock while everyone else is taking the day off.
[deleted]
Managers, if you're reading this, and you have engineers/developers "on call" but not contractually (off book), make it so, because it slightly sucks when you're having Christmas drinks but can't enjoy yourself because you might need to drive somewhere and climb up a ladder to tend to a product.
If somebody asked me to be on call, during a holiday, without overtime, there's no way I'm driving anywhere. If it's worth it to the business, pay me for the time I'm no longer getting to enjoy with my family, or shove it.
Yes, would be nice to not need to mentally prepare to make that call.
Big up the on call heroes! Hope you're getting paid well, hope you get no red lights on the bug hotlines.
My wakeup alarm this morning was 9am when OpsGenie let me know I'm on-call today. Praying for peace.
Related:
System Administrator Appreciation Day
https://www.google.com/search?q=system+administrator+appreci...
System Administrator Appreciation Day
https://www.google.com/search?q=system+administrator+appreci...
Firefighting sucks. I salute all of the ways people ensure that on-call runbooks are thoroughly documented, not shipping code on Fridays, and robust, simplicity-based engineering that reduces problems during and especially afterhours.
It tends to be a tradition for me to be oncall over at least one of the winter holidays, this year I (again) get to preside over a vendor vs vendor showdown where the only loser is our hardware!
[deleted]
non-internet-connected data center
The lizard people do not fuck around. And they don't care about your stupid holidays either.
If you’re atheist or of a religion that isn’t relevant then why do you care? There’s no spiritual meaning. It’s just a day off. Get off the fence and own your convictions.
Winter holiday celebrations & festivals predate the major world religions. Don't presume there's no meaning for people who don't follow a particular religion.
As if this week had attempted to take a measure of blood from my body, I'll be on-call next week. Looking forward to all things quiet on the HEP network front.
For some of us, we look forward to the peace and quiet
Two live video productions, including one on the evening of 31st. I managed to push back on last minute infra/workflow changes (:
Thanks for the salute, but we also accept cash :)
I salute those in the startup world - the ones in a team of 5 and they're the only Ops person who always gets paged.
Been there, done that.
Been there, done that.
Thanks for all the great work; I hope no one has an outage this holiday and has time to enjoy family and alone.
Keep up the good work, folks
Keep up the good work, folks
I'm on call 12 hours a day and hoping things are very quiet next week. Best wishes to everyone else too!
I wouldn't mind honestly. Seems like a good excuse to skip the social obligations.
We've been on freeze for weeks now in preparation for the holiday season.
> Thank you for keeping systems available and safe.
theres that word "safe" again. What systems are dangerous otherwise? Do you mean like traffic lights or something? The API serving ads to your mobile game isn't dangerous.
theres that word "safe" again. What systems are dangerous otherwise? Do you mean like traffic lights or something? The API serving ads to your mobile game isn't dangerous.
'Safe' in the context of systems can mean hacking attempts, safe from data leaks and other emergencies relative to the system that may arise. It can refer to things that are dangerous for the system itself.
[deleted]
Thanks, been there many seasons, and same to you all.
Holiday workers are the true heroes.
Thanks. Honestly, all I do is check emails once or twice a day and maybe respond to a DM if there's an emergency, but it's nice to be appreciated.
Agreed, there are some gigs that just really require support to exist - I know this first-hand from working at a Zoo (very large exotic animal rescue basically). Animals do not take holidays. They need to eat and do animal things in spite of our costumes that day.
On the flip side, having worked Cinema on Christmas Day two years I think, there is no amount of Grace and Patience I can give that is enough to those earning their living. Still have a hat and polo. Why? I had to buy them!
On the flip side, having worked Cinema on Christmas Day two years I think, there is no amount of Grace and Patience I can give that is enough to those earning their living. Still have a hat and polo. Why? I had to buy them!
Thank you but I wish you were not doing it during your night time. Let someone in their timezone for whom it's daytime do it. Insist on it, refuse being exploited. Take this opportunity to tell your company to hire the time timezones where it's day during your night. If it's not important enough for them then they anyway don't need to you remain awake at night. Don't harm yourself. It's not just losing sleep, that leads to harmful effects in your body that keep showing for long and some damages might not even be reversible.
It's not bravery, it's not being a hero as many are putting in comments - it's just plain exploitation if you are awake at night doing this.
Your health and wellbeing is not worth any money and definitely not worth someone's shopping cart and checkout page working smoothy over the holidays when your Western colleagues just took off en masse. Not even close. I know a lot of you would be from a third world country like me (yes, that's a thing!). Stay strong and work for this exploitation to end.
It's not bravery, it's not being a hero as many are putting in comments - it's just plain exploitation if you are awake at night doing this.
Your health and wellbeing is not worth any money and definitely not worth someone's shopping cart and checkout page working smoothy over the holidays when your Western colleagues just took off en masse. Not even close. I know a lot of you would be from a third world country like me (yes, that's a thing!). Stay strong and work for this exploitation to end.