Ask HN: If everyone at your company vanished how long would the lights stay on?
First: No, I do not work at Twitter.
Second: I think about his question from time to time and wonder what it is like elsewhere.
30 comments
I work at Amazon. The internet would be in trouble. I know very little about the fulfillment side of things, but I would expect things to stop there pretty much immediately. There is of course automation, robots, etc. but also many people in the loop and once they vanish I'd expect things to enter some kind of passive failure mode. Hopefully these failures would be isolated from the rest of the services.
The more interesting question is how long AWS and retail services would continue running and how they would fail. I don't think anyone could say. All people worldwide vanishing in an instant isn't a realistic scenario to plan for—if it happens, then the world has bigger problems and in any case, who would be around to execute the recovery plan?
Still for the sake of the thought experiment...I'm assuming in this scenario the world sees fit to keep supplying the data centers with power and other utilities, but no one has access to the buildings, tries to get in, bomb them etc. In this case, I would expect things to fail gradually, then suddenly and catastrophically, in the period of a few days to a few months. Failure should happen mostly independently by availability zone.
The more interesting question is how long AWS and retail services would continue running and how they would fail. I don't think anyone could say. All people worldwide vanishing in an instant isn't a realistic scenario to plan for—if it happens, then the world has bigger problems and in any case, who would be around to execute the recovery plan?
Still for the sake of the thought experiment...I'm assuming in this scenario the world sees fit to keep supplying the data centers with power and other utilities, but no one has access to the buildings, tries to get in, bomb them etc. In this case, I would expect things to fail gradually, then suddenly and catastrophically, in the period of a few days to a few months. Failure should happen mostly independently by availability zone.
Well I pride myself by creating self healing, easy to operate kubernetes clusters. Soooo as long as the hoster provides me VM's or Nodes and would not kick me out because of the invoice, i recon until a critical system update is needed.
So maybe 2-5 years? :D
with k3s the cluster iself can autoupdate and with the right settings it can update the system too.
PS: running does not mean its secure, just means that it wont die right away :D
So maybe 2-5 years? :D
with k3s the cluster iself can autoupdate and with the right settings it can update the system too.
PS: running does not mean its secure, just means that it wont die right away :D
I'd expect between a week and a month of operation before one of our central systems clogs. After that, many services would still respond to requests but provide stale data. Further two months I'd assume billing issues would bring most everything down.
Probably not for long considering the electricity will be cut off if it's not paid. None of the services run etc. are located within the company building, so presumably most of those things would run just fine indefinitely.
I work in oil and gas production. It's actually a very interesting question. We could possibly operate without a single bit of human input for a couple of weeks if everything goes well. I'm not sure how much electrical grids are reliant on human input, but there's no reason our gas supply would necessarily disappear immediately.
It really makes me think that apocalypse / societal collapse scenarios are likely to be more survivable than we think, particularly in Year 1.
It really makes me think that apocalypse / societal collapse scenarios are likely to be more survivable than we think, particularly in Year 1.
Well, production, shipping would stop. Servers/services would likely stay running as long as power was on.
My building uses motion-controlled lights. I think they switch themselves off after about 15 minutes of no motion... so about 15 minutes.
Servers run until vendor contracts get terminated.
But actual lights? In shared office bldgs it actually depends on the renter's requests to the cleaning service. I've worked many places where the crew would turn the light on/off when entering/leaving (though would be shocked seeing me working in the dark); but I've worked a place or two where they requested all lights stay on all night, because "when mgmt shows up early in the morning they can't find the light switches next to the door" (made up quote but you get it)
But actual lights? In shared office bldgs it actually depends on the renter's requests to the cleaning service. I've worked many places where the crew would turn the light on/off when entering/leaving (though would be shocked seeing me working in the dark); but I've worked a place or two where they requested all lights stay on all night, because "when mgmt shows up early in the morning they can't find the light switches next to the door" (made up quote but you get it)
ive often wondered toungue in cheek if google data centers are fullof dust,cobwebs,mummified remains, and a rackfarm humming away keeping the [lights] on, according to a cost-benefit curve analysis, labour efficiency algorithm. /s
I've been in a couple "lights out" datacenters and there always seem to be electrical/cooling/backups engineers in and around running regular checks. Which is great! Also services for cleaning public spaces.
Come on, there are very few mummified remains at most Google data centers. I did the last count, and it was barely 40%.
Answering from the cloud perspective.
While I have seen some cloud infrastructures running on life support with minimal supervision for years, AWS tends to shut down, deprecate or migrate backing services every now and then.
ASGs stop autoscaling. RDS forces version upgrades. EBS volumes go poof. Kube APIs vanish. Upstream AMIs, docker images or apt packages get moved to different repo URLs. Sooner or later some auto upgrade or scaling group would go awry and take out a persistent data store or cause services to fail.
There’s also the possibility for an unpatched vulnerability to get exploited.
I give it about 3-6 months, depending on the architecture pattern.
While I have seen some cloud infrastructures running on life support with minimal supervision for years, AWS tends to shut down, deprecate or migrate backing services every now and then.
ASGs stop autoscaling. RDS forces version upgrades. EBS volumes go poof. Kube APIs vanish. Upstream AMIs, docker images or apt packages get moved to different repo URLs. Sooner or later some auto upgrade or scaling group would go awry and take out a persistent data store or cause services to fail.
There’s also the possibility for an unpatched vulnerability to get exploited.
I give it about 3-6 months, depending on the architecture pattern.
I would add to that question: If every datacenter was power-cycled at the same time and employees were not allowed to log in, how long until customers could use your site? i.e. The Ultimate Chaos Monkey Test
Lights? Where we're going we don't need lights...
https://en.wikipedia.org/wiki/Lights_out_(manufacturing)
https://en.wikipedia.org/wiki/Lights_out_(manufacturing)
> FANUC, a Japanese robotics company, has been operating as a lights-out factory since 2001. Robots are building other robots at a rate of about 50 per 24-hour shift and can run unsupervised for as long as 30 days at a time. "Not only is it lights-out," says Fanuc vice president Gary Zywiol, "we turn off the air conditioning and heat too."
Enslave them, treat them badly and leave them alone for weeks at a time with the express intent of increasing their own numbers. What could possibly go wrong.
Enslave them, treat them badly and leave them alone for weeks at a time with the express intent of increasing their own numbers. What could possibly go wrong.
And now I want to watch Event Horizon
Something goes wrong on business day 1 without anybody to respond to customer issues and touch the database.
They'd stay on until someone turned them off, we don't have timers on them.
Oh, you didn't mean the actual building lights ...
Oh, you didn't mean the actual building lights ...
Our landlord is pretty lax, so maybe 3-4 months. If our users get creative and work together they could probably keep the thing running indefinitely. Our website will work for 10 years, but it's just static and informational.
Forever, or at least until the credit cards expire and our hosting/domain payment fails.
Our users are all internal, so I expect it'd last a long while given no inbound requests.
The data sources we use do change format from time to time, so I assume one of those would break eventually.
The data sources we use do change format from time to time, so I assume one of those would break eventually.
The big thing would be disk failures followed by cert expiration. A handful of months, tops
I'm the only one here, and it's a simple content website.
The domain has just been renewed for 5 years. The VPS is paid by credit card, and is set to auto-renew. This should last until the card expires in early 2027. There's money in that bank account so it should be fine.
Then Cloudflare will serve the website for a bit longer, probably a month or two.
After that, only archive.org copies of the website will remain, but at least the pages are self-contained (by design) and will run just fine when served from an archive.
So until about 2027.
There would be issues though. Information would grow outdated, links would break, mailboxes would get filled, etc.
The tax office would probably cause problems if I stop doing bookkeeping though. My tax advisor could delay their wrath, but eventually they'd freeze my account and payments to the web host would stop. I give it a few months, tops.
The domain has just been renewed for 5 years. The VPS is paid by credit card, and is set to auto-renew. This should last until the card expires in early 2027. There's money in that bank account so it should be fine.
Then Cloudflare will serve the website for a bit longer, probably a month or two.
After that, only archive.org copies of the website will remain, but at least the pages are self-contained (by design) and will run just fine when served from an archive.
So until about 2027.
There would be issues though. Information would grow outdated, links would break, mailboxes would get filled, etc.
The tax office would probably cause problems if I stop doing bookkeeping though. My tax advisor could delay their wrath, but eventually they'd freeze my account and payments to the web host would stop. I give it a few months, tops.
It's a radioluminescent paint company, so about six and a half thousand years.