> a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally
From a security perspective, I’m blown away that a single person apparently had the technical permissions to do such a thing. I can’t think of any valid reason that a single person would have the ability to disconnect every single data center globally. The fact that such functionality exists seems like a massive foot-gun.
At a minimum I would expect multiple layers of approval, or perhaps regionalized permissions, so that even if this person did run an incorrect command, the system turns around and says “ok we’ll shut down the US data centers but you’re not allowed to issue this command for the EU data centers, so those stay up”.
From a security perspective, I’m blown away that a single person apparently had the technical permissions to do such a thing. I can’t think of any valid reason that a single person would have the ability to disconnect every single data center globally. The fact that such functionality exists seems like a massive foot-gun.
At a minimum I would expect multiple layers of approval, or perhaps regionalized permissions, so that even if this person did run an incorrect command, the system turns around and says “ok we’ll shut down the US data centers but you’re not allowed to issue this command for the EU data centers, so those stay up”.