Peiter “Mudge” Zatko says that Twitter lacked plans for cold-boot(documentcloud.org)
documentcloud.org
Peiter “Mudge” Zatko says that Twitter lacked plans for cold-boot
https://www.documentcloud.org/documents/22186965-twitter-whistleblower-disclosure
3 comments
It the power generating world, this is called a Black Start condition, and plans exist to handle it. Given the nature of the "profit uber alles" atmosphere of the previous management, it's not really surprising that this is the case.
They should at least give everyone in the core engineering staff a burner phone and a printed (on paper, laminated) list of all the others, for use ONLY in the event of a blackout. Also perhaps, they should install a Starlink terminal for each data center, unconnected to all the other networking, with a stack of USB sticks for copying boot code?
They should at least give everyone in the core engineering staff a burner phone and a printed (on paper, laminated) list of all the others, for use ONLY in the event of a blackout. Also perhaps, they should install a Starlink terminal for each data center, unconnected to all the other networking, with a stack of USB sticks for copying boot code?
When they had all the domain knowledge of how it was setup I would imagine they could get it running again.
Probably not now though
Probably not now though
56. Cascading data center problems: In or around the spring of 2021, Twitter's primary data center began to experience problems from a runaway engineering process, requiring the company to move operations to other systems outside of this datacenter. But, the other systems could not handle these rapid changes and also began experiencing problems. Engineers flagged the catastrophic danger that all the data centers might go offline simultaneously. A couple months earlier in February, Mudge had flagged this precise risk to the Board because Twitter data centers were fragile, and Twitter lacked plans and processes to "cold boot." That meant that if all the centers went offline simultaneously, even briefly, Twitter was unsure if they could bring the service back up. Downtime estimates ranged from weeks of round-the-clock work, to permanent irreparable failure.