Six hours before a trans-Pacific flight on Friday afternoon my key production database started experiencing high latency (PostgreSQL on Heroku). Had recently installed an add-on to sync data from the DB to Fivetran. Three most experienced engineers including myself paired to remove the add-on to ensure the weekend was drama-free, and instead deleted the entire database - as a result of a UX issue within the Heroku console.
Recovery took 30 minutes (Through Heroku support as Heroku did not allow backups-via-replication outside their own backup system), but that was a very long 30 minutes.
Second worst was a cleanup system that removed old CloudFormation stacks automatically by only retaining the latest version of a specific stack. Deployed a canary version of an edge (Nginx+Varnish) stack for internal testing. Cleanup script helpfully removed the production stack entirely.
Recovery took 30 minutes (Through Heroku support as Heroku did not allow backups-via-replication outside their own backup system), but that was a very long 30 minutes.
Second worst was a cleanup system that removed old CloudFormation stacks automatically by only retaining the latest version of a specific stack. Deployed a canary version of an edge (Nginx+Varnish) stack for internal testing. Cleanup script helpfully removed the production stack entirely.