100%, and we have all of those things. Canary acts as the last line of defence, and honestly, when Canary detects and rolls back, it is already an incident that is being auto-mitigated with a limited blast radius.
To reduce the potential blast radius, we are working on a cohort-based canary, which will allow us to validate against a minimal, stable subset of traffic with the desired properties.
TLDR:
A DNS automation bug removed all the IP addresses for the regional endpoints. The tooling that was supposed to help with recovery depends on the system it needed to recover. That’s a classic “we deleted prod” failure mode at AWS scale.
It broke. I have 2 BoA accounts and some time ago it made 1 of them inactive and I just couldn’t make them both work. The one I re add becomes the one Mint sees and another one becomes inactive.
It also miscalculated cash flow. Graph shows negative when in reality it is positive.
To reduce the potential blast radius, we are working on a cohort-based canary, which will allow us to validate against a minimal, stable subset of traffic with the desired properties.