Debugging (2003)(airs.com)
airs.com
Debugging (2003)
https://airs.com/ian/essays/debug/debug.html
5 comments
If the bug resulted from a rollout, you might be able to undo it. Or it might be a data problem, or it might be the result of factors out of your control.
If the code in PROD was fine before the rollout, and there is a bug in the newly deployed code; then I was saying above to undo that deployment going back to the previous version then replicate the bug, preferably locally.
It seems like there a misunderstanding about what I wrote or there is a misunderstanding about how kube works and the misunderstanding isn't mine.
It seems like there a misunderstanding about what I wrote or there is a misunderstanding about how kube works and the misunderstanding isn't mine.
> If the code in PROD was fine before the rollout, and there is a bug in the newly deployed code
Yes, if. My point is that that's a distinct subset of the possibilities, and you might not know whether it's true. Maybe you roll out a change at 10:00 and at 10:20 you start getting reports that certain key functionality isn't working. So you run your rollback... and now instead of 20% of users complaining, it's 100%, because that last update included a database migration. Okay, maybe you're very good about your data model so that doesn't happen, but you roll back... and nothing changes. It turns out that the problem was caused by a 3rd-party API you consume falling over. With the right guardrails, rolling back can often be an early option to improve the situation, but it's not a 100% cure-all.
Yes, if. My point is that that's a distinct subset of the possibilities, and you might not know whether it's true. Maybe you roll out a change at 10:00 and at 10:20 you start getting reports that certain key functionality isn't working. So you run your rollback... and now instead of 20% of users complaining, it's 100%, because that last update included a database migration. Okay, maybe you're very good about your data model so that doesn't happen, but you roll back... and nothing changes. It turns out that the problem was caused by a 3rd-party API you consume falling over. With the right guardrails, rolling back can often be an early option to improve the situation, but it's not a 100% cure-all.
This is a great read
If it can't be replicated, loop back to the reporter and seek any clarifying details. If that fails, then communicate to management the extremely high risk of attempting a fix.