When Feature Flags Go Wrong (2016)(infoq.com)
infoq.com
When Feature Flags Go Wrong (2016)
https://www.infoq.com/articles/feature-flags-gone-wrong
5 comments
Could you tell us more about how you did your auditing? Did you add more logging?
Yap it was done using the logging facility with a separate rotation, log forwarding rules, and such.
Once worked on a project where the feature flagging utility code was named FeatureMasking.IsSuppressed("feature name"), which would lead one to wonder whether it was the feature or the masking which was suppressed.
Worse, rather than cleanly blocking off full sections of code, it was common to find feature flag checking embedded within multiline expressions like
if (a>b) && c.IsNotEmpty && FeatureMasking.IsSuppressed("whatever") && d <= e ...
Worse, rather than cleanly blocking off full sections of code, it was common to find feature flag checking embedded within multiline expressions like
if (a>b) && c.IsNotEmpty && FeatureMasking.IsSuppressed("whatever") && d <= e ...
Sure, in practice not all features interact (so you can be sure on module A in the backend is orthogonal to changing color of the UI buttons), but sometimes you can't be sure and feature combinations and interactions never tested will appear in production and break unexpectedly.
One common thing I noticed is that feature A and feature B are both tested by toggling one on, doing a test, toggling it off. The test passes, everything looks good, product is shipped. Customer then turns A and B feature on and product goes up in flames.
I like the idea of auditing configuration changes (not just features others config options as well). I had implemented that across a few products and it saved the day during debugging. For example we noticed customer has been toggling feature combinations we never tested, or they reported "product stopped working" but looking at the they've been tweaking the hell out of it, which made it stop working as we had configured it.