Working with feature flags

Once upon a time, you start with feature flags

The last 3 years I've been working with feature flagging to deliver software to end customers that you can toggle on and off. The kind of software we've been toggling is web apps and backend API's. One of the reasons why we're using feature flags is to be able to have a workflow that allows us to continously delivery code into the master branch that eventually will be deployed into production. This "eventually" means that code could reach production within 2 hours or it could take a few days. It all depends on the current stories we're working on, and how long they take to develop. The important part is that we don't know for sure when the code reaches production, so it's important that the functionality sometimes is toggled off.

So feature flagging is essentially

A way for us to continually integrate features into the master branch, without activating them in our test, uat nor prod environment, unless it's tested and verified. (Our QA person activates the feature flag in uat for example, before starting to test the feature). In a nutshell, our workflow is the following;

1. Create feature flag in LaunchDarkly.
2. Build the feature in a branch, using the feature flag, and create a pull request (PR).
3. Someone else approves and merges the PR into master. The feature is now marked as ready for QA.
4. The code gets built into our test environment automatically.
5. Someone turns the feature flag on in test for example, and tests the feature.
6. (It could very well be the case here that another critical bug shows up in production, that someone has to fix immediately. In this case, the fix is done and master deployed into production, causing our feature to follow. This is why it's important that the feature flag is turned off in production untill it's verified.)
7. QA passes and test is (if not already done) deployed into production, and feature flag turned on.

And here are some of the benefits for us

• The feature flag can provide a way to repeatedly deploy small parts of a feature into production. Feature flag only activated once the entire feature is complete. This can be beneficial because the code is frequently integrated with master.
• Get a feeling that you have a modern workflow. Remember that positive attitude between colleagues and trying new things to find what's best for you is critical for any reasonable team today. Feature flagging is one of those things you might wanna try.
• Ability to turn the feature off if it doesn't work as expected. This can be golden, compared to having to rollback, but should rarely happen. We have a QA phase for a reason.

But of course, complexity and problems follow

• It is harder to main the codebase as long as the feature flag(s) are there. That's a bold statement yes indeed. You really do not want more feature flags than really needed in a domain that's already complicated. It's way harder to reason about logic that's not just made of up if-else statements, but also feature flags.
• Your application must work even if your feature flag provider does not answer in terms of transient network error or a server breakdown. I think Launch Darkly that i've been using has very nice resilience. It's worth to quote some info from their website here:

A feature flag will always be available—all flags are stored locally and backed up by our globally distributed CDN provider. Even if the CDN goes down (we use Fastly, a widely-used CDN provider), your servers will continue operating with the last set of feature flag rules, so your customers will continue seeing the right set of features.
from: https://launchdarkly.com/how-it-works/

• This depends on your kind of application. But be sure the communication between your services are intact no matter if your flag is on or off. You'll have to decide whether to only feature flag the backend, or the frontend too, for example. Again - more complexity.
• Because of the previous tips here, feature flags must be cleant up as fast as possible, when they are of no usage anymore. This cleanup process adds context switching, possible new cleanup tasks to your board, and will take time from your colleagues (and possible even another QA phase). This by itself adds absolutely no value to your customers, but do take development time.

So here's some guidance on what works well

• Use a minimum amount of feature flags. Personally i've just used them in e-commerce systems, and for my team I think when your getting above 5 different flags it's starting to eat brain power (remember each flag can be activated independently in each environment too).
• Use a descriptive name for the feature flag. "<story_id>-Purchase" is too vague. "<story_id>-EnableCreditCardPurchaseForPrivateCustomers" is a better name. Not only developers activate feature flags, but QA and others too. It's better when the whole team easily can understand what the feature flag is doing.
• Cleanup the feature flag as soon as possible. Cleanup essentially means 1) remove the feature flag code, 2) deploy the new code and 3) delete the flag in your provider. The sooner you can cleanup, the less headache your team will get.
• One flag per customer-centric functionality. I.e. use flags so that one flag corresponds to one (or more) customer centric functionality. Never require a bunch of flags to be turned on at the same time for something to function.
• Write tests for your feature flag code. It's important that the system is intact both when the flag is on and off.
• Come up with a level of abstraction and suits your team/product. Rember feature flagging is essentially if-statements. Keep the if's on the same abstraction level if possible, i.e. try have all of them in your MVC controller, or in your domain model, but never spread them unless necessary. It helps to keep the logic easier to reason about.

Summary

Feature flagging is very fun at first glance. You feel awesome! But it's also very complex, and can damn well eat a lot of your development efficiency. So for your own sake, only flag when you have to. There's no point in using feature flags because it's "what people do". As everything else, it's a trade off. Trade wisely, and you'll get the real powerful benefits of feature flags!

What could've been better with this post? Please write a comment below. I appreciate honest feedback. / Jim