Implement Feature Toggels And Perform Dark Launches

Implement Feature Toggles

 Feature toggles is the mechanism to selectively enable and disable features without requiring a production code deployment.

Feature toggles can also control which features are visible and available to specific user segments (e.g., internal employees, segments of customers).

Feature toggles are usually implemented by wrapping application logic or UI elements with a conditional statement, where the feature is enabled or disabled based on a configuration setting stored somewhere.

Feature Toggles Help us achieve below benefits

1.       Roll back easily

2.       Gracefully degrade performance

a.       By allowing only few users to have access a resource hungry feature such as recommendations in Amazon.

b.       By toggling off a resource hungry feature.

Perform Dark Launches

Feature toggles allow us to deploy features into production without making them accessible to users, enabling a technique known as dark launching. 

This is where we deploy all the functionality into production and then perform testing of that functionality while it is still invisible to customers. 

For large or risky changes, we often do this for weeks before the production launch, enabling us to safely test with the anticipated production-like loads giving us confidence that our service will perform as it needs to. 

Furthermore, when we launch a feature, we can progressively roll out the feature to small segments of customers, halting the release if any problems are found.

 That way, we minimize the number of customers who are given a feature only to have it taken away because we find a defect or are unable to maintain the required performance.

Case Study

Dark Launch of Facebook Chat (2008) For nearly a decade, Facebook has been one of the most widely visited Internet sites, as measured by pages viewed and unique site users.

In 2008, it had over seventy million daily active users, which created a challenge for the team that was developing the new Facebook Chat functionality.

Eugene Letuchy, an engineer on the Chat team, wrote about how the number of concurrent users presented a huge software engineering challenge: “The most resource-intensive operation performed in a chat system is not sending messages. It is rather keeping each online user aware of the online-idle-offline states of their friends, so that conversations can begin.” 

Implementing this computationally-intensive feature was one of the largest technical undertakings ever at Facebook and took almost a year to complete.

 Part of the complexity of the project was due to the wide variety of technologies needed to achieve the desired performance, including C++, JavaScript, and PHP, as well as their first use of Erlang in their back-end infrastructure. 

Throughout the course of the year-long endeavor, the Chat team checked their code in to version control, where it would be deployed into production at least once per day. 

At first, the Chat functionality was visible only to the Chat team. Later, it was made visible to all internal employees, but it was completely hidden from external Facebook users through Gatekeeper, the Facebook feature toggling service. 

As part of their dark launch process, every Facebook user session, which runs JavaScript in the user browser, had a test harness loaded into it—the chat UI elements were hidden, but the browser client would send invisible test chat messages to the back-end chat service that was already in production, enabling them to simulate production-like loads throughout the entire project, allowing them to find and fix performance problems long before the customer release. 

By doing this, every Facebook user was part of a massive load testing program, which enabled the team to gain confidence that their systems could handle realistic production-like loads.

The Chat release and launch required only two steps: modifying the Gatekeeper configuration setting to make the Chat feature visible to some portion of external users, and having Facebook users load new JavaScript code that rendered the Chat UI and disabled the invisible test harness. 

If something went wrong, the two steps would be reversed. 

When the launch day of Facebook Chat arrived, it was surprisingly successful and uneventful, seeming to scale effortlessly from zero to seventy million users overnight. 

During the release, they incrementally enabled the chat functionality to ever-larger segments of the customer population—first to all internal Facebook employees, then to 1% of the customer population, then to 5%, and so forth.

As Letuchy wrote, “The secret for going from zero to seventy million users overnight is to avoid doing it all in one fell swoop.”

Note : Material in this blog is taken from  the below book , I highly recommend to read this book to understand Devops and read alot of  case studies of companies transitioning from monolithic to Service Oriented Architecture/devops.

Kim, Gene. The DevOps Handbook: . IT Revolution Press. Kindle Edition.