When it comes to feature testing, you’re in a bind.
On the one hand, you need real-world data and feedback from real-world users. You know that every new feature you develop is, at best, an educated guess about what your real-world users want from you. No matter how educated that guess might be, and no matter much internal validation you perform, you can only generate meaningful data and feedback on each new feature you develop by releasing it to real-world users to test out in their real-world environments.
On the other hand, it’s risky to give real-world users an unproven feature. You know that every new feature you release might have something wrong with it. Maybe there’s a technical issue you missed during development. Maybe it just doesn’t align to user expectations as closely as you hoped. No matter the issue, releasing an unproven feature can cause real harm to your brand’s user relationships.
This is a tricky problem and one that is never going to be fully solved. But, thankfully, there are methods you can follow to minimize the problem, and collect real-world data and feedback while mitigating the impact when something (inevitably) goes wrong.
In this piece, we’ll explore one of these methods— rollbacks.
What is a Feature Rollback?
It’s a simple practice, with powerful implications.
When you perform a rollback, you take some code out of a live environment. Back in the day rollbacks could be truly massive. Software products used to be updated in giant new releases that could include a wide range of changes— including multiple new features and significant changes to existing features. If one of these huge releases had some fatal bugs in it, or just wasn’t well-received by users, then the entire thing might need to be rolled back (even if the issues were contained within just a few elements of the release).
All of this has changed with the adoption of Agile methodology. Releases keep getting smaller and more incremental, and so do rollbacks. Most modern Product Managers have adopted phased release plans, where they only release a single new or upgraded feature at a time— and often only to individual segments. And when modern Product Managers do release multiple new or upgraded features at once, the different features are separate from each other.
This evolution has changed the way rollbacks happen. After a new release, Product Managers can now isolate the individual feature(s) that have proven unfit for live usage and perform a targeted rollback on them, and them alone. The whole rollback process is now much faster, much nimbler, much more precise— and delivers much greater benefits.
Why Should Product Managers Perform Rollbacks?
When a Product Manager properly structures and deploys rollbacks, they improve their ability to test new features in a real-world environment with real-world users with a minimal level of risk. An imperfect feature is no longer the end of the world. If a feature has development issues or poor alignment with user requirements, you can perform a rollback and remove it from a live environment in real-time with just one click.
For Product Managers, this changes the game. The more mature your rollback capability, the more you can afford to make mistakes. Your risk shrinks, giving you the freedom to test more features with more users earlier in the development cycle, ultimately leading you to iterate your products faster and faster.
Now, rollbacks are not a silver bullet. They don’t absolve you from doing everything you can to develop the highest-quality features possible before you test them. But rollbacks allow you to test new features with greater confidence and reduced concerns about creating problems for your users.
When Should You Perform a Rollback? Two Common Use Cases
For most Product Managers, there are two common use cases why you might need to perform a rollback.
Rollback Use Case 1: Your Feature has a Bug
This first use case is pretty self-explanatory.
You might have the most robust and thorough QA and testing processes in the world. It’s still highly likely your new features will still have one or more bugs in them when you release them into a live environment. Maybe they’re issues you just didn’t think to search for or didn’t know how to search. Maybe they’re issues that only show up in live environment after hundreds of real-world users tool around with the feature.
Regardless of the reason, if significant technical issues pop up in your new feature, then you’ll likely want to perform a rollback on that feature to fix it. With the right rollback process, you can react to these errors in near-real-time and remove the feature—and maybe even fix it—in minutes before it impacts too many users.
Rollback Use Case 2: Your Feature is Poorly Received
This second use case is a little more sophisticated.
Essentially, after you release a new feature you monitor how users respond to it, and how well it’s hitting your business KPIs. If your new feature is not performing as expected, and is generating negative usage data and user feedback, then you can perform a rollback to remove it from its live environment. If it isn’t hitting—or at least tracking towards—its business goals, then it might not be worth keeping live.
After you roll back your feature, you can either utilize the data and feedback you collected to fix the feature and help it better align to user expectations and business requirements, or you can decide that the feature was fundamentally misguided and just needs to be retired.
With the right rollback process, you can also review and respond to the usage data and user feedback you receive in near-real-time, and prevent too many users from getting too disgruntled about receiving a feature that misses the mark.
What Do These Two Use Cases Have in Common?
In one word: speed.
In both use cases, rollbacks are most effective—and mitigate the most risk—when you are able to first monitor feature performance in real-time, to then translate that performance into a quick “yes/no” decision to rollback (or not), and finally to execute on that rollback decision as rapidly as possible.
The faster you can go through this entire process, the lower the chance that you will create a prolonged negative user experience. In some scenarios, the decision to perform a rollback and the execution of that rollback need to happen in minutes.
It’s a daunting mandate, but here are a few tips to help you meet it.
How to Make Faster Rollback Decisions
It’s challenging to decide—in the moment—whether or not to rollback a feature. Even the best feature release can be complex and chaotic.
There are multiple moving parts to monitor…
There are many different data and feedback points to take into consideration…
And there’s a lot of emotion at play…
You and your team just spent weeks, maybe even months, pouring your blood, sweat, and tears into designing and developing the new feature that you’re testing. If your users love it, then you get that sugar high of knowing you just completed a job well done, and you can just sit back and watch the good data and feedback roll in. But if your users don’t immediately respond as positively as you hoped, then it’s easy to experience an emotional crash and to want to rollback the feature before you even know if the bad response is consistent, let alone what you should do to fix your errors in the next iteration.
For these reasons, and many more, it’s hard to make the right rollback decision in the moment during a feature release test. Instead, it’s better to make your rollback decisions before you release your new features into the wild.
Here’s what we mean.
Basically, before you release any new features to any real-world users, you first decide what success and failure looks like for this feature in objective, data-driven terms.
Then, you decide how much data your release will need to generate before you can make an accurate call about whether the feature is a success or a failure.
Finally, you use these parameters during your release to make objective “yes/no” decisions about whether or not you should rollback your feature at any point. Instead of getting caught up in the moment, you just monitor the performance metrics that you decided were most important, and once they hit the thresholds you set prior to release you simply follow the plan and you either rollback the feature or you don’t— no real-time agonizing required.
How to Execute Rollbacks Faster
In the past, it was near-impossible to perform a rollback quickly from a technical perspective. You needed to have a technical team standing by, waiting to dig into the code to turn off live features, or to revert to a prior state of the entire platform. The entire process was slow, it was labor-intensive, and it took your technical teams away from their valuable development work.
Software has solved all of these problems. With our own feature management platform, you can rollback a feature in real-time by just toggling a single field with just one click. You don’t need any technical expertise to do so. You don’t need to develop and test a complex rollback process prior to feature release. You don’t even need to think about the technical details— you can save all of that thinking to create the right strategic decision trees that we outlined in the prior section.
AB Tasty also gives you—or any non-technical user—the ability to perform sophisticated feature releases and rollbacks. You can release multiple features at once, monitor how each feature is performing individually, and only rollback the features that aren’t delivering. You can roll out a feature to multiple user segments and only rollback that feature to the individual segments that aren’t responding well to it. We designed our server-side solution to make the execution of rollbacks faster, easier, and far more intuitive than they ever were before.
