You’ve heard the stories, right?
Coal mining was a dangerous job. At any moment the coal mines could fill up with lethal—but impossible to detect—gasses. So coal miners thought up a simple trick to stay alive. They would bring a canary down into the mine with them. The canaries were more sensitive to those fatal gasses than the coal miners were, so if something deadly began to fill the mine, the canary would die first, and the coal miners would know to exit before they suffered the same fate.
Well, software developers took these stories to heart and developed their own way to detect danger in their work before it became fatal to their application. They developed “canary releases”— a best practice of feature release management where they would release a new feature to a small subset of users to test it for issues in a live environment. If something was critically wrong with the new feature, the canary release would reveal it to the developers before the feature hit the broad user base and caused real harm.
Software developers weren’t saving their own lives, but they were potentially saving the life of the software they were developing. They were able to find and fix issues quickly, to collect real-world usage data that translated into better software, and to overall lower the risk of the most critical moments in their development and release lifecycle.
As a product manager, running canary releases has become a “must-do”. Even a single bug or misaligned feature can ruin months of hard work if you don’t catch it while it’s still small, contained, and manageable.
That’s the bad news. The good news is: you can prevent this potential crisis scenario if you just run a canary release… and you can do so quickly, easily, and intelligently by simply asking yourself three critical questions.
Question 1: Which Users to Invite to Your Canary Release?
This is a critical question to ask, and, unfortunately, there is no single “right” answer. The specific user group you choose will depend on the exact nature of your canary release, and what you are looking to get out of it.
Broadly speaking, there are two schools of thought here:
- Release to a “warm” audience.
- Release to a “cold” audience.
The first “warm” school of thought says you should deploy your canary release to early adopters who you give some advance warning. Basically, these are users who you know feel positively about you and your product, and who use it all the time. After selecting them you will then tell them in advance that they’re going to receive some new features that these features aren’t quite perfect yet and that they should expect some hiccups.
The upside here is simple— you get some feedback and data from users who are playing with your new features in a real-world environment, but you do so in the most risk-averse manner possible.
The downside is a little more subtle. Because you’re releasing your feature to a “warm” audience, you can’t guarantee that they’re going to give you 100% honest feedback. And because this group is often made up of early adopters and power users, you can’t guarantee they are going to play with new features—and work around technical issues— in the same manner as your normal, everyday users would.
The second “cold” school of thought says you should just choose a random selection of users, whom you may or may not give advance notice that they’re going to be testing out some new, potentially-buggy features.
This is a much riskier user group to test new features with, but it also gives you the most honest, realistic, and actionable data. You will see how a broad range of users leverage your feature, you will generate much more honest feedback (especially if something isn’t working correctly), and overall you will develop a much clearer picture of what would actually happen if you released your new feature today.
So, which user group should you choose?
In practice, you’re probably going to choose a user base that falls somewhere between these two extremes. You will most likely deploy some advanced segmenting and personalization that will release your new features to an audience that is highly relevant and well-suited to the specific features you’re testing, but that will also be partitioned well enough that you won’t risk permanently damaging your product’s bottom line if your release goes poorly.
In fact, with the right approach, you can launch to both— running tests with multiple audiences, who can each give you different forms of valuable feedback and data. You can even run multiple tests, on multiple audiences, for multiple features, much easier than before.
Question 2: Which Features Will You Test in Your Canary Release?
This question to be pretty simple to answer. You really could only launch a canary release for a relatively bounded set of features tests. But in recent years your range of options has exploded. Here’s what’s changed.
Historically, canary releases were pretty resource-intensive to launch. They could compromise your code base. They could require a lot of time, attention, and effort to maintain. And they could dramatically slow down the overall development process. Any time you wanted to run a canary release, you had to make sure it was 100% worthwhile— and this set some big boundaries on the types of features you could test.
For the most part, it was only worth running a canary release for high-touch, high-visibility features that were foundational to the application. Things like huge overhauls of an entire feature set or functional corner of your application. It generally wasn’t worth the resources to run a canary release to test a single, small, isolated feature.
This limited your options. It prevented you from being fast, nimble, and responsive. You had to wait for enough features to be ready to launch before you could test even one of them. You couldn’t rapidly iterate each feature from the feedback and data you received from your release. In short— your canary release wouldn’t be particularly agile.
But today, with the right approach you have a lot more flexibility about what features you can test during your canary release. You can now test single, small, isolated features—as soon as they are ready to launch—in a real-world environment without adding substantial operational overhead to your team.
Question 3: How Will You Actually Execute Your Canary Release?
Canary releases were so resource-intensive because everything involved used to be, more or less, manual.
You needed to figure out some way to only send some of your users to your updated version of the application. You needed to manage, configure, and monitor both short-term and long-term changes to your code. You needed to manually track each user group, across different tests, documenting their usage, their feedback, and any KPIs their tests directly impacted. And you had to do all this while maintaining perfect functionality on the standard version of your application.
The whole process ate up a lot of hours. It was error-prone, And it was impossible to run quick, sophisticated tests with even the best team on your side.
So what’s changed?
In one word: software.
You can now automate a lot of the processes involved in running a canary test, and run every experiment through a single dashboard on a single centralized platform.
With the right platform you’ll be able to:
- Design and configure the entire test
- Test multiple features independent of each other
- Create multiple highly-personalized segments and testing experiences
- Launch all of your tests with the click of a button
- Monitor the performance of each feature and user group in real-time
- Run the whole thing yourself without a significant time investment
Best of all, the right platform will reduce the risk associated with a canary release even further. With a simple feature flag function, you will be able to toggle the individual features you are testing on and off, at any time, in response to their real-time performance. You won’t have to roll back your entire release if just one feature is going wrong with just one feature group— you can switch that specific feature off for that specific user group and let the rest of the test keep rolling.
We’ve bundled all of this functionality, and more, into our own platform, Flagship. We believe that canary releases are a critical element of any agile product development plan, and we’ve done what we could to make them as simple, easy, and cost-effective to run as possible.
You can learn more about our platform here, and get started launching your first canary release ASAP!