Author: AB Tasty

AB Tasty is a complete tool for website and conversion rate optimization. We serve as your digital lab, equipped with everything you need to create experiments that will help you to better understand your users and customer journeys so that you can create the clearest and most engaging user experience possible, ensuring your website performs well and yields the maximum results.

Article

Feb 9, 2021
10min read

Use Case User Onboarding: What Impact Do Release Teams Have on UX?

AB Tasty

AB Tasty

Share on

LinkedIn Facebook Twitter

According to a PWC survey, one in three customers would leave a brand after just one bad experience. Hence, your company may invest a lot of time and money optimizing your digital product to stay relevant in today’s often crammed markets.

A critical part of the overall product experience is user onboarding: get it right and win loyal customers, but get it wrong and lose those users forever.

So it makes sense to continuously tweak the user onboarding process – the perfect job for a product team. Such a team often consists of 5 to 8 people, including product managers, designers, and developers. Different companies work with various product team sizes and configurations – whatever is best for their use case. However, we rarely see DevOps engineers in these teams because many view DevOps as just a vehicle for successful feature releases. 

Ultimately, however, these DevOps engineers have to get up at night to fix a newly deployed feature that crashes the app every time a user navigates through the onboarding process.

We want to ask you: Can an app whose onboarding process doesn’t work technically be successful, and do release teams significantly impact UX after all? Let’s find out.

 

In this article, we’ll be exploring how to:
[toc content=”.entry-content”]

 

Make users feel right at home with a great onboarding experience

Most apps require an onboarding process to show new users how to achieve their goals as efficiently and conveniently as possible. 

For this, we need to keep in mind that the onboarding experience can affect your relationship with prospects – both positively and negatively. 

No matter how good your app actually is, the first impression counts! 

Large companies like Slack or Dropbox also frequently overhaul their user onboarding to ensure users have a comfortable, fun, and productive start to their product. But see for yourself. The following images show an excerpt from Slack’s onboarding process from 2014 and 2021. Of course, the design has changed drastically, but you can also see that instead of reading where the team name comes up in the Slack interface, we actually see the user interface and our team name on it. These improvements are certainly not the results of guesswork but of meticulously coordinated optimization workflows.

Slack Team Name

–

Slack Company Name
The evolution of Slack’s onboarding process (Source)

 

As even big enterprises invest in optimizing their onboarding processes, we realize that we should do the same and not rest on our laurels. The question remains, how do you make sure you are building the right onboarding experience in the right way? 

And this is where cross-functional product teams and Flagship come into play!

 

Leverage Flagship to unite product teams and ensure great UX

At AB Tasty, when we work towards a great user experience, we focus on two main themes:

  1. Release the right feature: We step into our users’ shoes and conduct experiments and tests to ensure that the feature delivers value and looks and feels good.
  2. Deploy the feature right: It’s not just about functionality and looks. We utilize feature management to ensure that what we’ve created works flawlessly at all times and on different platforms.
    –
Flagship chart
Flagship provides a shared environment for experimentation and feature management

–

Flagship gives you the means to get the most out of both: data-driven experimentation and feature management to create and release features for great customer experiences. So we see release teams as an integral part of creating value for our users. This may not be the most popular opinion. Still, now we’d like to tell you more about why we think DevOps should be more closely integrated with product teams.

It’s no secret that teams that work toward a common goal are more likely to reach their true potential than those that don’t. By isolating DevOps from product teams, you probably can’t count on the positive effects of unity and passion necessary to create and release great products. For this reason, we encourage product teams to work more closely with DevOps. Release teams also care about delivering value and great experiences to users. And they bring the skills required to do so to the table.

Flagship provides product managers, developers, and DevOps engineers with a shared environment for experimentation and feature management. You get easy access to all the data and tools needed to have a productive conversation about the product optimization process in a common data-driven language. Simultaneously, instead of isolating specific roles and responsibilities in silos, each member of the product team can focus on doing their job while continuing to work as a collective force.

Now, let’s take a look at how Flagship’s experimentation and feature management capabilities enable product teams to deliver outstanding user experiences.

 

Deploy the feature right with feature management

First, let’s talk about a few examples of how feature management and releasing a feature right can positively impact your users’ onboarding experience.

Suppose you want to add tooltips to your onboarding process to help users navigate your product’s dashboard confidently. The product team prepares the new feature accordingly and thoroughly tests the functionality on the test servers. After everything seems to be working, they roll out the new feature for all users in one fell swoop. Hopefully, it’s not Friday afternoon, as the changeover could cause unforeseen problems on the production server, like:

  • Your user is stuck in an infinite loop that they can’t exit
  • User input isn’t saved, e.g., in a form
  • The app crashes repeatedly
  • The user is sent back to the start for no apparent reason

 

Just imagine what such behavior means for users going through your onboarding process and looking forward to finally using your product when it suddenly stops working. Poof, the magic moment has passed. The user has most likely lost confidence in your app due to bad UX.

 

Flagship makes code deployments stress-free

With Flagship’s feature management capabilities, your product teams can publish new features with ease – even on Friday afternoons.

Feature management enables release teams to provide the new tooltips feature to a selected target group before continually rolling it out to everyone. This way, you can be sure that the new feature works under realistic conditions, i.e., on production servers with real users.

Through controlled and monitored rollouts, DevOps teams immediately know whether something isn’t working correctly. This enables them to react on time and be glad that only a few users have noticed the error.

For example, suppose the developers wrapped the tooltip feature in a feature flag (which they really should be doing). In that case, they can quickly deactivate it via the flagship dashboard if a problem occurs. Of course, they can also configure automatic code rollbacks based on KPIs to react even faster.

Proper feature management can de-stress your release teams: Gone are the sleepless nights spent dealing with damage control! If you want to learn more about the benefits of feature management for tech teams, we recommend our blog post here.

 

Release the right feature with experimentation

Perhaps you have great empaths on your product teams and feel like you know your users pretty well. Still, it is wise to experiment and test to create an onboarding process that your users will love.

Let’s look at the tooltip example from before again. Suppose that after your product team successfully integrated the tooltips into user onboarding, your analytics data shows that something must be wrong. Many users still don’t know how to use your app and abandon the process midway through. If you can’t identify and resolve the problem right away, you need to leverage other means to improve the tooltip’s user experience. 

First, make sure that everything is fine from a technical point of view. Next, your product team should start working on possible variants to improve the tooltips’ presentation and functionality. You can then experiment and test with Flagship to determine which of these variants and ideas offer the best user experience.

For example, you could utilize A/B tests to see if showing a how-to video before displaying the tooltips helps users get started with your product. Or experiment with different tooltips sequences – perhaps the process is easier to understand if you change the tooltips’ order.

You’re also free to experiment with different colors, copy, UI elements, call to action, and so on. To make your experiments as meaningful as possible, you can define which users see which feature variant and track user acceptance, test results, and KPIs in the Flagship dashboard. 

Another advantage of Flagship is that you can utilize 1-to-1 personalization based on audience segments to provide users with unique experiences. For example, after a user registered for a paid subscription, show them a customized welcome message and add more value to their onboarding experience.

 

… What about client-side tools for experimentation?

Many client-side experience optimization tools, such as our AB Tasty, can also perform most of these experiments – without code deployments. However, the advantage of coding your experiments for a critical process such as user onboarding is that you don’t potentially slow it down with automatically generated UI overlays. Instead, tests and experiments with Flagship are fast, secure, and flicker-free, as they come directly from the server and don’t have to be calculated in the user’s browser. Of course, client-side tools still have their justification and unique uses – Flagship is a great tool to complement your client-side strategy.

 

Wrapping up

If you want to provide users with the best possible onboarding experience, you need cross-functional teams who know how to release the right feature and how to release a feature right. One of our goals is to advocate the importance of release teams to great UX – whether a product technically works is as important as how it looks and behaves.

Using Flagship’s experimentation and feature management capabilities, product teams can benefit from a shared platform to collaborate on improving the onboarding experience in a productive and data-driven way.

Would you like to try Flagship for your product teams? Book a demo and see how experimentation and feature management can transform your users’ onboarding experience from okay to Yay.

 

Share on

LinkedIn Facebook Twitter

Subscribe to
our Newsletter

bloc Newsletter EN

We will process and store your personal data to respond to send you communications as described in our  Privacy Policy.

Article

Jan 22, 2021
8min read

Why You Should be Testing in Production

AB Tasty

AB Tasty

Share on

LinkedIn Facebook Twitter

In a perfect world, you release a product that is bug-free and works exactly as it should and so there is no need for further testing.

However, both product managers and developers know that it’s not as simple as that. They need a way to make sure that there is a process in place that reveals any issues in code in a live production environment.

This is where testing in production comes in.

But it’s also one of the highly debated topics out there with those who say you should always test in production, and those who are more wary of the concept and say you never should.

In this article, we’ll look into these two different perspectives and share our own point of view on this controversial topic and we’ll guide you through the best ways to reap the benefits of this type of testing.  

What is testing in production?

To keep it short and simple, testing in production is a software development practice of running different tests on your product when it’s in a live environment in real time. 

It allows you to test new code changes on live users rather than a test or staging environment.

This type of testing is not meant to be a replacement for your QA team or eliminating a unit test or integration test. In other words, it is not supposed to replace testing before production but to complement these tests.

To do or not to do: That is the real question

These are big benefits, and they are enough to create consensus among many developers and product managers who say “Yes, always!” to the practice.

But there’s also another group of developers and product managers who say “No, never!” to testing in production. 

On the one hand, they admit all of the great benefits that testing in production can deliver. On the other hand, they also believe that the practice carries too many potential downsides and that its benefits just aren’t worth taking on the risks the practice can bring.

Which side are we on?

We believe testing in production is a cornerstone practice for anyone in the software development world. And we believe it is particularly important for Product Managers, as it gives them a powerful method to generate real-world feedback and performance data they need to make sure they are always building a viable pipeline of products.

You can also check out our memes and gifs for test in production.

But even though we are great advocates of this practice, we still want to consider the point of view of those who are “No, never!” when it comes to this type of testing.

Once we acknowledge these issues, we can start to map out some ways to mitigate the practice’s potential downsides and focus on its benefits instead.

What are the big risks of testing in production?

To be blunt: a lot of things can go wrong when you test in production.

  • You risk deploying bad code
  • You may accidentally leak sensitive data
  • It can possibly cause system overload
  • You can mess up your tracking and analytics
  • You risk releasing a poorly designed product or feature

The list goes on and on. Anything that can go wrong, could go wrong.

Worst of all— if something does go wrong when you are testing in production, your mistake will have real-world consequences. Your product might crash at a critical moment of real-time usage. 

You might also end up collecting inaccurate KPIs and creating issues with your business stakeholders. 

Worse case scenario: your poorly designed product or feature might result in multiple paying customers leaving your product for a competitor instead.

Those who say “No, never!” to testing in production are correct to consider the practice highly risky, and we understand why they stay away from it.

And yet, while we acknowledge these concerns, when it comes down to it, we believe that this form of testing is an essential aspect of modern software development.

Why should you still test in production?

When done properly, testing in production gives you some great benefits that you just can’t get through any other method.

Collect real-world data and feedback

Testing in production allows you to collect user data in terms of users’ engagement with your new features. This enables the collection of valuable feedback from the customers that matters the most, which in turn would allow you to optimize the user experience based on this feedback. 

This will also allow you to brainstorm ideas for features that you may not have considered before.

Uncover bugs

Since you’re testing on live users, you would be able to discover any bugs or issues that you may have otherwise missed in the development stage. Thus, you can ensure your new products and features are stable and capable of handling a high volume of real-world usage.

It is worth noting that there are certain technical issues that will never show up until you put your product or feature in front of real-world users. 

Therefore, you can monitor the performance of your releases in real life so that developers can analyze performance and optimize the releases accordingly.

Higher quality releases

Because you’re receiving continuous feedback from your users, developers can improve the products resulting in high quality releases that meet your customers’ needs and expectations. 

Additionally, you can verify the scalability of your product or feature through load testing in production.

Support a larger strategy of incremental release

Testing in production helps facilitate an environment of continuous delivery. 

This is especially true when you roll out your releases to a certain percentage of users so that they may no longer have to wait long periods of time before they have access to your brand new features. 

This way, you can limit the blast radius as with incremental releases, you would not have affected all of your users.

Perhaps, most importantly: you already are testing in production, even if you didn’t know it!

Most of Agile development and product management’s best practices are forms of testing in development. We’re talking about very common practices like:

If you are following any of these practices—and many more like them—then you are already running tests with real-world users in a live production environment. 

You are already testing in production, whether you call it that or not, even if you thought you were in the “No, never!” camp this whole time.

Testing in production done right

If testing in development is inevitable these days, then you should spend less time debating its pros and cons, and more time finding the most effective and responsible way to follow the practice.

We believe in this perspective so strongly that we’ve built an entire product suite around helping product developers gain all of the benefits of the practice while minimizing their risks. 

Feature flags – a software development practice that allows you to enable or disable functionality without deploying code – are at the core of this new platform.

By wrapping your features in a flag and deploying them into production without making them visible to all users, you can safely perform all of the testing in production that you need. 

With feature flags—combined with the rest of AB Tasty— you can:

  • Deploy smaller releases that minimize the impact of failure.
  • Only test your new features on your most loyal and understanding users.
  • Personalize their tests so they know to expect a few hiccups with the release.
  • Immediately toggle off underperforming features with a single click.

Read The Many Uses of Feature Flags to Control Your Releases for more use cases and examples.

With feature flags and a little planning, you can dramatically reduce the risk and increase the sophistication of the testing in production you are already performing. 

This means more real-world user data, more reliable products & features, and less worry about seeing how your hard work performs outside of the safe confines of development and staging environments.

Share on

LinkedIn Facebook Twitter