Article

7min read

Using Failed A/B Test Results to Drive Innovation

“Failure” can feel like a dirty word in the world of experimentation. Your team spends time thinking through a hypothesis, crafting a test, and finally when it rolls out … it falls flat. While it can feel daunting to see negative results from your a/b tests, you have gained valuable insights that can help you make data-driven, strategic decisions for your next experiment. Your “failure” becomes a learning opportunity.

Embracing the risk of negative results is a necessary part of building a culture of experimentation. On the first episode of the 1,000 Experiments Club podcast, Ronny Kohavi (formerly of Airbnb, Microsoft, and Amazon) shared that experimentation is a time where you will “fail fast and pivot fast.” As he learned while leading experimentation teams for the largest tech companies, your idea might fail. But it is your next idea that could be the solution you were seeking.

“There’s a lot to learn from these experiments: Did it work very well for the segment you were going after, but it affected another one? Learning what happened and why will lead to developing future strategies and being successful,” shares Ronny.

In order to build a culture of experimentation, you need to embrace the failures that come with it. By viewing negative results as learning opportunities, you build trust within your team and encourage them to seek creative solutions rather than playing it safe. Here are just a few benefits to embracing “failures” in experimentation:

 

  1. Encourage curiosity: With AB Tasty, you can test your ideas quickly and easily. You can bypass lengthy implementations and complex coding. Every idea can be explored immediately and if it fails, you can get the next idea up and running without losing speed, saving you precious time and money.
  2. Eliminate your risks without a blind rollout: Testing out changes on a few pages or with a small audience size can help you gather insights in a more controlled environment before planning larger-scale rollouts.
  3. Strengthen hypotheses: It’s easy to fall prey to confirmation bias when you are afraid of failure. Testing out a hypothesis with a/b testing and receiving negative results confirms that your control is still your strongest performer, and you’ll have data to support the fact that you are moving in the right direction.
  4. Validate existing positive results: Experimentation helps determine what small changes can drive a big impact with your audience. Comparing negative a/b test results against positive results for similar experiments can help to determine if the positive metrics stand the test of time, or if an isolated event caused skewed results.

 

In a controlled, time-limited environment, your experiment can help you learn very quickly if the changes you have made are going to support your hypothesis. Whether your experiment produces positive or negative results, you will gain valuable insights about your audience. As long as you are leveraging those new insights to build new hypotheses, your negative results will never be a “failure.” Instead, the biggest risk would be allowing a status quo continuing to go unchecked.

“Your ability to iterate quickly is a differentiation,” shares Ronny. “If you’re able to run more experiments and a certain percentage are pass/fail, this ability to try ideas is key.”

Below are some examples of real-world a/b tests and the crucial learnings that came from each experiment:

 

Lesson learned: Removing “Add to Basket” CTAs decreased conversion

In this experiment, our beauty/cosmetics client tested removing the “Add to Basket” CTA from their product pages. The idea behind this was to test if users would be more interested in clicking through to the individual pages, leading to a higher conversion rate. The results? While there was a 0.4% increase in visitors clicking “Add to Basket,” conversions were down by 2%. The team took this as proof that the original version of the website was working properly, and they were able to reinvest their time and effort into other projects.

Beauty client add to basket use case

 

Lesson learned: Busy form fields led to decreased leads

A banking client wanted to test if adjusting their standard request form would drive passage to step 2 and ultimately increase the number of leads from form submissions. The test focused on the mandatory business identification number field, adding a pop-up explaining what the field meant in the hopes of reducing form abandonment. The results? They saw a 22% decrease in leads as well as a 16% decrease in the number of visitors continuing to step 2 of the form. The team’s takeaways from this experiment were that in trying to be helpful and explain this field, their visitors were overwhelmed with information. The original version was the winner of this experiment, and the team saved themselves a huge potential loss from hardcoding the new form field.

Banking client form use case

 

Lesson learned: Product availability couldn’t drive transactions

The team at this beauty company designed an experiment to test whether displaying a message about product availability on the basket page would lead to an increase in conversions by appealing to the customer’s sense of FOMO. Instead, the results proved inconclusive. The conversion rate increased by 1%, but access to checkout and the average order value decreased by 2% and 0.7% respectively. The team determined that without the desired increase in their key metrics, it was not worth investing the time and resources needed to implement the change on the website. Instead, they leveraged their experiment data to help drive their website optimization roadmap and identify other areas of improvement.

Beauty client availability use case

 

 

Despite negative results, the teams in all three experiments leveraged these valuable insights to quickly readjust their strategy and identify other places for improvement on their website. By reframing the negative results of failed a/b tests into learning opportunities, the customer experience became their driver for innovation instead of untested ideas from an echo chamber.

Jeff Copetas, VP of E-Commerce & Digital at Avid, stresses the importance of figuring out who you are listening to when building out an experimentation roadmap.  “[At Avid] we had to move from a mindset of ‘I think …’ to ‘let’s test and learn,’ by taking the albatross of opinions out of our decision-making process,” Jeff recalls. “You can make a pretty website, but if it doesn’t perform well and you’re not learning what drives conversion, then all you have is a pretty website that doesn’t perform.”

Through testing you are collecting data on how customers are experiencing your website,  which will always prove to be more valuable than never testing the status quo. Are you seeking inspiration for your next experiment? We’ve gathered insights from 50 trusted brands around the world to understand the tests they’ve tried, the lessons they’ve learned, and the successes they’ve had.

You might also like...

Subscribe to
our Newsletter

bloc Newsletter EN

AB Tasty's Privacy Policy is available here.

Article

8min read

What Exactly Does Customer Experience (CX) Mean? And How Can We Measure It?

It’s a competitive world online. In order to beat your competition, you need world-class customer experience (CX). 

But what exactly does customer experience mean, anyway? 

This is a non-technical guide to customer experience; we’ll cover what CX is, why it’s important, how it benefits your business, and how you can measure it. 

By the end of the article, you’ll have everything you need to start measuring, improving, and delivering an exceptional experience for your customers.

The Definition of Customer Experience

Customer experience is a big deal. 

So much so that a study by Econsultancy found that the number one priority for more than 2,000 marketers was customer experience when asked the question:

“Over the next five years, what is the primary way your organization will seek to differentiate itself from competitors?”

So what is customer experience? 

Customer experience (CX for short) is the perception of your customers’ experience with your business or brand. 

Every time a customer interacts with your brand and everything your business does, it has an impact on how your customers perceive your business. 

Customer experience is the culmination of all of these experiences and interactions, impacting your decision whether to return to the business or make a purchase. 

Simply put, the key to your success is epic customer experience. 

The Importance of Customer Experience for Your Business

The core benefits of improving your customer experience include the following:

  1. Increase the loyalty of your customers
  2. Increase the happiness and satisfaction of your customers
  3. Increase positive reviews and recommendations
  4. Reduce returns 
  5. Reduce complaints
  6. Reduce churn 

There is no doubt that to be a successful business, you need to put your customers first. If you do, it’s not only good for them, but it’s also good for your business. 

The Customer Journey and Customer Experience

As we’ve seen, customer experience is concerned with the holistic experience of your customers with your brand. 

This means that every step of the customer journey has the potential to impact on your customer’s experience. 

In terms of customer experience, the stages of a customer’s journey include discovery, research, purchase, and post-purchase customer support. 

It should go without saying, but in terms of customer experience, there’s one extremely important caveat to the customer journey.

Instead of considering each stage of the journey from the business’s perspective, it’s imperative to consider these touchpoints from the perspective of the customer. 

To improve customer experience at each stage, we need to understand how and why they are different, and what we can do to improve the experiences of your customers.

To better understand customer experience, let’s break down the customer journey into stages.  

Stage One: Consideration

When a customer changes from an observer to a potential customer, they enter the “Consideration” phase. 

It’s at this stage that a customer is making a choice whether to consider your brand or exclude it from their buying journey. 

nike brand

At this point, branding is hugely important. Well-known brands have the upper hand at this point, but if you own a smaller company, the best way you can make an impact is by having a clear message that solves a customer’s problem. 

Stage Two: Evaluation

There are two ways in which a potential customer can enter the evaluation stage. 

First, they may have “selected” your brand during the consideration stage or, interestingly, they may enter the cycle fresh in the evaluation stage. 

While this may seem odd, it’s common sense. As a potential customer researches products similar to yours, it’s likely that they stumble across new brands, as well as the brands they’ve previously considered. 

The way you advertise to customers at this stage of the cycle should, therefore, reflect both of these types of customers and you should take into account the fact that many people have not yet interacted with your brand at all. 

Stage Three: Buying

Hooray! This is the stage we’ve all been waiting for. And it’s more than likely the stage that most marketers focus on. 

Everything from the text on the ‘Buy’ button to the copy on the product page is A/B tested, iterated, and scrutinized. 

Of course, this is highly important, but by placing too much emphasis on this one stage of the cycle, it could be detrimental to the customer experience overall. 

Stage Four: Post-Purchase 

Often overlooked, the post-purchase stage is an equally important part of the customer experience journey. 

How you treat your customers after they have bought from you will affect whether they choose to repeat the process…or not.

The effect of customer satisfaction after a purchase can have a huge impact on your bottom line. Consider these statistics:

  • It can cost up to 25 times as much to acquire a new customer than to retain an existing one
  • If you increase retention rates by 5%, then you can increase profits by 25% to 95%

And here’s the punch line: only 18% of businesses focus on customer retention. 

This represents a huge opportunity for you to beat your competition, increase your profits, and improve your customer experience. Win, win, win. 

The Importance of Website Personalization

“Website personalization is the real-time individualization of a website to suit each visitor’s unique needs and guide them through a custom conversion funnel.” 

And it goes hand in hand with customer experience. 

Personalization can be done at any stage of the funnel, from personalized email marketing to exit-intent popups and discount offers. 

A well-targeted personalized message evokes an emotional response and creates a stronger, more memorable connection with your brand. 

How to Measure Customer Experience

Instead of measuring the ROI of customer experience, which will likely take the focus off customer satisfaction, you should use one or more of the following industry-specific metrics: 

NPS (Net Promoter Score)

The Net Promoter Score, or NPS, measures customer experience and can also predict business growth. 

It’s a simple measurement, but it’s been adopted around the world and is seen as one of the best ways to measure your customer’s experience. 

To run the test, ask your customers on a scale of one to ten whether or not they would recommend your brand or business to a friend. 

Respondents are then grouped into “Promoters” (score 9-10), “Passives” (score 7-8), or “Dectractors” (0-6). 

Once you’ve collected your responses, subtract the percentage of Detractors from the percentage of Promoters, and you have your Net Promoter Score. 

It’s simple, straightforward, and it works. 

CES (Customer Effort Score)

Like the NPS, CES is another way to measure customer experience. 

Also using a survey, it asks customers to rank their experience with the business ranging from “Very Difficult” to “Very Easy.” 

The theory is, if you make it easy for your customers to make a purchase or solve their problem, then it’s more likely that they will continue to pay for your product or service. 

The easiest way to increase customer loyalty is not through “wowing your customers,” but rather through making it easier to get their job done.

CSAT (Customer Satisfaction Score) 

The final customer service metric is the Customer Satisfaction Score. 

It’s a basic test and, due to the difference in opinion over what is classed as “satisfactory,” its validity can be questioned. 

Nevertheless, it’s simple to do and can offer insight into how well your customers experience your brand.

The survey asks the question “How satisfied were you with your experience?” in order to gauge customer satisfaction with their purchase or interaction with your business. 

The response should be on a scale from 1–3, 1–5, or 1–10.

TTR (Time to Resolution)

Time to Resolution, or TTR, is the average time it takes for your customer service team to resolve an issue raised by a customer. 

One of the best ways to reduce customer frustration is to ensure the timely resolution of their issues. 

Unfortunately, if you don’t measure your response times, it’s impossible to know whether or not you’re hitting the mark. 

To test your TTR, simply add up all of the time it takes to resolve issues and divide the total by the number of tickets addressed. 

TTR is the average length of time it takes for customer service teams to resolve an issue. 

Conclusion

In a digital world, customer expectations are higher than ever, and word of good (or bad) service travels at the speed of light. Customer experience requires constant care and attention. Focus on a long-term customer experience strategy and always offer a customer-centric strategy. 

By understanding your customer experience journey and testing your customers’ satisfaction levels, you can improve your customers’ perceptions of your brand and, in turn, improve brand loyalty, increase retention, and achieve higher profits.