Article

7min read

The ROI of Experimentation

When you hear ‘A/B Testing’, do you think straight away of revenue gain? Uplift? A dollars and cents outcome? 

According to David Mannheim, CEO of the Conversion Rate Optimization (CRO) agency User Conversion, you probably do – and shouldn’t. Here’s why:

Unfortunately, it’s just not that simple. 

Experimentation is more than just a quick strategy to uplift your ROI

In this article we will discuss why we experiment, the challenges of assessing return on investment (ROI), prioritization, and what A/B testing experimentation is really about. 

Why do we experiment?

Technically speaking, experimentation is performed to support or reject a hypothesis. Experimentation provides you with valuable insights into cause-and-effect relationships by determining the outcome of a certain test when different factors are manipulated in a controlled setting. 

In other words, if there is no experiment, there is no way to refute a hypothesis and reduce the risk of losing business or negatively impacting metrics.

Experimentation is about prioritization, minimizing risk and learning from the outcome. The tests you choose to implement should be developed accordingly. It’s not necessarily about making the “right” or “wrong” decision, experimentation helps you make better decisions based on data.

In visual terms, experimentation will look something like this:

ROI frustration backlog

Online experiments in the business world must be carefully designed to learn, accomplish a specific purpose, and/or measure a key performance indicator that may not have an immediate financial effect. 

However, far too often it’s the key stakeholders (or HIPPOs) who decide what tests get implemented first. Their primary concern? The amount of time it will take to see a neat revenue uplift.

This tendency leads us to the following theory:

The ROI of experimentation is impossible to achieve because the industry is conditioned to think that A/B testing is only about gain.

Frustrations and challenges of ROI expectations 

You may be asking yourself at this point, What’s so bad about expecting revenue uplift from A/B tests? Isn’t it normal to expect a clear ROI?

It is normal, however, the issue isn’t just that simple.

We’ve been conditioned to expect a neat formula with a clean-cut solution: “We invested X, we need to get Y.”  

This is a misleading CRO myth that gets in the way. 

Stakeholders have come to erroneously believe that every test they run should function like this – which has set unrealistic ROI expectations for conversion optimization practitioners

As you can imagine, this way of thinking creates frustration for those implementing online experimentation tests.

Experiment backlog example

What people often overlook is the complexity of the context in which they are running their experimentation tests and assessing their ROI.

It’s not always possible to accurately measure everything online, which makes putting an exact number on it next to impossible. 

Although identifying the impact of experiments can be quite a challenge due to the complexity of the context, there are some online tools that exist to measure your ROI efforts as accurately as possible. 

AB Tasty is an example of an A/B testing tool that allows you to quickly set up tests with low-code implementation of front-end or UX changes on your web pages, gather insights via an ROI dashboard, and determine which route will increase your revenue.

Aside from the frustration that arises from the ingrained ROI expectation to be focused on immediate financial improvement, three of the biggest challenges of the ROI of experimentation are forecasting, working with averages, and multiple tests at once.

Challenge #1: Forecasting

The first challenge with assessing the ROI of experimentation is forecasting. A huge range of factors impacts an analyst’s ability to accurately project revenue uplift from any given test, such as:

  • Paid traffic strategy
  • Online and offline marketing
  • Newsletters
  • Offers
  • Bugs
  • Device traffic evolution
  • Season
  • What your competitors are doing
  • Societal factors (Brexit)

In terms of estimating revenue projection for the following year from a single experiment– it’s impossible to predict an exact figure. It’s only possible to forecast an ROI trend or an expected average. 

Expecting a perfectly accurate and precise prediction for each experiment you run just isn’t realistic – the context of each online experimentation test is too complex.

Challenge #2: Working with averages

The next challenge is that your CRO team is working with averages – in fact, the averages of averages.

Let’s say you’ve run an excellent website experiment on a specific audience segment – and you experienced a high uplift in conversion rate. 

If you then take a look at your global conversion rate for your entire site, there’s a very good chance that this uplift will be swallowed up in the average data. 

Your revenue wave will have shrunk to an undetectable ripple. And this is a big issue when trying to assess overall conversion rate or revenue uplift – there are just too many external factors to get an accurate picture.

With averages, the bottom line is that you’re shifting an average. Averages make it very difficult to get a clear understanding. 

On average, an average customer, exposed to an average A/B test will perform… averagely

Challenge #3: Multiple tests

The third challenge of ROI expectations happens when you want to run multiple online experiments at one time and try to aggregate the results. 

Again, it’s tempting to run simple math equations to get a clear-cut answer for your gain, but the reality is more complicated than this. 

Grouping together multiple experiments and the results of each experiment will provide you will blurred results

This makes ROI calculations for experimentation a nightmare for those simultaneously running tests. Keeping experiments and their respective results separate is the best practice when running multiple tests.

Should it always be “revenue first”?

Is “revenue first” the best mentality? When you step back and think about it, it doesn’t make sense for conversion optimizers to expect revenue gain, and only revenue gain, to be the primary indicator of success driving their entire experimentation program.

What would happen if all businesses always put revenue first?

That would mean no free returns for an e-commerce site (returns don’t increase gain!), no free sweets in the delivery packaging (think ASOS), the most inexpensive product photographs on the site, and so on.

If you were to put immediate revenue gain first – as stakeholders so often want to do in an experimentation context – the implications are even more unsavory. 

Let’s take a look at some examples: you would offer the skimpiest customer service to cut costs, push ‘buy now!’ offers unendingly, discount everything, and forget any kind of brand loyalty initiatives. Need we go on?

In short, focusing too heavily on immediate, clearly measurable revenue gain inevitably cannibalizes the customer experience. And this, in turn, will diminish your revenue in the long run.

What should A/B testing be about?

One big thing experimenters can do is work with binomial metrics

Avoid the fuzziness and much of the complexity by running tests that aim to give you a yes/no, black or white answer.

binomial metrics examples

Likewise, be extremely clear and deliberate with your hypothesis, and be savvy with your secondary metrics: Use experimentation to avoid loss, minimize risk, and so on.

But perhaps the best thing you can do is modify your expectations

Instead of saying, experimentation should unfailingly lead to a clear revenue gain, each and every time, you might want to start saying, experimentation will allow us to make better decisions.

Good experimentation model

These better decisions – combined with all of the other efforts the company is making – will move your business in a better direction, one that includes revenue gain.

The ROI of experimentation theory

With this in mind, we can slightly modify the original theory of the ROI of experimentation:

The ROI of experimentation is difficult to achieve and should be contextualized for different stakeholders and businesses. We should not move completely away from a dollar sign way of thinking, but we should deprioritize it. “Revenue first” is not the best mentality in all cases- especially in situations as complex as calculating the ROI of experiments.

Subscribe to
our Newsletter

bloc Newsletter EN

We will process and store your personal data to respond to send you communications as described in our  Privacy Policy.

Article

6min read

How to Become a Data-Centric Company

Get caught up with the series introduction here or read the previous installment, Building Customer-Centric Cultures with Data.

In our next installment of the  Customer-Centric Data Series, we spoke to Stephen Welch, Managing Director, and Ian Bobbit, Chief Analytics Officer from Realise about how businesses can become data-centric companies. Realise, a part of the larger Unlimited Group, helps brands make data-driven decisions to maximize growth. 

They discussed how organizations can structure themselves better to become data-driven, what teams, structure, tech stack and KPI’s you need to look out for, as well as some details on personalization and customer lifetime value.

What are the main challenges facing companies that want to become data-centric?

The first thing both Ian and Stephen stress is that most companies already have a large amount of data, but that data can sometimes be siloed by teams. The first challenge is being able to consolidate that information and understand its potential. 

Another challenging factor is needing the buy-in from key stakeholders. There needs to be a senior leader in the team who wants to learn from data. Having someone on board, who can manage across multiple teams, will help companies identify data that describes their customers and behavior on a day-to-day basis.

So if we know that the data is already there, the other challenge is to ensure it is being used to reach the correct conclusions. Often companies can have a strategy that is not evidence-based. Becoming data-centric is about being able to recognize effective KPI’s and data about your consumer behavior.

“We try and understand who your customers are and how they interact with your business. Therefore we’ll be quite focused on the customer touch points that’s a really an area that gets us for us,” says Stephen.

Transforming a company to becoming data-centric

Being able to transform a company to be truly data led is not an easy process. Key stakeholders need to be involved and teams need to be able to speak to one another. Ian and Stephen both identified conflicting team goals as one of the reasons companies are not as effective as they could be. 

“What we’re really trying to do by becoming more data-centric is provide rich and broader information such as context around the customer that shows needs and behaviors,” says Stephen “We also look at how to engage the business beyond just one specific channel.”

The initial stage often begins with a data-rich area used to prove the effectiveness of change in one specific area and get buy-in from the larger company stakeholders. Next, is to ask business leaders questions like what do they want to achieve? Where do they want to get to? Where are they currently? All these factors help identify what data the company should be looking at to build its data maturity curve.

The search for personalization

We know from our customers at AB Tasty that personalization is one of the most sought-after features for CX. The way to achieve that is through data. One of the reasons it is so difficult to get right, according to Realise, is that the idea of “personalization” means different things to different people. Ian points out that once you start to personalize, you need to have the resources to create content for each different segment and this, in turn, can lead to some very complicated workflows and messaging. 

“It requires an awful lot of data, thinking and planning because once you’ve started automated personalized columns, it becomes quite complicated quite quickly,” says Stephen.

Both Ian and Stephen are excited about the new technology appearing on the market to support this, but urge caution as to whether this actually improves companies bottom line, efficiency, as well as overall CX.

Customer Lifetime Value

What they do value as a metric is CLV, complementary to looking at your data in a holistic way. As we approach more difficult times for companies, being able to concentrate on giving your brand value is really important. Ian and Stephen are enthusiastic about brands that are less focused on transactional value with their marketing.

Stephen spoke about a brand that looked at the metrics of their mailing over a year to calculate the incremental increase, rather than looking at the transactional value of each one: “Whatever you’re looking at, if you don’t look at that longer-term affinity and engagement for future value, you’re missing a trick.”

Customer Loyalty Schemes are also part of CLV and Realise works hard to help companies improve them. Part of this is being able to understand who your customers are and what value they are looking for from your brand, in addition to identifying the target metrics you hope to achieve through loyalty schemes  Measuring loyalty can be difficult and the cost of running such schemes is often expensive. Companies need to create a business case for it, with clear expectations and markers of success.

The KPI’s for a Data-Centric Company

No two businesses are the same, but we pressed Stephen and Ian to give us an idea of what KPI’s they look for. It is important to see which reports teams are accessing and what metrics they use on a day-to-day basis. To know for certain that companies are looking at future growth, measuring acquisition, churn and NPS is key. 

Engagement is also a crucial metric for parsing Customer Lifetime Value. Stephen adds that Data-centric companies should also look at their spend. Sometimes they look at the profit of a particular action, but don’t actually benchmark to see if they could have achieved more. 

Each company can be different, but you can approach CLV with a different focus each time – your company (how much profit it is making), your customer (how they are behaving) and your staff (do they have the right tools to help them make decisions).

You can find out more about our Customer Centric led by looking at our previous installment on How To Measure Your Digital Impact.