Article

5min read

A/A Testing: A Waste of Time or Useful Best Practice?

A/A TestingA/A testing is little known and subject to strong discussions on its usefulness, but it brings added value for those who are looking to integrate an A/B testing software with rigor and precision.

But before we begin…

What is A/A testing?

A/A testing is a derivative of A/B testing (check out A/B testing definition). However, instead of comparing two different versions (of your homepage, for example), here we compare two identical versions.

Two identical versions? Yes!

The main purpose of A/A testing is simple: verify that the A/B testing solution has been correctly configured and is effective.

We use A/A testing in three cases:

  • To check that an A/B testing tool is accurate
  • To set a conversion rate as reference for future tests
  • To decide on an optimal sample size for A/B tests

Checking the accuracy of the A/B Testing tool

When performing an A/A test, we compare two strictly identical versions of the same page.

Of course, the purpose of an A/A test is to display similar values in terms of conversion. The idea here is to prove that the test solution is effective.

Logically, we will organize an A/A test when we set up a new A/B test solution or when we go from one solution to another.

However, sometimes a “winner” is declared on two identical versions. Therefore, we must seek to understand “why” and this is the benefit of A/A testing.

  • The test may not have been conducted correctly
  • The tool may not have been configured correctly
  • The A/B testing solution may not be effective.

Setting a reference conversion rate

Let’s imagine that you want to set up a series of A/B tests on your homepage. You set up the solution but a problem arises: you do not know to which conversion rate to compare the different versions to.

In this case, an A/A Test will help you find the “reference” conversion rate for your future A/B tests.

For example, you begin an A/A Test on your homepage where the goal is to fill out a contact form. When comparing the results, you get nearly identical results (and this is normal): 5.01% and 5.05% conversions. You can now use this data with the certainty that it truly represents your conversion rate and activate your A/B tests to try to exceed this rate. If your A/B tests tell you that a “better” variant achieves 5.05% conversion, it actually means that there is no progress.

Finding a sample size for future tests

The problem in comparing two similar versions is the “luck” factor.

Since the tests are formulated on a statistical basis, there is a margin of error that can influence the results of your A/B testing campaigns.

It’s no secret how to reduce this margin of error: you have to increase the sample size to reduce the risk that random factors (so-called “luck”) skew the results.

By performing an A/A test, you can “see” at what sample size the test solution comes closest to “perfect equality” between your identical versions.

In short, an A/A test allows you to find the sample size at which the “luck” factor is minimized; you can then use that sample size for your future A/B tests. That said, A/B tests generally require a smaller sample size.

A/A testing: a waste of time?

The question is hotly debated in the field of A/B Testing: should we take the time to do an A/A test before doing an A/B test?

And that is the heart of the issue: time.

Performing A/A tests takes considerable time and traffic

In fact, performing A/A tests takes time, considerably more time than A/B tests since the volume of traffic needed to prove that the two “identical variants” lead to the same conversion rate is significant.

The problem, according to ConversionXL, is that A/A testing is time-consuming and encroaches on traffic that could be used to conduct “real tests,” i.e., those intended to compare two variants.

Finally, A/A testing is much easier to set up on high traffic sites.

The idea is that if you run a site that is being launched or has low traffic, it is useless to waste your time doing an A/A test: focus instead on optimizing your purchase tunnel or on your Customer Lifetime Value: the results will be much more convincing and, especially, must more interesting.

An interesting alternative: data comparison

To check the accuracy of your A/B Testing solution, there is another way that is easy to set up. To do this, your A/B Testing solution needs to integrate another source of analytic data.

By doing this, you can compare the data and see if it points to the same result: it’s another way to check the effectiveness of your test solution.

If you notice significant differences in data between the two sources, you know that one of them is:

  • Either poorly configured,
  • Or ineffective and must be changed.

Did you like this article? We would love to talk to you more about it.

Subscribe to
our Newsletter

bloc Newsletter EN

We will process and store your personal data to send you communications as described in our  Privacy Policy.

Article

8min read

Recommendation vs Personalization

Recommendation and personalization are often wrongly used as interchangeable terms relating to online marketing. They are both essential practices for almost all businesses with an online presence, however. Along with A/B testing and broad optimization techniques, these tools are the future for marketing in the 21st century. While they are complex methods of reaching and retaining customers and visitors, the premise itself is quite simple, and learning how to optimize a website and customer experience is key to helping a business reach its full potential.

The Difference Between Recommendation and Personalization

For most people, personalization and recommendation are the same things. When a business tailors its service to meet our needs it can use recommendations to personalize our interactions, so these words are synonyms, right? Well not exactly.

A recommendation is a form of personalization, but personalization is not a form of recommendation. For example, YouTube might suggest related videos based on previous viewing habits, this is a recommendation based on what other YouTube users also watched. A restaurant, however, might suggest a table by the window based on a previous booking you have made. This is personalization, as it is based on the specific habits of the individual and not a broad algorithm. The more you know about a person, not just their viewing habits, the better. In other words, a recommendation is often built around items, whereas personalization is built around individuals.

There is of course much overlap, and the more informed and well designed a recommendation engine becomes, more on that in a moment, the closer to personalization such methods become. For now, though, it is important to separate the two categories and their techniques by definition and practice to understand fully their implications, potential, and use.

Recommendations

Recommendations are best known to most as algorithms that suggest further content on media websites. The previously mentioned YouTube relies heavily on this model in order to keep users on the site for as long as possible to generate ad revenue.

But the concept of recommendations is not confined to media companies and viewing habits.

There are three main recommendation concepts. Each one of these concepts has their own advantages for specific sectors. These are:

  • Recommendation engines
  • Product recommendation
  • Rating recommendation

Recommendation Engines

Sometimes referred to as a recommender system, recommender engines are the previously mentioned algorithms that are primarily used for media sites. Netflix, for example, might use your previous viewing habits to recommend another series or film. If, for example, you have watched Star Trek, it stands to reason that it will recommend another Sci-Fi series. So far, so simple. However, by tracking the viewing habits of other customers, Netflix might well find that Star Trek viewers are also often interested in nature documentaries. What’s more, specific Star Trek releases, such as the original series, might correlate with specific nature documentaries, such as those related to large predators of the sea.

So how does Netflix find such seemingly unrelated links? By tracking every one of its many millions of viewers. In 2006 Netflix offered a reward of $1m to find the most effective algorithm in tracking and predicting user behavior. The original winners of the prize improved the system by 10%, which may not seem like a lot, but such enhancements are worth enormous sums of money. The better the system worked, the more people joined, the more people joined, the more data Netflix had to work with and the better the system became. This snowball effect has led to them becoming one of the most successful media companies in the world.

But it isn’t only media companies that use recommender engines. To some degree, search engines are recommender engines, filtering out unrelated data to make results more effective.

Product Recommendation

Product recommendations are simply an extension of the recommendation engine’s ability to filter out irrelevant items, but in this case, it is related to products. It requires it’s own category as it relates purchasing items rather than content.

E-Commerce, which also uses many other features of both recommendation and personalization, has always been an innovator in the field of recommendation engines. Most famously, Amazon uses the technique in various ways to increase its sales by a reported 35%. It should be noted that Amazon is notoriously secretive about such data, however, so this is something of an estimation.

The most successful product recommendation engines don’t just provide suggestions on site. Email conversion rates, sales garnered by links sent via emails, are known to be extraordinarily high for companies like Amazon. This is partly made possible by the data collected by recommendation engines and well-targeted campaigns.

Rating Recommendations

Rating recommendations work across all sectors, or at least they can in theory. The previously mentioned Netflix and Amazon both have rating systems that provide feedback from other customers. For Amazon, it is the ubiquitous star rating, where users rate each product out of five. For Netflix, this is a thumbs up or thumbs down rating, which also helps the recommendation engine filter out specific suggestions, making the algorithm more personalized.

Rating recommendations are sometimes referred to as “Implicit feedback” (which also includes comments). Surveys have shown that the vast majority of users, 88% according to a BrightLocal survey in 2014, are influenced as much by this feedback as a personal recommendation from a friend.

Personalization

Personalization, unlike recommendation, is only at the beginning of its potential.

This is partly because the more a company knows about a person, the more effective it becomes. What is sometimes referred to as one to one marketing, it’s ambition and scope could change the way we interact with technology forever.

In truth, right now the technology available isn’t capable of collating the individual data to reach anything like the potential businesses crave, and there are many issues relating to privacy that the Internet is still coming to terms within its relative infancy. It seems, however, that in the future businesses will likely be using some form of personalization.

It should be noted that there are two types of personalization, product, and website. These are two very different concepts.

Product Personalization

Product personalization is a much simpler concept and one that most of us have used, or at least been aware of, for some time. A common example would be choosing the color of a piece of clothing from a varied selection, such as a shoe. Sometimes product personalization can become quite detailed, allowing customers to construct a product almost from scratch.

Website Personalization

Website personalization, by contrast, uses the complex, big data. The devil is in the detail, and the detail can be minute. What sets it apart from the recommendation is this data is personal.

Age, gender, location, the shopping habits and ratings left on websites, social media likes, incentives that might have been successful from other marketing campaigns, the time of day and even how the weather is at that current moment all can be taken into consideration.

This all begs the question, how does personalization help? Firstly, people are bombarded with images and information every time they go online. Usually, at best, this vaguely relates to some interest or other, at worst it is an irrelevant distraction. As seen above, this information can help websites target offers, present the customer with the most relevant and helpful information and suggest the most likely products they might wish to purchase.

App personalization uses the same principles as website personalization. At this point, app personalization is quite some way behind website personalization in terms of its development, but the gap is narrowing as more and more businesses become aware of its advantages.

Optimization

The only way in which a business can be sure that their recommendation engine or personalization system is as effective as it can be is by A/B testing. This is part of the process of optimization called recommendation testing. AB Tasty provides recommendation tools that pinpoint optimization and filter out those that are ineffective. Every aspect of website interaction can be improved, leading to a higher return on investment. Server-side A/B testing also gives you more flexibility to test recommendation algorithms.

Advantages of optimization include:

  • Better experience for the customer, cultivating customer retention
  • Higher basket totals, more sales per visit
  • Focused recommendation for better conversion rates
  • Improved general content, better website ranking
  • Greater return on investment
  • Focused Email campaigns
  • Improved leads and ranking
  • Attracting newcomers
  • Filtering out unwanted ads (SPAM)
  • Increasing basket totals

Each and every aspect of optimization has the potential to focus the attention on what is working for a website and what can be improved upon. Several metrics, both simple and complex, can form a formidable and deeply insightful method for optimization, for all business when applies correctly.