A/B, Split or Multivariate Test: How to Choose the Right One

In digital marketing, you should always aim for the best user experience. Don’t put up with status quo – every marketing strategy could be better. There might be a hidden opportunity for your marketing strategy just around the corner. In this blog post, we will explain to you how to explore the unknown sea of online possibilities in a scientific way.

With analytics data and a little bit of creativity, you should be able to get some ideas about how to optimize your conversion rate on a website or the landing pages you are using in your campaigns. To get structured and streamlined data from your assumptions you need to conduct testing. Marketing professionals are using A/B testing, split testing, multivariate testing, and multipage testing to grow conversion rate, website and landing pages performance. As Justin Cutroni, Analytics Advocate at Google, recalls, “testing is critical because it takes the opinions out of the decision-making process”.

Even though testing follows a scientific method, there is no need for a degree in statistics with AB Tasty. This article will help you understand the best practices, benefits, and limitations of each of these testing methods so you can make the right choice and conduct a successful testing campaign.

A/B testing, an easy and powerful method

A/B testing (or A/B/n testing) is a method of website optimization in which you are comparing two (A/B) or more versions of the same page (A/B/n) – by looking at the conversion rates and metrics that matter to your business (clicks, page views, purchases…) using live traffic. A/B/n testing is used when you need to test more than two content variations. A/B/n method will allow you to test three or more variations of a page or landing page – instead of testing only one variation against the control version of the page.

Of course, the variation can appear on one page or on a group of pages. For instance, if you change your product page title formatting, you can target all your product pages at once. They will all be considered as variations of the same group.

Doing A/B tests and split tests are essentially the same concept.  “A/B” refers to the two variations of the same URL where changes are made “live” using Javascript on the original page. SaaS tools that provide you with a visual editor like AB Tasty allow you to create these changes quickly without technical knowledge, whereas “split” refers to the traffic redirection towards one variation or the other(s), each hosted on its own URL and fully redesigned in the code. For a complete redesign with a lot of changes, it may be faster to recode the page; but for most tests, the visual editor is enough. You can, therefore, consider A/B tests to work the same as split tests.

The variation page may be different in many aspects, depending on the testing hypothesis you put forth and your industry goals: layout, design, pictures, headlines, sub-headlines, call to actions, offers, button colors, etc.

The number of conversions on each page’s version is compared once each variation gets enough visitors.

In A/B tests only the impact of the design as a whole is tracked, not individual elements that are changed – even though many design elements might be changed on variation page version simultaneously.

TIP: Keep in mind that testing is all about comparing the performances of variations. Of course, you’ll have to change the same element in each variation. It is also recommended not to make too many changes between the control and variation versions of the page. You should try to make just a few important or radical changes, so you will be able to understand the reasons behind the (good or bad) results of the experiment. In the long term, continuous improvement process will lead to better and lasting performance.

To become an expert in A/B Testing, don’t forget to download our A/B Testing Ebook.

When to use A/B tests?

A/B tests are a great method to test radically different ideas for conversion rate optimization (complete redesign) or small changes on a page. A/B testing is the right method to choose if you don’t have a large amount of traffic to your site – because A/B tests can deliver reliable data very quickly, without a large amount of traffic. In the case of low traffic and only a few variations of the page, you should opt for A/B testing over other methods. A/B tests are a great method to maximize test time to achieve fast results.

If you have a high-traffic website, you can evaluate the performance of a much broader set of variations. However, there is no need to test 20 different variations of the same element, even if you have the adequate traffic! This means you have no strategy and that your testing strategy relies on chance.

Multivariate testing: understand the cause and effect

Multivariate tests or multi-variant tests are the same as the A/B test in their core mechanism and philosophy. The difference is, of course, that the multivariate testing allows you to compare a higher number of variables and the interactions between each other, and to test changes to multiple sections on a single page.

For multivariate testing you need to identify a few key page sections and then create variations for those sections specifically – you aren’t creating variations of a whole page as is the case with A/B testing.

TIP: Use multivariate testing when several elements combinations on your website or landing page are called into question.

Multivariate testing reveals more information about how these changes to multiple sections interact with one another. In multivariate tests website traffic is split into each possible combination of a page – where the effectiveness of the changes is measured.

TIP: Use multivariate testing to optimize an existing website or landing page without doing significant investment in redesign when elements are supposed to interact with each other.

Keep in mind that the multivariate testing is more complicated than the A/B testing. Multivariate tests are the best suited for more advanced marketing testers because they give far more possible combinations of experience to the visitor on your website or landing page. Too many changes on a page at once can quickly add up – and you’re left with a very large number of possible combinations that must be tested.

Multivariate test example

You decide to run a multivariate test on one of your landing pages. You decide to change two elements on your landing page. On the first version, you add a video instead of the image. On the second version, you add a slider instead of the image.

For each page variation, you add another version of the headline. This means that now you have 3 versions of the main content and 2 versions of the headline = 6 different combinations of the landing page.

When to use multivariate testing?

Multivariate tests are recommended for the sites that have a large amount of daily traffic – you will need a site with a high volume of traffic to test multiple combinations, and it will take a longer time to obtain meaningful data from the test.

With the multivariate testing method, you can make subtle changes to your website or landing pages and test how these elements interact with one another. This will allow you to incrementally improve on an existing design, while the test results can be used to apply to a larger website or landing page redesign.

Multipage testing: ensure a consistent user experience

Multipage testing is a testing method similar to standard A/B testing. In A/B testing, changes can be made to one specific page or on a group of pages. If the changed element appears on several pages, you can choose whether to change it on each page or not. However, if the element is on several pages but not quite identical, appears at a different place or has a different name, you’ll have to set up a multipage test.

Multipage tests allow you to conduct this kind of experiments: the changes you make are implemented consistently over several pages. This means that multipage tests allow you to link together variations of different pages and are especially useful when funnel testing. In multipage tests, site visitors are funneled into one funnel version or the other. You need to track the way visitors interact with the different pages they are shown, so you can determine which funnel variation is the most effective.

TIP: You must ensure that the users see a consistent variation of changes throughout a set of pages. This is the key to getting usable data and allows one variation to be fairly tested against another.

Multipage test example

You conduct a multipage test with a free shipping coupon displayed in the funnel at different places against the original purchase funnel without a coupon. For example, you could offer visitors a free shipping coupon on a product category page – where can see “Free shipping over €50 spent” as a static banner on the page. Once the visitor adds a product into the shopping cart he goes to the shopping cart page. From there he can continue shopping or go to the checkout. In both cases, you can show him a new dynamic message – “Add €X to the cart and get the free shipping” if the cart is under 50€ (where X is the difference between €50 and the cart value). Again, you can experiment where the message will be placed – near the “Proceed to checkout” button, near the “Continue shopping” button, near the shipping cost for his order or somewhere else – and with the call-to-action variations of the message.

This kind of test will help you understand visitors’ purchase behavior better – how is a free shipping coupon displayed at the right place and time-related to reduced shopping cart abandonment rate and increased sales. After enough visitors come to the end of the purchase funnel through the different designs, you will be able to compare the effect of design styles – easily and effectively.

How to test successfully?

Remember that the pages being tested need to receive substantial traffic so the tests will give you some relevant data to analyze. Here are a few keys to understand for how long you should run a test and how statistics calculation works with AB Tasty.

Whether you will be using A/B testing, split testing, multivariate testing or multipage testing to increase your conversion rate, website, and landing pages’ performance, remember to use them correctly. Each type of test has its own requirements and is adapted to specific situations, with advantages and disadvantages. All tests are powerful optimization methods that complement one another.  Using the right test for the right situation will help you get the most out of your site and the best return on investment for your testing campaign.