The complete guide to A/B testing​

A/B testing, also known as split testing, is a marketing technique that involves comparing two versions of a web page or application to see which performs better. These variations, known as A and B, are presented randomly to users. A portion of them will be directed to the first version, and the rest to the second. A statistical analysis of the results then determines which version, A or B, performed better, according to certain predefined indicators such as conversion rate. In other words, you can verify which version gets the most clicks, subscriptions, purchases, and so on. These results can then help you optimize your website for conversions.

A/B testing examples

Many of you are looking for ideas for your next A/B tests. If you have been reading the previous chapters, you know there is no magic bullet and that textbook cases are site-specific. However, since you just can’t help yourself, here are some links to a few examples. More examples of A/B tests and results

• 5 A/B Test Case Studies and What You Can Learn From Them?
• 100 Conversion Optimization Case Studies
• 50 A/B Split Test Conversion Optimization Case Studies

What type of websites are relevant for A/B testing ?

Any website can benefit from A/B testing, since they all have a ‘reason for being’ – and this reason is quantifiable. Whether you’re an online store, a news site or a lead generation site, you are aiming to improve your conversion rate, whatever kind of conversion that may be.

Lead

The term “lead” is used to mean a sales leads, or a prospective client. Especially relevant here are e-mails sent in order to boost sales. In this case, A/B testing makes use of information about the nature of the people contacted, like their sex or age range.

Media

In a media context, it’s more relevant to talk about “editorial A/B testing”.  In industries that work closely with the press, the idea behind A/B testing is to test the success of a given content category – for example, to see if it’s a perfect fit with the target audience.  Here, as opposed to the above example, A/B testing has an editorial function, not a sales one. A/B testing content headlines is a common practice in the media industry.

E-commerce

Unsurprisingly, the aim of using A/B testing in an e-commerce context is to measure how well a website or online commercial app is selling its merchandise. A/B testing uses the number of completed sales to determine which version performs best. It’s particularly important to look at the home page and the design of the product pages, but it’s also a good idea to consider all the visual elements involved in completing a purchase (buttons, calls-to-action).

What A/B tests should you use?

There are several types of A/B tests. You should choose the one that best fits your particular situation.

• Classic A/B test. The classic A/B test presents users with two variations of your pages at the same URL. That way, you can compare two or several variations of the same element.
• Split tests or redirect tests. The split test redirects your traffic towards one or several distinct URLs. If you are hosting new pages on your server, this could be an effective approach.
• Multivariate or MVT test. Lastly, multivariate testing measures the impact of multiple changes on the same web page. For example, you can modify your banner, the color of your text, your presentation, and more.

In terms of technology, you can:

• Use A/B testing on websites. A/B testing on the web makes it possible to compare a version A and B of a page. After this, the results are analyzed according to predefined objectives—clicks, purchases, subscriptions, and so on.
• Use A/B testing for native mobile iPhone or Android applications.   A/B testing is more complex with applications. This is because it is not possible to present two different versions once the application has been downloaded and deployed on a smartphone. Workarounds exist so that you can instantly update your application. You can easily modify your design and directly analyze the impact of this change.
• Use server-side A/B testing via APIs. An API is a programming interface that enables connection with an application for data exchange. APIs let you automatically create campaigns or variations from saved data.

It is possible to test on multiple devices with solutions like AB Tasty.

A/B testing and conversion optimization

Conversion optimization and A/B testing are two ways for companies to increase profits. Their promise is a simple one: generate more revenues with the same amount of traffic. In light of high acquisition costs and complex traffic sources, why not start by getting the most out of your current traffic?

Amazon is very familiar with A/B testing – they’re constantly testing to improve UX and conversion rates 

Surprisingly, average conversion rates for e-commerce sites continue to hover between 1% and 3%. Why? Because conversion is a complex mechanism that depends on a number of factors, including the quality of traffic generated, user experience, offer quality, the website’s reputation, as well as what the competition is doing. E-commerce professionals will naturally aim to minimize any negative impact the interplay of the above elements might have on consumers along the buyer journey.  A variety of methods exist to help them achieve this, including A/B testing, a discipline that uses data to help you make the best decisions. A/B testing is useful to establish a broader conversion optimization strategy, but it is by no means sufficient all on its own.  An A/B testing solution lets you statistically validate certain hypotheses, but alone, it cannot give you a sophisticated understanding of user behavior. However, understanding user behavior is certainly key to understanding problems with conversion. It’s therefore essential to enrich A/B testing with information provided by other means. This will allow you to gain a fuller understanding of your users, and crucially, help you come up with hypotheses to test. There are many sources of information you can use to gain this fuller picture:

• Web analytics data. Although this data does not explain user behavior, it may bring conversion problems to the fore (e.g. identifying shopping cart abandonment). It can also help you decide which pages to test first.
• Ergonomics evaluation. These analyses make it possible to inexpensively understand how a user experiences your website.
• User test. Though limited by sample size constraints, user testing can provide a myriad of information not otherwise available using quantitative methods.
• Heatmap and session recording. These methods offer visibility on the way that users interact with elements on a page or between pages.
• Client feedback. Companies collect large amounts of feedback from their clients (e.g. opinions listed on the site, questions for customer service). Their analysis can be completed by customer satisfaction surveys or live chats.

Recommended solutions for conversion optimization:

Web Analytics	Heatmaps	Session Recording
Live Chat	User Tests	User Feedback

How to find A/B test ideas?

Your A/B tests must be complemented by additional information in order to identify conversion problems and offer an understanding of user behavior. This analysis phase is critical, and must help you to create “strong” hypotheses. The disciplines mentioned above will help. A correctly formulated hypothesis is the first step towards a successful A/B testing program and must respect the following rules. Hypotheses must:

• be linked to a clearly discerned problem that has identifiable causes
• mention a possible solution to the problem
• indicate the expected result, which is directly linked to the KPI to be measured

For example, if the  identified problem is a high abandon rate for a registration form that seems like it could be too long, a hypothesis might be: “Shortening the form by deleting optional fields will increase the number of contacts collected.”

Use the entire array of solutions available to understand your users’ obstacles

More articles about formulating test hypotheses:

• How to Formulate a Strong A/B Test Hypothesis?
• 7 Steps to Formulate a Strong A/B Test Hypothesis
• A/B Testing: Example of a good hypothesis

What should you A/B test on your website?

What should you test on your site? This question comes up again and again because companies often don’t know how to explain their conversion rates, whether good or bad. If a company could be sure that their users were having trouble understanding their product, they wouldn’t bother testing the location or color of an add-to-cart button – this would be off topic.  Instead, they would test various wordings of their customer benefits. Every situation is different. Rather than providing an exhaustive list of elements to test, we preferred to give you an A/B testing  framework to identify these elements. Below are some good places to start: