What is A/B Testing?
A/B testing, also known as split testing, is a marketing technique that involves comparing two versions of a web page or application to see which performs better. These variations, known as A and B, are presented randomly to users. A portion of them will be directed to the first version, and the rest to the second. A statistical analysis of the results then determines which version, A or B, performed better, according to certain predefined indicators such as conversion rate.
In other words, you can verify which version gets the most clicks, subscriptions, purchases, and so on. These results can then help you optimize your website for conversions.
In other words, you can verify which version gets the most clicks, subscriptions, purchases, and so on. These results can then help you optimize your website for conversions.
Table of contents
- The Ultimate A/B Testing Guide
- What is A/B Testing?
- A/B testing examples
- What types of websites are relevant for A/B testing?
- What A/B tests should you use?
- A/B testing and conversion optimization
- How to find A/B test ideas?
- What should you A/B test on your website?
- Tips and best practices for A/B testing
- Choosing an A/B testing software
- Understanding A/B testing statistics
- Other forms of A/B testing
- The best resources on A/B testing and CRO
What types of websites are relevant for A/B testing?
Any website can benefit from A/B testing since they all have a ‘reason for being’ – and this reason is quantifiable. Whether you’re an online store, a news site, or a lead generation site, A/B testing can help in various way. Whether you’re aiming to improve your ROI, reduce your bounce rate, or increase your conversions, A/B testing is a very relevant and important marketing technique.
Lead
The term “lead” refers to a prospective client when we’re talking about sales. E-mail marketing is very relevant to nurturing leads with more content, keeping the conversation going, suggesting products, and ultimately boosting your sales. with A/B testing e-mails, your brand should start to identify trends and common factors that lead to higher open and click-through rates.
Media
In a media context, it’s more relevant to talk about “editorial A/B testing”. In industries that work closely with the press, the idea behind A/B testing is to test the success of a given content category. For example, if you want to see if it’s a perfect fit with the target audience. Here, as opposed to the above example, A/B testing has an editorial function, not a sales one. A/B testing content headlines is a common practice in the media industry.
E-commerce
Unsurprisingly, the aim of using A/B testing in an e-commerce context is to measure how well a website or online commercial app is selling its merchandise. A/B testing uses the number of completed sales to determine which version performs best. It’s particularly important to look at the home page and the design of the product pages, but it’s also a good idea to consider all the visual elements involved in completing a purchase (buttons, calls-to-action).
What A/B tests should you use?
Classic A/B test: The classic A/B test presents users with two variations of your pages at the same URL. That way, you can compare two or several variations of the same element.
Split tests or redirect tests: The split test redirects your traffic toward one or several distinct URLs. If you are hosting new pages on your server, this could be an effective approach.
Multivariate or MVT test: Lastly, multivariate testing measures the impact of multiple changes on the same web page. For example, you can modify your banner, the color of your text, your presentation, and more.
In terms of A/B testing technology, you can:
A/B test on websites
By A/B testing on the web, you can compare two versions of your page. After, the results are analyzed according to predefined objectives—clicks, purchases, subscriptions, etc.
A/B test native mobile apps
A/B testing in apps is complex because you can’t show two versions after download. Yet, quick updates allow easy design changes and direct impact analysis.
Server-side A/B test via APIs
An API is a programming interface for connecting with applications to exchange data, allowing automatic campaign creation or variation from stored data.
Choosing an A/B testing software
Choosing the best A/B testing tool is difficult.
We can only recommend you use AB Tasty. In addition to offering a full A/B testing solution, AB Tasty offers a suite of software to optimize your conversions. You can also personalize your website in terms of numerous targeting criteria and audience segmentation.
But, in order to be exhaustive, and also to provide you with as much valuable information as possible when it comes to choosing a vendor, here are a few articles to help you choose your A/B testing tool with software reviews:
Understanding A/B testing statistics
The test analysis phase is the most sensitive. The A/B testing solution must at least offer a reporting interface indicating the conversions saved by variation, the conversion rate, the percentage of improvement compared with the original, and the statistical reliability index saved for each variation. The most advanced ones narrow down the raw data, segmenting results by dimension (e.g. traffic source, geographical location of visitors, customer typology, etc.).
Before it is possible to analyze test results, the main difficulty involves obtaining a sufficient level of statistical confidence. A threshold of 95% is generally adopted. This means that the probability that result differences between variations are due to chance is very low. The time necessary to reach this threshold varies considerably according to site traffic for tested pages, the initial conversion rate for the measured objective, and the impact of modifications made. It can go from a few days to several weeks. For low-traffic sites, it is advisable to test a page with higher traffic. Before the threshold is reached, it is pointless to make any conclusions.
Furthermore, the statistical tests used to calculate the confidence level (such as the chi-square test) are based on a sample size close to infinity. Should the sample size be low, exercise caution when analyzing the results, even if the test indicates a reliability of more than 95%.
With a low sample size, it is possible that leaving the test active for a few more days will greatly modify the results. This is why it is advisable to have a sufficiently sized sample. There are scientific methods to calculate the size of this sample, but from a practical standpoint, it is advisable to have a sample of at least 5,000 visitors and 75 conversions saved per variation.
There are two types of statistical tests:
- Frequentist tests. The chi-square method, or Frequentist method, is objective. It allows for analysis of results only at the end of your test. The study is thus based on observation, with a reliability of 95%.
- Bayesian tests. The Bayesian method is deductive. By taking from the laws of probability, it lets you analyze results before the end of the test. Be sure, however, to correctly read the confidence interval. Check out our dedicated article to see all there is to know about the advantages of Bayesian statistics for A/B testing.
Lastly, although site traffic makes it possible to quickly obtain a sufficiently sized sample, it is recommended that you leave the test active for several days to take into account differences in behavior observed by weekday, or even by time of day. A minimum duration of one week is preferable, ideally two weeks. In some cases, this period can even be longer, particularly if the conversion concerns products for which the buying cycle requires time (complex products/services or B2B). As such, there is no standard duration for a test.
Other forms of A/B testing
A/B testing is not limited to modifications to your site’s pages. You can apply the concept to all your marketing activities, such as traffic acquisition via e-mail marketing campaigns, AdWords campaigns, Facebook Ads, and much more.
Resources for going further with A/B testing:
- A Beginner’s Guide to A/B Testing Your Email Campaigns
- 7 Tips for Implementing A/B Testing in Your Social Media Campaigns
- A Crash Course on A/B Testing Facebook Ad Campaigns
- How to Improve Your Click Rates with AdWords A/B Testing
The best resources on A/B testing and CRO
We recommend you read our very own blog on A/B testing, but other experts in international optimization also publish very pertinent articles on the subject of A/B testing and conversion more generally. Here is our selection to stay up to date with the world of CRO.
Blogs to bookmark: