Tag: EN

Article

Feb 26, 2016

9min read

5 A/B Test Case Studies and What You Can Learn From Them

AB Tasty

AB Tasty’s note: This is a guest post by Umesh Kumar, digital marketing evangelist at Designhill.

A/B Testing isn’t a rocket science to understand and implement. It is just about testing two different versions of a page on your site to see which one attracts more audience. More than anything else, this test helps you know and understand your consumers better. After you run an A/B test, you will find that a few more have added in your earlier list of customers.

It surely is one of the best ways to improve your conversion rates. In fact, an article published in CrazyEgg.com reveals that using correct testing methods can increase conversion rates up to 300 percent. But it’s shocking that still, the majority of marketers choose not to use A/B test experiments. Don’t believe us? Check out the following stats:

For most businesses, conversion rates range from just 1 to 3%. (Source: truconversion.com)
Most companies spend $92 to acquire customers, but only $1 to convert them. (Source: tweakyourbiz.com)
Only 22% of businesses are satisfied with their conversion (Source: Contentsquare.com)
44% of companies use A/B testing when trying to optimize conversion rates. (Source: Unbounce.com)
85% of marketers plan to focus more on conversion rate optimization this year. (Source: Truconversion.com)

Given these stats, it’s no surprise that many marketers steer clear of A/B testing for optimizing their site. But, how exactly can you optimize your conversions with A/B Testing? The answer is simple! Why not do what smart marketers do? Learn lessons from companies that have emerged as shining examples of A/B testing genius.

No matter what is the nature of your business, there is no harm in taking a step back and learning from others’ achievements. To help you, we have listed 5 classic case studies that will provide you with interesting test hypotheses and give you an insight on what and how visitors think. You can learn a lot from these case studies and use the learning to take on the conversion challenges in your way to success headfirst. These examples are quite simple to implement with any A/B testing tool.

Case Study 1: Conversions improve by 11.5% by Adding FAQs, Statistics and Social Proof on Websites

Test Carried Out By Kiva.org, an innovative non-profit organization, allows people to lend money via the Internet to low-income entrepreneurs and students across countries. Kiva conducted an A/B Test as they wanted to increase the number of donations from first-time visitors to their landing page.

Hypothesis: Giving more information to visitors coming to Kiva’s landing page will help boost the number of donors.

Result: Donations increased by 11.5% after adding an information box at the bottom of the landing page.

Version A – original: (left)

Version B: Addition of information box (FAQ, social proof & statistics)

ab-test-1

What You Can Learn from This Test?

Ensure that your landing page is designed in such a way that it answers all questions that a visitor may have. In this case, the information box at the bottom of the page helped the organization explain about themselves and their services, providing statistics. The information increased their trustworthiness and credibility as a site.

Case Study 2: 2.57% increase in Open Rates 5.84% Higher Click-through Rate (CTR) by Changing the Subject Line of an Email

Test Carried Out By Designhill.com, one of the fastest-growing peer-to-peer crowdsourcing platforms that connect graphic artists with design seekers. They did an email blast a few days before Christmas to promote its content and increase the click-through rate.

Hypothesis: Just mentioning the title of the blog in the subject line of the email would get the majority of click-through rather than requesting recipients to review the post with the blog’s title.

Just writing “Top 10 Off-Page SEO Strategies for Startups in 2015” in the subject line of the email would get the majority of click-through rather than writing “Check out My Recent Post – Top 10 Off-Page SEO Strategies for Startups in 2015”.

Result: The company was able to score 5.84% higher CTR and 2.57% higher open rate by including just the title of the blog in the subject line.

img_56cf0fba1f873

What You Can Learn From This Test:

Your subject line is the first thing the recipient of your email sees. Therefore, your subject line must have the power to entice the readers to open the mail and know more about your products or services. Because after all, it doesn’t really matter what your offer is if it is not opened by your readers. Therefore, choose your words wisely as they will have a higher impact on open rates and click-through. But it’s not only important to ensure great subject lines, but you must also ensure that your logo design is laid out in a way that vital information pop-up. For example, your logo design and contact details must be easily locatable. For another, CTAs and other links must be out of clutter. Read our Beginner’s Guide to A/B Testing your Emails.

Case Study 3: 49% Increase in CTR by Adding Text in the Call-to-Action Button

Test carried out by Fab, an online community whose members can buy and sell apparel, home goods, accessories, collectibles, etc.

Hypothesis: Making the “Add to Cart” button clearer (by adding text) will lead to an increase in the number of people adding items to their shopping carts.

Result: There was an increase of 49% in CTR over the original after the text “Add to Cart” was included in the CTA button rather than just an image or symbol.

In the following image, you’ll see that the original design (on the far left) features a small shopping cart with a “+” sign and no text. The two versions (middle and right) added text-based designs. Version A helped increase cart adds by 49% over the original.

img_56cf0fdd0a773

What You Can Learn From this Test:

Text connects better with visitors than images or symbols which may confuse them. Therefore, try and have a direct and clear CTA which will help consumers know of their actions.

It makes no sense to have a CTA that your visitors don’t understand. They don’t get to know what the button does actually.

Case Study 4: Conversion Rate Improved by 7.46% by Adding an Image Rather than a Blank Background.

Test Carried Out By A company who wanted to check if customers get attracted to a blank background or one with pictures more.

Hypothesis: Using a photo in the background will lead to more conversions than a blank background.

Result: The conversion rate of the background with a photo was 25.14% as compared to 7.68% for the one without a photo.

img_56cf1044b8de0 img_56cf101ca63db

What you can learn from this test:

You must have heard that a picture is worth a thousand words. People love visuals and there could be no other better place than your site to impress them with the pictures of your products. Having an image of your product or service in the background of your site can drive conversions as they get to see (visualize) what they’ll be getting. Images hold the power to grab the attention of your audience and turn them into customers.

Case Study 5: Leads Increase by 256% after Having a Mobile-Optimized Landing Page

Test Carried Out By Rasmussen College, a for-profit private college and Public Benefit Corporation, who wanted to increase leads from Pay-Per-Click traffic on their mobile site.

Hypothesis: Creating a new mobile-friendly website, featuring a click-through menu, will improve conversions.

Result: Conversions increased by 256% after a new mobile-only (mobile responsive) site was made.

img_56cf10a3b0a5b

What you can learn from this test:

Hammad Akbar, founder of TruConversion.com says that “Unpleasant mobile experiences and a lack of mobile-friendliness makes 48% users believe that the company doesn’t care about their business.” A mobile-responsive website enhances the browsing experience of the site visitors. It is essential if you don’t want to lose customers just because your site took time to load. Keep the design of your site simple with only the basic information on the first page. Try and find different way of improving mobile navigation such as having a drop-down menu.

Conclusion

We hope that after reading this post, you are inspired to hold some amazing A/B tests on your own site. It is actually exciting and amazing to see what your customers like or dislike. But don’t forget that these tests are mere a guiding tool and nothing can replace your own tests and judgments about your visitors as well as the site. Remember, there is always a scope for improvement.

So, happy testing!

Download our guide to learn all there is to know about A/B testing!

Author Bio: Umesh Kumar is a digital marketing evangelist who works with Designhill, the fastest-growing custom design marketplace, to develop and execute their marketing strategies. He started his online marketing career in 2008 and is actively engaged within internet business modeling, website development, social networks, lead generation, search engine optimization, and search engine marketing. In addition, he loves blogging and shares his expertise about tips, tricks, and upcoming trends in the world of digital marketing. Get in touch Facebook | Twitter | Google+

You might also like...

See all

Article

6min read

Minimal Detectable Effect: The Essential Ally for Your A/B Tests

Hubert Wassner

May 6, 2025

Article

2min read

How We Made AB Tasty’s Feature Experimentation SDK OpenFeature-Compatible — And Even Easier to Adopt

Julie Dumont

Apr 30, 2025

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Subscribe to
our Newsletter

Article

Jan 9, 2015

4min read

User Testing and A/B Testing… Strengths and Differences

Anthony Brebion

Companies often opt for one or the other of these tests without realizing the benefits of using them together. Given that the benefits of one test may sometimes outweigh the disadvantages of the other, the lessons from user testing and A/B testing complement each other to refine your conversion optimization strategy.

All sites are trying to attract more traffic, but they are also seeking to increase their conversion rate in order to raise profits. The understanding and optimization of their customer paths are therefore essential. In order to understand the problems faced by their websites, it is possible to design a series of tests, such as user testing and A/B testing.

User testing

User testing, or usability testing, makes it possible to understand the needs of users and observe their behavior. This data is qualitative and offers a rich, detailed and very helpful source of information which provides the company with an understanding of what works or does not work in its offer.

Advantages

Provides better knowledge of the user as a person (and not just as a virtual visitor).
Offers comprehensive and often unsuspected sources of data through which the company discovers new areas of improvement.
Always offers a workable response to improve the user experience.

Weaknesses

Significant investment of time and money for purely theoretical results.
Observed results are not quantified and do not allow the company to estimate the revenue potential of the test findings.
Returns are based on the subjective behavior and opinion of a few users.

A/B testing

A/B tests are based on an idea pre-conceived by the company, which seeks to test and quantify the effectiveness or otherwise of a proposed change. They are based on specific, targeted changes whose outcomes will be automatically quantified.

Advantages

Rapid implementation and real-time monitoring (number of clicks, conversion rates, heatmap).
Quantified and representative results (KPIs and ROI simulation).
Allows significant flexibility in changing one or more points on a page.
Very cheap and requires minimal effort in the long term.

Weaknesses

Significant advance preparation (targeting areas to change/watch).
Not all tests always produce results.
Requires a large number of visits to statistically validate the assumptions.

Different but complementary tests

The choice between these two tests mainly depends on the question you want to answer: How much? Why? If your question concerns quantifiable results regarding a specific change on your website, then you must perform an A/B test. If the purpose of your question is to understand your consumers’ behavior, usability testing will better meet your needs.

Although users testing and A/B testing differs in their setup and objectives, their features can easily complement each other. It is recommended that you use both types of tests sequentially. Starting with a usability test allows you to locate the points of friction to address. You can then conduct an A/B test to validate the relevance of the changes you are considering.

The most important prerequisite for achieving valid results from your A/B testing is to build strong test hypotheses that will involve the modification of elements that are actually used in the conversion process or really hamper the process. Find out more about the formulation of test hypotheses.

By combining the two methods, it is possible to make reasonable use of your time and money, while obtaining usable quantitative and qualitative results. Using a tool such as A/B testing is advisable only when an idea for improvement has been identified. If you lack ideas, your users may not – test them!

You might also like...

See all

Article

6min read

Minimal Detectable Effect: The Essential Ally for Your A/B Tests

Hubert Wassner

May 6, 2025

Article

2min read

How We Made AB Tasty’s Feature Experimentation SDK OpenFeature-Compatible — And Even Easier to Adopt

Julie Dumont

Apr 30, 2025

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Subscribe to
our Newsletter

Article

Sep 2, 2014

4min read

Eight Mistakes not to Make in A/B Testing

Anthony Brebion

The following mistakes can completely distort the results of your A/B testing program. Learn how to avoid them!

1. Changing the control version during the test

Making changes to the control version (i.e. the original version of your page), especially the items you are testing, while the test is in progress, is to be strenuously avoided. The results of the variation will mean nothing if the original version on which the comparison is based has also changed.

2. Testing your changes over different time periods

The whole purpose of A/B testing is to divide your traffic between two versions of your page: a Control version A, and a Variation version B. Collecting data variation on Version A for some time and then doing the same on Version B is not A/B testing and will yield unusable results, because the test conditions have changed! There is no assurance that the traffic you will have fully directed to Version A and then to version B is the same; quite the contrary. Many other factors may come into play: if you test Version A for a normal working period and then test Version B during the holidays, the results you get will have been distorted by changes in timetables and user habits. For these reasons, one of the golden rules of A/B testing is that the two versions must be tested simultaneously: by deploying the variation alongside the control and by distributing the traffic between the two, you will be sure that any factors affecting traffic will affect both versions. You can then confidently learn from your results.

3. Simultaneously editing multiple variables

Actual A/B testing consists in testing a variation of a page that differs in one way only: a test, a variable. Testing several variations of several different elements is no longer A/B testing, but multivariate testing, a significantly more advanced technique. If you set an A/B test with several simultaneous variables, you will have great difficulty, once the test is complete, in determining precisely what variable(s) is(are) responsible for the results you have achieved. If you want to test several elements on a page using A/B tests, first test the most radical changes, then refine the changes with each successive test. Alternatively, you can use multivariate testing.

4. Drawing conclusions before achieving 95% reliability

In the AB Tasty reporting tool, each test is assigned a statistical reliability rate. This is a confidence level whose calculation depends in part on the duration of the test and on the traffic assigned. What you must bear in mind is the figure of 95%: if your test has a reliability rate of 95% or more, you can consider the results reliable. Below this rate, the results you get are just not reliable enough to draw definitive conclusions. If you take a decision on the basis of results without at least 95% reliability, you are relying on potentially false results; it is entirely possible that they will subsequently change.

5. Allowing a test to run for too long

Conversely, if your test has reached 95% statistical reliability and you have enough information to learn lessons, it is best not to let the test go on for too long. Stop your tests when they meet their target! Otherwise, you could waste time waiting for a marginal improvement of already good results and you expose some of your traffic to a variation that is already tested, whereas these visitors could be contributing to new tests. How long should I run my A/B tests?

6. Not being honest when interpreting data

For your A/B testing to be truly effective, you must accept the results, whatever they are (provided of course they have reached 95% statistical reliability). Of course, this is easier said than done when you see the variation that you spent weeks designing fail against the control version! The importance of A/B testing is precisely to tell you unambiguously which page version works best with your users, even though it was not your preferred version.

7. Surprising your most loyal visitors

If you have a high rate of repeat visitors, it is better not to surprise them too suddenly with a radically different variation. Your loyal visitors are regulars whom an abrupt change to your site could drive away, even though the variation in question may be not the one ultimately chosen. In addition, if you test your new visitors, you’ll be sure that their behaviour will not be biased.

8. Choosing your KPIs poorly

Finally, be sure to choose KPIs that are truly representative of your goals. If you choose a KPI that is too far from the goal, you will get a numerical result with no great connection to your goal. Another common mistake is to take only one KPI whereas the test that has been deployed can impact several others: it is then possible that the test improves the KPI which is being monitored at the expense of others, without you being able to realise this immediately.

Learn more tips to be up and running with A/B Testing. Download our A/B Testing ebook.

You might also like...

See all

Article

6min read

Minimal Detectable Effect: The Essential Ally for Your A/B Tests

Hubert Wassner

May 6, 2025

Article

2min read

How We Made AB Tasty’s Feature Experimentation SDK OpenFeature-Compatible — And Even Easier to Adopt

Julie Dumont

Apr 30, 2025

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Subscribe to
our Newsletter

Article

Jul 28, 2014

4min read

Before you Start A/B Testing, Define your Roadmap

Anthony Brebion

What tests should I launch? In what order? How do I stop several tests from interfering with each other? What risks are involved and how do I minimise these? Before you begin, these are the questions to ask. Follow our advice to establish your roadmap and get results from your A/B tests.

Before you start testing, you must mark out your route. A roadmap is essential, not only for clearly defining your goals, priorities and risks, but also for setting out a timetable. With this, you can track the progress of your project, and keep all of your contributors informed.

Your roadmap should, in particular, consider the following 8 aspects:

Test names

Be sure to give your tests precise, explicit names (preferably the same as you use in your AB Tasty tool). For example, a short name such as “[HP]wordingCTA” is preferable to an entire phrase such as “Test on the wording of the CTA on the homepage”.

Test descriptions

So that all your contributors are in step with your testing procedure and can follow it as it evolves, write a short, explicit description. For example, “Replace CTA wording ‘Create an account’ with ‘Sign up now’” will allow anybody to understand the content and goals of the test.

Test priority level

It is vital to rank your tests in the order of their importance in order to decide the order in which you will launch them. It is up to you to gauge:

The expected benefits of the test
The technical difficulty of the test

After the results of your first few tests, you will be able to adjust your ranking to optimise the effectiveness of your tests.

Test range

It is also crucial to include in your roadmap the range of pages targeted by your test. This way, you will stop different tests from interfering with each other (when you have several tests taking place at the same time on the same page) or possible side effects. To guard against this, we advise you to cut your site into slices and to give each slice a different colour in your roadmap: for example, blue for the homepage, orange for category pages, green for product pages, yellow for the conversion funnel, etc.

Primary and secondary KPIs

For each test, you should define a primary key performance indicator (KPI) (associated with a macro conversion). This indicator, which is the main reason for creating the test, will allow you to evaluate its benefits. It might be the click-through rate from the button “Add to basket”, the number of signups to your newsletter, revenue generated, etc.

You should also define secondary KPIs (associated with microconversions), which will complement your analysis and allow you to better understand the results of your test. Examples might include time spent on the site, number of pages seen, bounce rate, and so on.

Resources required

Some tests might require:

Technical development
Ergonomic development
A specific launch date

These kinds of requirements must be specified in your roadmap.

Launch date and estimated end date

This information will make your team’s reports easier to read, and make it easier to plan potential future tests. In the meantime it will allow you to plan your testing activity precisely.

Possible side effects, Who to contact, Alerts

You should include a space for “notes” for each test, which will be useful in case of any problems. Here, you can write contacts, useful information, important things to remember, and so forth. This will save you a lot of time and worry.

You might also like...

See all

Article

6min read

Minimal Detectable Effect: The Essential Ally for Your A/B Tests

Hubert Wassner

May 6, 2025

Article

2min read

How We Made AB Tasty’s Feature Experimentation SDK OpenFeature-Compatible — And Even Easier to Adopt

Julie Dumont

Apr 30, 2025

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Subscribe to
our Newsletter

Article

Jul 17, 2014

5min read

Six Ways for Getting Started with A/B Testing with Low Traffic

Anthony Brebion

A/B testing is a key tool when it comes to optimizing your conversion rates. However, an effective A/B test campaign requires certain conditions and, in particular, a substantial level of traffic and conversions. Should you rule out A/B testing if your traffic is too low? The answer is no, you’ll be glad to hear! There are methods which can be implemented to make the most of an A/B test campaign, even when traffic is low. The main obstacle to setting up A/B tests with low traffic lies in statistical significance. If the test results are to be considered reliable, the reliability index needs to be over 95%. Below 95%, the data are not deemed reliable and drawing conclusions can be risky. In simple terms, the lower the traffic, the more time it takes to reach that level. The following techniques will help you reduce that time.

#1: be patient!

First of all, if your traffic does not exceed a few hundred visits per month, you are no doubt better off waiting. Focus on building an audience by applying the conventional traffic acquisition levers (production of interesting and potentially viral content, building your presence on the social networks, etc.). At this stage, take the opportunity to collect qualitative information on your visitors’ behaviour (their impressions, their feedback, etc.). As your traffic is low, this feedback can be analysed manually and can provide valuable avenues to be explored once you have acquired more traffic.

#2: get more traffic temporarily

If your traffic is low, tackle the problem at the source! Temporarily increasing traffic to the pages you want to test is a good way of obtaining reliable statistics more rapidly. The simplest way to do this is via a pay-per-performance advertising model (pay per click): sponsored Google AdWords links, Facebook Ads, sponsored news on LinkedIn, etc. This means investing financially, but it rapidly brings traffic. If you have a large community, set up an emailing campaign to draw traffic to the page you want to test. You should of course make sure you redirect your campaigns to a landing page that is close to the final conversion stage, rather than to your website’s home page.

#3: test the pages with the most traffic

Make the most of the traffic you already have by focusing efforts on the pages where you get the most traffic. By doing this, you will increase your chances of getting significant results faster. When you have more traffic across the whole website, you will then be able to test the other pages.

#4: limit the number of variations

The more variations you compare, the less each one of them will get traffic and the more time it will take to get a sufficient sample for each variation. Don’t create more than two variations in addition to the original version. Depending on your traffic, you may need to limit yourself to a single variation. If this out-performs, start a second test to compare its performance with the variation you had set aside. Also forget about multivariate tests (MVT) which are only designed for websites with very high traffic levels.

#5: change the conversion measurement criteria

The aim of an A/B test is usually to increase the number of conversions. But how do you define conversion? As a sale? In a test where traffic is low, it is preferable to select a criterion related to your main criterion but which occurs more often. Rather than a sale, target the downloading of a test version or the viewing of a demonstration video, for example. These conversion criteria are always directly linked to your main criterion (here, the sale), but are likely to occur more often and will therefore bring you results faster. You can also target interaction with your website, for example by recording a conversion when a visitor spends a certain amount of time on your website or visits a certain number of pages.

#6: test significant changes rather than small modifications

We often think that A/B testing is used to define the best color for a call-to-action or to optimise a title. This is just one facet of A/B testing but this kind of micro-optimization can only be applied where traffic is substantial. Where traffic is low, look at the bigger picture! Test changes likely to lead to a high increase in conversions, not the details seeking a 0.1% increase. Change the position of elements, rewrite entire titles and test two completely different versions of your page. If you opt for this latter option, do not waste time modifying your page in the AB Tasty editor but instead opt for the split testing solution. The principle is simple: you create a completely different design for your page, you host it on your own server and you use it as a variation. Web visitors are redirected to this page in a fully transparent manner. AB Tasty takes care of the traffic breakdown and collects statistics for each version, according to the parameters you have indicated, just like a conventional test.

Conclusion

Just because your website has low traffic, it does not mean you should forget about A/B testing – on the contrary! Set your goals and set you up your first tests.