Tag: EN

Article

Jan 9, 2015

4min read

User Testing and A/B Testing… Strengths and Differences

Anthony Brebion

Companies often opt for one or the other of these tests without realizing the benefits of using them together. Given that the benefits of one test may sometimes outweigh the disadvantages of the other, the lessons from user testing and A/B testing complement each other to refine your conversion optimization strategy.

All sites are trying to attract more traffic, but they are also seeking to increase their conversion rate in order to raise profits. The understanding and optimization of their customer paths are therefore essential. In order to understand the problems faced by their websites, it is possible to design a series of tests, such as user testing and A/B testing.

User testing

User testing, or usability testing, makes it possible to understand the needs of users and observe their behavior. This data is qualitative and offers a rich, detailed and very helpful source of information which provides the company with an understanding of what works or does not work in its offer.

Advantages

Provides better knowledge of the user as a person (and not just as a virtual visitor).
Offers comprehensive and often unsuspected sources of data through which the company discovers new areas of improvement.
Always offers a workable response to improve the user experience.

Weaknesses

Significant investment of time and money for purely theoretical results.
Observed results are not quantified and do not allow the company to estimate the revenue potential of the test findings.
Returns are based on the subjective behavior and opinion of a few users.

A/B testing

A/B tests are based on an idea pre-conceived by the company, which seeks to test and quantify the effectiveness or otherwise of a proposed change. They are based on specific, targeted changes whose outcomes will be automatically quantified.

Advantages

Rapid implementation and real-time monitoring (number of clicks, conversion rates, heatmap).
Quantified and representative results (KPIs and ROI simulation).
Allows significant flexibility in changing one or more points on a page.
Very cheap and requires minimal effort in the long term.

Weaknesses

Significant advance preparation (targeting areas to change/watch).
Not all tests always produce results.
Requires a large number of visits to statistically validate the assumptions.

Different but complementary tests

The choice between these two tests mainly depends on the question you want to answer: How much? Why? If your question concerns quantifiable results regarding a specific change on your website, then you must perform an A/B test. If the purpose of your question is to understand your consumers’ behavior, usability testing will better meet your needs.

Although users testing and A/B testing differs in their setup and objectives, their features can easily complement each other. It is recommended that you use both types of tests sequentially. Starting with a usability test allows you to locate the points of friction to address. You can then conduct an A/B test to validate the relevance of the changes you are considering.

The most important prerequisite for achieving valid results from your A/B testing is to build strong test hypotheses that will involve the modification of elements that are actually used in the conversion process or really hamper the process. Find out more about the formulation of test hypotheses.

By combining the two methods, it is possible to make reasonable use of your time and money, while obtaining usable quantitative and qualitative results. Using a tool such as A/B testing is advisable only when an idea for improvement has been identified. If you lack ideas, your users may not – test them!

You might also like...

See all

Article

15min read

16 Experimentation Influencers You Should Follow

Maddie Ostrander

Jul 3, 2025

Article

3min read

Experiment Health Check: Proactive Monitoring for Reliable Experimentation

Emily Healy

Jul 1, 2025

Article

9min read

Heatmaps: Your Team’s Secret Weapon for Uncovering Website Gold

AB Tasty

Jun 24, 2025

Subscribe to
our Newsletter

Article

Sep 2, 2014

4min read

Eight Mistakes not to Make in A/B Testing

Anthony Brebion

The following mistakes can completely distort the results of your A/B testing program. Learn how to avoid them!

1. Changing the control version during the test

Making changes to the control version (i.e. the original version of your page), especially the items you are testing, while the test is in progress, is to be strenuously avoided. The results of the variation will mean nothing if the original version on which the comparison is based has also changed.

2. Testing your changes over different time periods

The whole purpose of A/B testing is to divide your traffic between two versions of your page: a Control version A, and a Variation version B. Collecting data variation on Version A for some time and then doing the same on Version B is not A/B testing and will yield unusable results, because the test conditions have changed! There is no assurance that the traffic you will have fully directed to Version A and then to version B is the same; quite the contrary. Many other factors may come into play: if you test Version A for a normal working period and then test Version B during the holidays, the results you get will have been distorted by changes in timetables and user habits. For these reasons, one of the golden rules of A/B testing is that the two versions must be tested simultaneously: by deploying the variation alongside the control and by distributing the traffic between the two, you will be sure that any factors affecting traffic will affect both versions. You can then confidently learn from your results.

3. Simultaneously editing multiple variables

Actual A/B testing consists in testing a variation of a page that differs in one way only: a test, a variable. Testing several variations of several different elements is no longer A/B testing, but multivariate testing, a significantly more advanced technique. If you set an A/B test with several simultaneous variables, you will have great difficulty, once the test is complete, in determining precisely what variable(s) is(are) responsible for the results you have achieved. If you want to test several elements on a page using A/B tests, first test the most radical changes, then refine the changes with each successive test. Alternatively, you can use multivariate testing.

4. Drawing conclusions before achieving 95% reliability

In the AB Tasty reporting tool, each test is assigned a statistical reliability rate. This is a confidence level whose calculation depends in part on the duration of the test and on the traffic assigned. What you must bear in mind is the figure of 95%: if your test has a reliability rate of 95% or more, you can consider the results reliable. Below this rate, the results you get are just not reliable enough to draw definitive conclusions. If you take a decision on the basis of results without at least 95% reliability, you are relying on potentially false results; it is entirely possible that they will subsequently change.

5. Allowing a test to run for too long

Conversely, if your test has reached 95% statistical reliability and you have enough information to learn lessons, it is best not to let the test go on for too long. Stop your tests when they meet their target! Otherwise, you could waste time waiting for a marginal improvement of already good results and you expose some of your traffic to a variation that is already tested, whereas these visitors could be contributing to new tests. How long should I run my A/B tests?

6. Not being honest when interpreting data

For your A/B testing to be truly effective, you must accept the results, whatever they are (provided of course they have reached 95% statistical reliability). Of course, this is easier said than done when you see the variation that you spent weeks designing fail against the control version! The importance of A/B testing is precisely to tell you unambiguously which page version works best with your users, even though it was not your preferred version.

7. Surprising your most loyal visitors

If you have a high rate of repeat visitors, it is better not to surprise them too suddenly with a radically different variation. Your loyal visitors are regulars whom an abrupt change to your site could drive away, even though the variation in question may be not the one ultimately chosen. In addition, if you test your new visitors, you’ll be sure that their behaviour will not be biased.

8. Choosing your KPIs poorly

Finally, be sure to choose KPIs that are truly representative of your goals. If you choose a KPI that is too far from the goal, you will get a numerical result with no great connection to your goal. Another common mistake is to take only one KPI whereas the test that has been deployed can impact several others: it is then possible that the test improves the KPI which is being monitored at the expense of others, without you being able to realise this immediately.

Learn more tips to be up and running with A/B Testing. Download our A/B Testing ebook.

You might also like...

See all

Article

15min read

16 Experimentation Influencers You Should Follow

Maddie Ostrander

Jul 3, 2025

Article

3min read

Experiment Health Check: Proactive Monitoring for Reliable Experimentation

Emily Healy

Jul 1, 2025

Article

9min read

Heatmaps: Your Team’s Secret Weapon for Uncovering Website Gold

AB Tasty

Jun 24, 2025

Subscribe to
our Newsletter

Article

Jul 28, 2014

4min read

Before you Start A/B Testing, Define your Roadmap

Anthony Brebion

What tests should I launch? In what order? How do I stop several tests from interfering with each other? What risks are involved and how do I minimise these? Before you begin, these are the questions to ask. Follow our advice to establish your roadmap and get results from your A/B tests.

Before you start testing, you must mark out your route. A roadmap is essential, not only for clearly defining your goals, priorities and risks, but also for setting out a timetable. With this, you can track the progress of your project, and keep all of your contributors informed.

Your roadmap should, in particular, consider the following 8 aspects:

Test names

Be sure to give your tests precise, explicit names (preferably the same as you use in your AB Tasty tool). For example, a short name such as “[HP]wordingCTA” is preferable to an entire phrase such as “Test on the wording of the CTA on the homepage”.

Test descriptions

So that all your contributors are in step with your testing procedure and can follow it as it evolves, write a short, explicit description. For example, “Replace CTA wording ‘Create an account’ with ‘Sign up now’” will allow anybody to understand the content and goals of the test.

Test priority level

It is vital to rank your tests in the order of their importance in order to decide the order in which you will launch them. It is up to you to gauge:

The expected benefits of the test
The technical difficulty of the test

After the results of your first few tests, you will be able to adjust your ranking to optimise the effectiveness of your tests.

Test range

It is also crucial to include in your roadmap the range of pages targeted by your test. This way, you will stop different tests from interfering with each other (when you have several tests taking place at the same time on the same page) or possible side effects. To guard against this, we advise you to cut your site into slices and to give each slice a different colour in your roadmap: for example, blue for the homepage, orange for category pages, green for product pages, yellow for the conversion funnel, etc.

Primary and secondary KPIs

For each test, you should define a primary key performance indicator (KPI) (associated with a macro conversion). This indicator, which is the main reason for creating the test, will allow you to evaluate its benefits. It might be the click-through rate from the button “Add to basket”, the number of signups to your newsletter, revenue generated, etc.

You should also define secondary KPIs (associated with microconversions), which will complement your analysis and allow you to better understand the results of your test. Examples might include time spent on the site, number of pages seen, bounce rate, and so on.

Resources required

Some tests might require:

Technical development
Ergonomic development
A specific launch date

These kinds of requirements must be specified in your roadmap.

Launch date and estimated end date

This information will make your team’s reports easier to read, and make it easier to plan potential future tests. In the meantime it will allow you to plan your testing activity precisely.

Possible side effects, Who to contact, Alerts

You should include a space for “notes” for each test, which will be useful in case of any problems. Here, you can write contacts, useful information, important things to remember, and so forth. This will save you a lot of time and worry.

You might also like...

See all

Article

15min read

16 Experimentation Influencers You Should Follow

Maddie Ostrander

Jul 3, 2025

Article

3min read

Experiment Health Check: Proactive Monitoring for Reliable Experimentation

Emily Healy

Jul 1, 2025

Article

9min read

Heatmaps: Your Team’s Secret Weapon for Uncovering Website Gold

AB Tasty

Jun 24, 2025

Subscribe to
our Newsletter

Article

Jul 17, 2014

5min read

Six Ways for Getting Started with A/B Testing with Low Traffic

Anthony Brebion

A/B testing is a key tool when it comes to optimizing your conversion rates. However, an effective A/B test campaign requires certain conditions and, in particular, a substantial level of traffic and conversions. Should you rule out A/B testing if your traffic is too low? The answer is no, you’ll be glad to hear! There are methods which can be implemented to make the most of an A/B test campaign, even when traffic is low. The main obstacle to setting up A/B tests with low traffic lies in statistical significance. If the test results are to be considered reliable, the reliability index needs to be over 95%. Below 95%, the data are not deemed reliable and drawing conclusions can be risky. In simple terms, the lower the traffic, the more time it takes to reach that level. The following techniques will help you reduce that time.

#1: be patient!

First of all, if your traffic does not exceed a few hundred visits per month, you are no doubt better off waiting. Focus on building an audience by applying the conventional traffic acquisition levers (production of interesting and potentially viral content, building your presence on the social networks, etc.). At this stage, take the opportunity to collect qualitative information on your visitors’ behaviour (their impressions, their feedback, etc.). As your traffic is low, this feedback can be analysed manually and can provide valuable avenues to be explored once you have acquired more traffic.

#2: get more traffic temporarily

If your traffic is low, tackle the problem at the source! Temporarily increasing traffic to the pages you want to test is a good way of obtaining reliable statistics more rapidly. The simplest way to do this is via a pay-per-performance advertising model (pay per click): sponsored Google AdWords links, Facebook Ads, sponsored news on LinkedIn, etc. This means investing financially, but it rapidly brings traffic. If you have a large community, set up an emailing campaign to draw traffic to the page you want to test. You should of course make sure you redirect your campaigns to a landing page that is close to the final conversion stage, rather than to your website’s home page.

#3: test the pages with the most traffic

Make the most of the traffic you already have by focusing efforts on the pages where you get the most traffic. By doing this, you will increase your chances of getting significant results faster. When you have more traffic across the whole website, you will then be able to test the other pages.

#4: limit the number of variations

The more variations you compare, the less each one of them will get traffic and the more time it will take to get a sufficient sample for each variation. Don’t create more than two variations in addition to the original version. Depending on your traffic, you may need to limit yourself to a single variation. If this out-performs, start a second test to compare its performance with the variation you had set aside. Also forget about multivariate tests (MVT) which are only designed for websites with very high traffic levels.

#5: change the conversion measurement criteria

The aim of an A/B test is usually to increase the number of conversions. But how do you define conversion? As a sale? In a test where traffic is low, it is preferable to select a criterion related to your main criterion but which occurs more often. Rather than a sale, target the downloading of a test version or the viewing of a demonstration video, for example. These conversion criteria are always directly linked to your main criterion (here, the sale), but are likely to occur more often and will therefore bring you results faster. You can also target interaction with your website, for example by recording a conversion when a visitor spends a certain amount of time on your website or visits a certain number of pages.

#6: test significant changes rather than small modifications

We often think that A/B testing is used to define the best color for a call-to-action or to optimise a title. This is just one facet of A/B testing but this kind of micro-optimization can only be applied where traffic is substantial. Where traffic is low, look at the bigger picture! Test changes likely to lead to a high increase in conversions, not the details seeking a 0.1% increase. Change the position of elements, rewrite entire titles and test two completely different versions of your page. If you opt for this latter option, do not waste time modifying your page in the AB Tasty editor but instead opt for the split testing solution. The principle is simple: you create a completely different design for your page, you host it on your own server and you use it as a variation. Web visitors are redirected to this page in a fully transparent manner. AB Tasty takes care of the traffic breakdown and collects statistics for each version, according to the parameters you have indicated, just like a conventional test.

Conclusion

Just because your website has low traffic, it does not mean you should forget about A/B testing – on the contrary! Set your goals and set you up your first tests.