The following mistakes can completely distort the results of your A/B testing program. Learn how to avoid them!
1. Changing the control version during the test
Making changes to the control version (i.e. the original version of your page), especially the items you are testing, while the test is in progress, is to be strenuously avoided. The results of the variation will mean nothing if the original version on which the comparison is based has also changed.
2. Testing your changes over different time periods
The whole purpose of A/B testing is to divide your traffic between two versions of your page: a Control version A, and a Variation version B. Collecting data variation on Version A for some time and then doing the same on Version B is not A/B testing and will yield unusable results, because the test conditions have changed! There is no assurance that the traffic you will have fully directed to Version A and then to version B is the same; quite the contrary. Many other factors may come into play: if you test Version A for a normal working period and then test Version B during the holidays, the results you get will have been distorted by changes in timetables and user habits. For these reasons, one of the golden rules of A/B testing is that the two versions must be tested simultaneously: by deploying the variation alongside the control and by distributing the traffic between the two, you will be sure that any factors affecting traffic will affect both versions. You can then confidently learn from your results.
3. Simultaneously editing multiple variables
Actual A/B testing consists in testing a variation of a page that differs in one way only: a test, a variable. Testing several variations of several different elements is no longer A/B testing, but multivariate testing, a significantly more advanced technique. If you set an A/B test with several simultaneous variables, you will have great difficulty, once the test is complete, in determining precisely what variable(s) is(are) responsible for the results you have achieved. If you want to test several elements on a page using A/B tests, first test the most radical changes, then refine the changes with each successive test. Alternatively, you can use multivariate testing.
4. Drawing conclusions before achieving 95% reliability
In the AB Tasty reporting tool, each test is assigned a statistical reliability rate. This is a confidence level whose calculation depends in part on the duration of the test and on the traffic assigned. What you must bear in mind is the figure of 95%: if your test has a reliability rate of 95% or more, you can consider the results reliable. Below this rate, the results you get are just not reliable enough to draw definitive conclusions. If you take a decision on the basis of results without at least 95% reliability, you are relying on potentially false results; it is entirely possible that they will subsequently change.
5. Allowing a test to run for too long
Conversely, if your test has reached 95% statistical reliability and you have enough information to learn lessons, it is best not to let the test go on for too long. Stop your tests when they meet their target! Otherwise, you could waste time waiting for a marginal improvement of already good results and you expose some of your traffic to a variation that is already tested, whereas these visitors could be contributing to new tests. How long should I run my A/B tests?
6. Not being honest when interpreting data
For your A/B testing to be truly effective, you must accept the results, whatever they are (provided of course they have reached 95% statistical reliability). Of course, this is easier said than done when you see the variation that you spent weeks designing fail against the control version! The importance of A/B testing is precisely to tell you unambiguously which page version works best with your users, even though it was not your preferred version.
7. Surprising your most loyal visitors
If you have a high rate of repeat visitors, it is better not to surprise them too suddenly with a radically different variation. Your loyal visitors are regulars whom an abrupt change to your site could drive away, even though the variation in question may be not the one ultimately chosen. In addition, if you test your new visitors, you’ll be sure that their behaviour will not be biased.
8. Choosing your KPIs poorly
Finally, be sure to choose KPIs that are truly representative of your goals. If you choose a KPI that is too far from the goal, you will get a numerical result with no great connection to your goal. Another common mistake is to take only one KPI whereas the test that has been deployed can impact several others: it is then possible that the test improves the KPI which is being monitored at the expense of others, without you being able to realise this immediately.
Learn more tips to be up and running with A/B Testing. Download our A/B Testing ebook.
