Multivariate testing (MVT)
A multivariate test is a test that simultaneously tests several combinations of several variables. The idea is to modify several elements simultaneously on the same page and then define which one, among all of the possible combinations, has the most impact on the indicators being tracked. Multivariate testing (MVT) helps test associations of variables, which is not the case of successive A/B (or A/B/C, etc.) tests.
What is a multivariate test?
During an A/B test, you may not modify more than one element at a time (for example, the wording of a button) in order to be able to measure the impact. If you modify both the button’s wording and colour (for example, a blue “Buy” button vs. a red “Make the most of it” button) and notice an improvement, how will you know if the wording or the colour really contributed to this performance? The impact of one change could be negligible or they each could have had an equal impact.
Multivariate testing looks to provide the solution. You can change a title and an image at the same time. With multivariate tests, you test a hypothesis for which several variables are modified and determine which combination from among all possible solutions performed the best. If you create 3 different versions of 2 specific variables, you then have nine combinations in total (number of variants of the first variable X number of variants of the second).
There are three benefits to MVT:
- Avoid performing successive A/B tests and save time since multivariate testing can be seen as performing several A/B tests on the same page at the same time.
- Determine the impact of each variable in measured gains.
- Measure the impact of interactions between different elements presumed to be independent (for example, page title and illustration visual).
What are the types of multivariate tests?
There are 2 main methods for performing multivariate tests:
- “Full Factorial”: This is the method generally referred to when we talk about multivariate testing. With this method, all of the possible combinations of variables are designed and tested over equal parts of traffic. If you test 2 variants of one element and 3 of another, each of the 6 combinations will therefore receive 16.66% of your traffic.
- “Fractional Factorial”: as its name suggests, only a fraction of possible combinations is effectively tested on your traffic. The conversion rate of untested combinations is statistically deducted based on those actually tested. This method has the disadvantage of being less precise, but requires less traffic.
Limits of Multivariate Tests
The first limit concerns the number of visitors needed for your multivariate test’s results to be significant. By multiplying the number of variables and versions tested in your multivariate test, you will quickly reach a large number of combinations. The sample assigned to each combination will be reduced proportionally.
Where, for a traditional A/B test, you would assign 50% of your traffic to the original version in the tool and the rest to the variant, you will only assign 5, 10, or 15% of your traffic to each combination in a multivariate test. In practice, this often translates into longer tests and an inability to reach the statistical significance needed to make a decision. This is especially true if you test pages deep within your site with low traffic, which is often the case in order tunnels or landing pages for your traffic acquisition campaigns.
The second limit is linked to the way the multivariate test is defined. In some cases, it’s the result of an admission of weakness: the users don’t know exactly what to test and think that by testing several things at once in a multivariate test, they will eventually find a solution they can take advantage of. We then often find small changes at work in these multivariate tests. A/B testing, on the other hand, requires great rigour and helps better identify test hypotheses, which generally lead to more creative tests, backed up by data, with better results.
The third limit is related to complexity. Conducting an A/B test is often easier than a multivariate test, especially when analysing the results. You don’t have to do complex mental gymnastics to try to understand why a particular element interacts positively with another in one case but not in another. Keeping the process simple and quick to perform helps maintain confidence and rapidly reiterate on optimisation ideas.