How to Use Qualitative Data to Plan your Next A/B Test

AB TastyBlogAnalyticsHow to Use Qualitative Data to Plan your Next A/B Test

If you’re already familiar with running A/B tests, you know that you should base it on information from web analyses, such as traffic and conversion rates, to work out which aspects of your website need optimizing.  However, this quantitative statistical data is just one part of the equation.  In order to get a true understanding of the behaviour of your users and choose the most relevant tests to set up, you must use qualitative data.

What is qualitative data?

Qualitative data enables you to understand the expectations and the experience of your users. Unlike quantitative data, it’s not about measuring and counting your users’ behaviour, it’s about observing them. If plain quantitative data shows you how they behave on your website, qualitative data shows you why they behave in a particular way. Optimising the user experience, in particular, requires a qualitative approach. Please note that you shouldn’t believe that qualitative data can replace quantitative data. Both types of information are complementary and necessary for a proper understanding of user behaviours.

Collecting qualitative data

With web analyses, we start with the problem, identify the causes and then find a solution. However, when we rely on quantitative data (e.g. traffic, bounce rate, conversion rate, etc), the causes are difficult to identify other than by guesswork. The tools presented in this article will enable you to collect qualitative data directly from your users to identify these causes.

Your first source of qualitative data on your users is the most accessible: your client services team! Start collecting your information directly from them, because they are in constant contact with your users and can give you the first essential insights.

1. Contact your users directly by email

You can collect qualitative data from your users without the need for any special tools. The advantage of email is that it not only makes it possible to easily segment new and old users, but also to personalise the message and so increase the chances that they will respond. It’s best to contact your users by email for a particular reason rather than out of the blue. So, for example, you can email them after a visit, a purchase (or a failure to purchase) or a registration.

Tools, such as Google Forms, can help you set up your email campaign by including your survey in the actual body of the email, thereby reducing the number of clicks required of the user. Once you’ve launched your campaign, categorise and analyse each response. If you’re using Google Forms, you’ll receive all responses directly in a Google Doc.

You can also use an online questionnaire tool, such as SurveyMonkey, to simply ask your users about their experience on your website and what problems they may have encountered. For example, you could ask them what annoys them, what they get stuck on, or if they feel that they lack information or features.

2. Learn about micro-surveys with Qualaroo

Qualaroo allows you to create very short surveys (a single question), aimed at your users, and to set them up directly on the pages of your choice. This gives you feedback from your users directly from your website and at the very time that they are involved in the action you want to optimize. By asking a single question by questionnaire and selecting the possible responses, you can be sure of getting qualitative feedback from which you can immediately draw concrete conclusions.

The advantage of Quaraloo, compared to a classic questionnaire, is that Qualaroo contains a single question (and its possible responses) asked directly on the page concerned via a pop-up at the bottom of the screen. From the user’s point of view, this type of question takes barely a second to answer with a single click and all without leaving the page.

3. Visualise interaction with heatmaps

Heatmaps show where your users click and scroll on a page, in a highly visual manner. Heat maps are an essential tool which gives you information that it would be impossible to see in any other way. Different types of heatmaps are available.


  • Heat maps show where users click on the page using ‘hot areas’ and ‘cold areas’.  This shows you which parts of the page users are interacting with most, which parts attract their attention most and, conversely, which parts they overlook. If your calls to action or the parts which you consider to be essential are not attracting users’ attention, you need to think about changing and testing their appearance or position. With a heat map, you can also see if users are trying to click on non-clickable elements, a classic source of frustration and to be avoided.


  • Scrollmaps show you just how far down a page your users will scroll with their mouse, which shows you exactly where their attention drops off or disappears all together. Use scrollmaps to find out whether certain elements are too low down on the page to be seen and which elements you need to make visible above the fold.


  • Overlay focuses on links and gives you the number of clicks on each of them. Overlay allows you to filter the results obtained from heat maps by comparing the ‘performances’ of different links on a page. For example, you can see whether some links are too small and are overlooked and how many clicks are made on links which are repeated in several places on a page, according to their position.


  • You can go even deeper in your analysis with Confetti, a tool which enables you to segment traffic and the visualisations according a great many criteria, e.g. origin of traffic, new visitors, etc.


These three tools are just some of the options for the collection of qualitative data. You can also play with user session replay tools or user testing services, such as UserTesting, or contact individual users to get their personal opinions!

When you collect such qualitative data, you will be better able to understand the reasons behind the problems that your users encounter and so facilitate the decision on which solutions to test in response to these problems.

Stop using intuition as the basis for your site modifications! Click here to start using a data-driven method for website optimization.

Share on Linkedin
Share on Facebook
Share on Twitter

You might also like...

Enable your product team to put users in the center of product strategy, while enabling engineering to deploy code faster.
Trusted by
Request a demo
Don't throw away hard-earned traffic.
Make your website work overtime - so you don't have to.
click-through rate
conversion rate
average order value
Trusted by
Request a demo