A/B Test Hypothesis Definition, Tips and Best Practices

If one were to check the dictionary for a hypothesis definition, they would surely find something along these lines:

This would seem simple enough. And yet irrelevant, incomplete, or poorly formulated A/B test hypothesis are at the root of many a neutral or negative test. In the follow article, we’ll quickly go over how to build an effective experiment hypothesis for Conversion Rate Optimization (CRO) tests.

Often, we imagine that doing A/B tests to improve your e-commerce site’s performance means quickly changing the color of your “buy” button, for example. We then think, rightfully (sometimes!) or wrongfully, that changing that icon from red to green will drastically increase our conversion rate.

Far too few people question the true origins of the success (or failure!) of the changes they put in place to improve their conversion rate (or other measurement data), and it is important to know how to determine both the problem and the hypotheses that will allow you to obtain the best results by insisting on the long term because you can often double your results!

So, instead of going in headlong and finding a “DIY” solution, it is often better to take a step back to:

1. Identify the real problem at the source of poor performance (for example, a high bounce rate on your landing page or a high dropout rate on your site’s order confirmation page)
2. Establish a hypothesis that could be at the root of the problem (for example, “our customers do not immediately understand the characteristics of our products when they read the pages on our e-commerce site”)

This second step, which may seem difficult because it requires a capacity for introspection and a critical look at the existing site, is nevertheless crucial for anyone who wants to see their KPIs improve drastically!

Also, the questions we will answer here will be:

• What is an A/B test hypothesis for an e-commerce site?
• A/B test hypotheses: the starting formula
• What are the elements to consider to ensure that the hypotheses made are the right ones and will guarantee you the best results?

What is an A/B test hypothesis?

As mentioned earlier, when we consult a dictionary, the word “hypothesis” has a very simple definition:

The first interesting point to raise in this definition relates to “the set of facts to be explained”: an A/B test hypothesis must always start with a clearly identified problem. A/B tests should not be done randomly, or you risk wasting time.

There are many information sources available to help identify these problems:

• Web analytics data. While this data does not explain internet users’ behavior, they can highlight conversion problems (identifying abandoned carts, for example). They also help prioritize the pages to be tested.
• Heuristic evaluation and ergonomic audit. These analyses allow you to assess the site’s user experience at a lower cost using an analysis grid.
• User tests. This qualitative data is limited by the sample size but can be very rich in information that would not have been detected with quantitative methods. They often reveal problems understanding the offer or site ergonomics. Even if the experience can be painful given the potential for negative remarks, it will allow you to gather qualified data with precise insights.
• Eye tracking or heatmaps. These methods provide visibility into how people interact with items within a page, not just between pages.
• Customer feedback. Companies already collect a lot of feedback from their customers (comments and opinions posted on the site, questions asked of customer service, etc.). As well as analyzing them, you can implement tools such as customer surveys or live chats to collect more information.

These tools will help you highlight the real problems that impact your site’s performance and save time and money in the long run!

Also, before going into more detail on the topic, let’s start with what forms the basis of an A/B test’s hypothesis!

A/B test hypothesis: the starting formula

Initially, making A/B test hypotheses may seem almost simple. It’s mainly about a change and the effect it produces:

At this stage, this formula is only a theoretical assumption that will need to be proven or disproven, but it will guide you in solving the problem.

An important point, however, is that the impact of the change you want to bring must always be measurable in quantifiable terms (conversion rate, bounce rate, abandonment rate, etc.).

Here are some examples of hypotheses phrased according to the formula explained above and that can apply to e-commerce:

Example #1
“Change our call to action from “BUY YOUR TICKETS NOW” to “TICKETS ARE SELLING FAST – ONLY 50 LEFT!” will improve our sales on our e-commerce site.”

Example #2
“Shortening the form by deleting optional fields such as phone and mailing address will increase the number of contacts collected.”

In addition, when you think about the solution you want to implement, include the “psychology” of the prospect by asking yourself the following:

“What psychological impact could the problem cause in the client’s mind?”

For example, if your problem is “there is a lack of clarity in the site’s registration process which impacts conversions to purchases,” then the psychological impact could be your prospect is confused when reading information. With this in mind, you can begin to think concretely about the solution to correct this feeling on the “client” side. In this case, we can imagine “including a progress bar that shows the different stages of registration.”

However, be careful: this “psychological” aspect should not be included when formulating your test hypothesis. You must always respect the following format: “If I change this, it will cause that effect.” Once you have gotten the results, you should then be able to say, “it is true”/”it is false.” Therefore, we must take care to rely on concrete and tangible assumptions.

Finally, since quitting now would be a shame, the end goal of A/B test hypotheses is to identify quickly what will help your business the most.

Optimizing the creation of effective A/B test hypotheses for e-commerce sites

There are many elements that will allow you to build effective A/B test hypotheses. Here are five to get started (and inspire you)! These best practices allow you to build your hypotheses by comparing your current site with the suggestions below and directly impact conversion performance.

ON THE HOMEPAGE

• The header/main banner explaining the products/services that the site offers can increase customers’ curiosity and extend their time on the site.
• A call-to-action visible right when arriving will increase the chance visitors will click
• A very visible “about” section will build prospects’ trust in the brand when they arrive on the site.

IN THE PRODUCT SECTIONS

• Filters save customers a lot of time. They can find what they are looking for quickly.
• Highlighting a selection of the most popular products at the top of the sections is an excellent starting point for generating sales.
• A “find out more” button or link under each product will encourage users to want to…find out more!

IN THE PRODUCT PAGES

• Product recommendations create a more personal experience for the user and help increase their average shopping cart
• If the “add to cart” button is the most visible element of the product listing, it will catch the prospect’s attention and increase the click rate.
• An “Add to Cart and Pay” button saves the customer time with most paying an average of 1 transaction at a time.
• Adding “social” sharing buttons is an effective way of turning the product listing into viral content.

IN THE CART

• The presence of logos such as “Visa certified” enhances customer confidence in the site
• A very visible button/link to “Proceed to payment” greatly encourages users to click

PAYMENT

• A single page for payment reduces the loading time between two pages and maintains the customer’s attention.
• Paying for an order without registration is very much appreciated by new prospects, who are not necessarily inclined to share their personal information when first visiting the site.
• If the user can see the different payment stages, then he or she will have visibility over the rest of the process. This reassures them and will help them finalize their purchase.

To conclude, while defining A/B test hypotheses can seem complex and methodical, think of all the benefits that this stage can have for your e-commerce site. So, the next time you want to optimize your performance, think “analysis and data” before “design and graphics”; it’s a good start to implementing effective hypotheses.