Category: A/B Testing

Article

Jun 7, 2023

8min read

10 Generative AI Ideas for Your Experimentation Roadmap

Emily Healy

Artificial intelligence has been a recurring theme for decades. However, it’s no longer science fiction – it’s a reality.

Since OpenAI launched its own form of generative AI, ChatGPT, in November 2022, the world has yet to stop talking about its striking capabilities. It’s particularly fascinating to see just how easy it is to get results after interacting with this bot which is comprised of deep-learning algorithms for natural language processing.

Even Google quickly followed by launching a new and experimental project, Gemini, to revolutionize its own Search. By harnessing the power of generative AI and the capacity of large language models, Google is seeking to take its search process to the next level.

Given the rapid growth of this technological advancement over the past few months, it’s time that we talk about generative AI in the context of A/B testing and experimentation.

Whether you’re curious about how AI can impact your experiments or are ready for inspiration we’ll discuss some of our ideas around using AI for A/B testing, personalization, and conversion rate optimization.

What is generative AI?

Generative AI is a type of artificial intelligence that doesn’t have programming limitations, which allows it to generate new content (think ChatGPT). Instead of following a specific, pre-existing dataset, generative AI learns from indexing extensive data, focusing on patterns and using deep learning techniques and neural networks to create human-like content based on its learnings.

The way algorithms capture ideas is similar to how humans gather inspiration from previous experiences to create something unique. Based on the large amounts of data used to craft generative AI’s learning abilities, it’s capable of outputting high-quality responses that are similar to what a human would create.

However, some concerns need to be addressed:

Biased information: Artificial intelligence is only as good as the datasets used to train it. Therefore if the data used to train it has biases, it may create “ideas” that are equally biased or flawed.
Spreading misinformation: There are many concerns about the ethics of generative AI and sharing information directly from it. It’s best practice to fact-check any content written by AI to avoid putting out false or misleading information.
Content ownership: Since content generated with AI is not generated by a human, can you ethically can claim it as your own idea? In a similar sense, the same idea could potentially be generated elsewhere by using a similar prompt. Copywriting and ownership are then called into question here.
Data and privacy: Data privacy is always a top-of-mind concern. With the new capabilities of artificial intelligence, data handling becomes even more challenging. It’s always best practice to avoid using sensitive information with any form of generative AI.

By keeping these limitations in mind, generative AI has the potential to streamline processes and revolutionize the way we work – just as technology has always done in the past.

10 generative AI uses for A/B testing

In the A/B testing world, we are very interested in how one can harness these technological breakthroughs for experimentation. We are brainstorming a few approaches to re-imagine the process of revolutionizing digital customer experiences to ultimately save time and resources.

Just like everyone else, we started to wonder how generative AI could impact the world of experimentation and our customers. Here are some ideas, some of them concrete and some more abstract, as to how artificial intelligence could help our industry:

DISCLAIMER: Before uploading information into any AI platform, ensure that you understand their privacy and security practices. While AI models strive to maintain a privacy standard, there’s always the risk of data breaches. Always protect your confidential information.

1. Homepage optimization

Your homepage is likely the first thing your visitors will see so optimization is key to staying ahead of your competitors. If you want a quick comparison of content on your homepage versus your competitors, you can feed this information into generative AI to give it a basis for understanding. Once your AI is loaded with information about your competitors, you can ask for a list of best practices to employ to make new tests for your own website.

2. Analyze experimentation results

Reporting and analyzing are crucial to progressing on your experimentation roadmap, but it’s also time-consuming. By collecting a summary of testing logs, generative AI can help highlight important findings, summarize your results, and potentially even suggest future steps. Ideally, you can feed your A/B test hypothesis as well as the results to show your thought process and organization. After it recognizes this specific thought process and desired results, it could aid in generating new test hypotheses or suggestions.

3. Recommend optimization barriers

Generative AI can help you prioritize your efforts and identify the most impactful barriers to your conversion rate. Uploading your nonsensitive website performance data gathered from your analytics platforms can give AI the insight it needs into your performance. Whether it suggests that you update your title tags or compress images on your homepage, AI can quickly spot where you have the biggest drop-offs to suggest areas for optimization.

4. Client reviews

User feedback is your own treasure trove of information for optimization. One of the great benefits of AI that we already see is that it can understand large amounts of data quickly and summarize it. By uploading client reviews, surveys and other consumer feedback into the database, generative AI can assist you in creating detailed summaries of your users’ pain points, preferences and levels of satisfaction. The more detailed your reviews – the better the analysis will be.

5. Chatbots

Chatbots are a popular way to communicate with website visitors. As generative AI is a large language model, it can quickly generate conversational scripts, prompts and responses to reduce your brainstorming time. You can also use AI to filter and analyze conversations that your chatbot is already having to determine if there are gaps in the conversation or ways to enhance its interaction with customers.

6. Translation

Language barriers can limit a brand that has a presence in multiple regions. Whether you need translations for your chatbot conversations, CTAs or longer form copy, generative AI can provide you with translations in real time to save you time and make your content accessible to all zones touched by your brand.

7. Google Adwords

Speed up brainstorming sessions by using generative AI to experiment with different copy variations. Based on the prompts you provide, AI can provide you with a series of ideas for targeting keywords and creating copy with a particular tone of voice to use with Google Adwords. Caution: be sure to double-check all keywords proposed to verify their intent.

8. Personalization

Personalized content can be scaled at speed by leveraging artificial intelligence to produce variations of the same messages. By customizing your copy, recommendations, product suggestions and other messages based on past user interactions and consumer demographics, you can significantly boost your digital consumer engagement.

9. Product Descriptions

Finding the best wording to describe why your product is worth purchasing may be a challenge. With generative AI, you can get more ambitious with your product descriptions by testing out different variations of copy to see which version is the most promising for your visitors.

10. Predict User Behavior

Based on historical data from your user behavior, generative AI can predict behavior that can help you to anticipate your next A/B test. Tailoring your tests according to patterns and trends in user interaction can help you conduct better experiments. It’s important to note that predictions will be limited to patterns interpreted by past customer data collected and uploaded. Using generative AI is better when it’s used as a tool to guide you in your decision-making process rather than to be the deciding force alone.

The extensive use of artificial intelligence is a new and fast-evolving subject in the tech world. If you want to leverage it in the future, you need to start familiarizing yourself with its capabilities.

Keep in mind that it’s important to verify the facts and information AI generates just as you carefully verify data before you upload. Using generative AI in conjunction with your internal experts and team resources can assist in improving ideation and efficiency. However, the quality of the output from generative AI is only as good as what you put in.

Is generative AI a source of competitive advantage in A/B testing?

The great news is that this technology is accessible to everyone – from big industry leaders like Google to start-ups with a limited budget. However, the not-so-great news is that this is available to everyone. In other words, generative AI is not necessarily a source of competitive advantage.

Technology existing by itself does not create more value for a business. Rather, it’s the people driving the technology who are creating value by leveraging it in combination with their own industry-specific knowledge, past experiences, data collection and interpretation capabilities and understanding of customer needs and pain points.

While we aren’t here to say that generative AI is a replacement for human-generated ideas, this technology can definitely be used to complement and amplify your already-existing skills.

Leveraging generative AI in A/B testing

From education to copywriting or coding – all industries are starting to see the impact that these new software developments will have. Leveraging “large language models” is becoming increasingly popular as these algorithms can generate ideas, summarize long forms of text, provide insights and even translate in real-time.

Proper experimentation and A/B testing are at the core of engaging your audience, however, these practices can take a lot of time and resources to accomplish successfully. If generative AI can offer you ways to save time and streamline your processes, it might be time to use it as your not-so-secret weapon. In today’s competitive digital environment, continually enhancing your online presence should be at the top of your mind.

Want to start optimizing your website? AB Tasty is the best-in-class experience optimization platform that empowers you to create a richer digital experience – fast. From experimentation to personalization, this solution can help you activate and engage your audience to boost your conversions.

You might also like...

See all

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Article

3min read

Tag V4: Elevating your Experience Building with an Improved Modification Engine

Julie Dumont

Feb 25, 2025

Article

2min read

New Visual Studio Code Extension: Dev-Friendly Experimentation & Personalization

Julie Dumont

Feb 21, 2025

Subscribe to
our Newsletter

Article

Jun 5, 2023

10min read

How Long Should You Run an A/B Test?

Anthony Brebion

One of the most popular questions when starting with experimentation is: How long should an A/B test run before you can draw conclusions from it?

Determining the ideal A/B test duration can be a challenge for most businesses. You have to factor in your business cycles, traffic flow, the sample size needed and be aware of other business campaigns.

Even if you reach your sample size in a few days… is it okay to end your test then? How long should you really wait?

In this article, we will discuss potential mishaps if your testing cycle is too short, give insights into which factors you need to consider and share advice on finding the best duration for your A/B tests.

Looking for fast statistical reliability? At AB Tasty, we provide a free A/B test duration calculator, which also has capabilities for a sample size calculator.

What happens if you end an A/B test too soon?

The underlying question is a crucial one and can be summed up as follows: At what point can you end a test that appears to be yielding results?

The answer depends on the relevance of the analysis and on the actual benefits of the test.

In fact, it’s not all that unusual to see tests yield good results during the trial phase and no longer see those results once the modifications are introduced.

In most cases, a disappointing observation of this nature comes down to an error during the trial phase: the test was ended too soon and the results at that point were misleading.

Let’s look at an example that illustrates the nature of the problem.

How long to run an A/B test

The graph above shows the change in the conversion rate of two versions of a page that were the subject of a test. The first version appears to break away and perform extremely well. The discrepancy between the two versions is gradually eroded as time goes by – two weeks after the starting point there’s hardly any observable difference at all.

This phenomenon where the results converge is a typical situation: the modification made does not have a real impact on conversion.

There is a simple explanation for the apparent outperformance at the start of the test: it’s unusual for the samples to be representative of your audience when the test starts. You need time for your samples to incorporate all internet user profiles, and therefore, all of their behaviors.

If you end the test too soon and allow your premature data to be the deciding factor, your results will quickly show discrepancies.

How to determine the duration of your A/B test

Now that the problem has been aired let’s have a look at how you can avoid falling into this trap.

The average recommended A/B testing time is 2 weeks, but you should always identify key factors relevant to your own conversion goals to determine the best length for a test that will meet your goals.

Let’s discuss several criteria you should use as a foundation to determine when you can trust the results you see in your A/B testing:

The statistical confidence level
The size of the sample
The representativeness of your sample
The test period and the device being tested

1. The statistical confidence level

All A/B testing solutions show a statistical reliability indicator that measures the probability of the difference in the results observed between each sample not being a matter of chance.

This indicator, which is calculated using the Chi-squared test, is the first indicator that should be used as a basis. It is used by statisticians to assert that a test is deemed reliable when the rate is 95% or higher. So, it is acceptable to make a mistake in 5% of cases and for the results of the two versions to be identical.

Yet, it would be a mistake to use this indicator alone as a basis for assessing the appropriate time to end a test.

For the purposes of devising the conditions necessary to assess the reliability of a test, this is not sufficient. In other words, if you have not reached this threshold then you cannot make the decision. Additionally, once this threshold has been reached, you still need to take certain precautions.

It’s also important to understand what the Chi-squared test actually is: a way of rejecting or not rejecting what is referred to as the null hypothesis.

This, when applied to A/B testing, is when you say that two versions produce identical results (therefore, there’s no difference between them).

If the conclusion of the test leads you to reject the null hypothesis then it means that there is a difference between the results.

However, the test is in no way an indication of the extent of this difference.

2. The size of the sample

There are lots of online tools that you can use to calculate the value of Chi-squared by giving, as the input parameters, the four elements necessary for its calculation (within the confines of a test with two versions).

AB Tasty can provide you with our own sample size calculator for you to find the value of Chi-squared.

By using this tool, we have taken an extreme example in order to illustrate this exact problem.

Sample size required for A/B testing

In this diagram, the Chi-squared calculation suggests that sample 2 converts better than sample 1 with a 95% confidence level. Having said that, the input values are very low indeed and there is no guarantee that if 1,000 people were tested, rather than 100, you would still have the same 1 to 3 ratio between the conversion rates.

It’s like flipping a coin. If there is a 50% probability that the coin will land heads-up or tails-up, then it’s possible to get a 70 / 30 distribution by flipping it just 10 times. It’s only when you flip the coin a very large number of times that you get close to the expected ratio of 50 / 50.

So, in order to have faith in the Chi-squared test, you are advised to use a significant sample size.

You can calculate the size of this sample before beginning the test to get an indication of the point at which it would be appropriate to look at the statistical reliability indicator. There are several tools online that you could use to calculate this sample size.

In practice, this can turn out to be difficult, as one of the parameters to be given is the % improvement expected (which is not easy to evaluate). But, it can be a good exercise to assess the pertinence of the modifications being envisaged.

Pro Tip: The lower the expected improvement rate, the greater the sample size needed to be able to detect a real difference.

If your modifications have a very low impact, then a lot of visitors will need to be tested. This serves as an argument in favor of introducing radical or disruptive modifications that would probably have a greater impact on the conversion.

img_548ab40f4fb20

3. The representativeness of your sample

If you have a lot of traffic, then getting a sufficiently large sample size is not a problem and you will be able to get a statistical reliability rate in just a few days, sometimes just two or three.

Having said that, ending a test as soon as the sample size and statistical reliability conditions have been met is no guarantee that results in a real-life situation are being reproduced.

The key point is to test for as long as you need to in order for all of your audience segments to be included.

Actually, the statistical tests operate on the premise that your samples are distributed in an identical fashion. In other words, the conversion probability is the same for all internet users.

But this is not the case: the probability varies in accordance with different factors such as the weather, the geographical location and also user preferences.

There are two very important factors that must be taken into account here: your business cycles and traffic sources.

Your business cycles

Internet users do not make a purchase as soon as they come across your site. They learn more, they compare, and their thoughts take shape. One, two or even three weeks might elapse between the time they are the subject of one of your tests and the point at which they convert.

If your purchasing cycle is three weeks long and you have only run the test for one week, then your sample will not be representative. As the tool records visits from all internet users, they may not record the conversions of those that are impacted by your test.

Therefore, you’re advised to test over at least one business cycle and ideally two.

Your traffic sources

Your sample must incorporate all of your traffic sources including emails, sponsored links and social networks. You need to make sure that no single source is over-represented in your sample.

Let’s take a concrete situation: if the email channel is a weak source of traffic but significant in terms of revenue and you carry out a test during an email campaign, then you are going to include internet users who have a stronger tendency to make a purchase in your sample.

This would no longer be a representative sample. It’s also crucial to know about major acquisition projects and, if possible, not to test during these periods.

The same goes for tests during sales or other significant promotional periods that attract atypical internet users. You will often see less marked differences in the results if you re-do the tests outside these periods.

It turns out that it’s quite difficult to make sure that your sample is representative, as you have little control over the kind of internet users who take part in your test.

Thankfully, there are two ways of overcoming this problem.

The first is to extend the duration of your test more than is necessary in order to get closer to the normal spread of your internet users.
The second is to target your tests so that you only include a specific population group in your sample. For example, you could exclude all internet users who have come to you as a result of your email campaigns from your samples, if you know that this will distort your results. You could also target only new visitors so that you do not include visitors who have reached an advanced stage in their purchasing process (AKA visitors who are likely to convert regardless of which variation they see).

4. Other elements to keep in mind

There are other elements to bear in mind in order to be confident that your trial conditions are as close as they can be to a real-life situation: timing and the device.

Conversion rates can vary massively on different days of the week and even at different times of the day. Therefore, you’re advised to run the test over complete periods.

In other words, if you launch the test on a Monday morning then it should be stopped on a Sunday evening so that a normal range of conversions is respected.

In the same way, conversion rates can vary enormously between mobiles, tablets and desktop computers. So with devices, you’re advised to test your sites or pages specifically for each device. This is easy to accomplish by using the targeting features to include or exclude the devices if your users show very different browsing and purchasing behavior patterns.

These elements should be taken into account so that you do not end your tests too soon and get led astray by a faulty analysis of the results.

They also explain why certain A/A tests carried out over a period of time that is too short, or during a period of unusual activity, can present differences in results and also differences in statistical reliability, even when you may not have made any modifications at all.

The ideal A/B test duration

Running and A/B testing requires a thorough consideration of various factors such as your personal conversion goals, statistical significance, sample size, seasonality, campaigns, traffic sources, etc. All factors deserve equal attention when determining the best practices for your business.

Just remember to be patient, even if you reach your sample size early. You may be surprised by the ending results.

As A/B testing is an iterative process, continuous experimentation and conversion rate optimization will lead to better results over time.

You might also like...

See all

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Article

3min read

Tag V4: Elevating your Experience Building with an Improved Modification Engine

Julie Dumont

Feb 25, 2025

Article

2min read

New Visual Studio Code Extension: Dev-Friendly Experimentation & Personalization

Julie Dumont

Feb 21, 2025

Subscribe to
our Newsletter

Article

May 25, 2023

10min read

When to Make the Leap from Client- to Server-Side Testing

Rowan Haddad

As companies start out their experimentation journey, they may find themselves experimenting with small changes on their website such as the design of a CTA and other small changes to explore how they perform and their impact on key KPIs such as conversion rate and transaction.

However, as an experimentation program matures, companies are more likely to want to delve deeper into more sophisticated types of testing which require more expertise and a more advanced tool.

This is the point when many companies are ready to take the plunge from client- to server-side experimentation.

In this article, we will explore when is the right time to make the shift to server-side testing and discuss the importance of running these types of tests by outlining the various scenarios where server-side experiments are more appropriate.

Client- vs server-side testing

Before we go deeper into server-side testing, we will quickly point out the differences between client- and server-side tests to understand why you need both types of testing as your experimentation and optimization program evolves.

In client-side testing is where experimentation occurs on the client side through Javascript that runs in the browser. Therefore, client-side tools enable you to create variations of your pages by changing the content sent by your server to users in the web browser. The user then gets one variation of your altered content based on your targeting rules.

Put simply, all the work happens at the level of the browser thanks to Javascript. Because of this, client-side testing is usually best for surface-level changes such as layout, design and colors to measure their performance and impact on key KPIs.

Meanwhile, all the work in server-side testing happens at the server level rather than the browser. In other words, it’s your server that randomly sends a user the modified variation. As a result, the experimentation tool works on the server instead of inside your users’ browsers.

Perhaps one of the biggest benefits of client-side tests is that it’s easy to implement and no special expertise is required to run these tests on the front-end of the website.

However, because of its advanced capabilities, server-side tests require technical expertise and coding skills and so developers are usually the ones running these tests on the back-end.

What are the benefits of server-side testing?

Before we go into when you should be using server-side tests, we will outline some of their benefits in order to better understand their use cases.

Omnichannel experimentation

Client-side solutions are usually limited to devices with web browsers, whether that’s on desktop, mobile or tablet. This means you cannot run experiments on mobile apps or connected devices.

Server-side solutions, for their part, allow you to experiment across multiple channels including mobile apps which significantly widens your playing field and opens up numerous opportunities to A/B test beyond web browsers so you can cover all your bases.

Enhanced performance

Perhaps one of the biggest advantages of server-side tests is the absence of the flicker effect, which is often a major downside associated with client-side solutions.

The flicker effect occurs when the original page loads and is replaced by the test variation and is usually seen by the user.

Unlike client-side tools that require adding a Javascript tag to your pages, you don’t need to add this tag in server-side tools. That’s because experiments are rendered on the server before being pushed to the client and so all the major work occurs at the server level- it’s not noticeable on the client side.

In other words, during a server-side test, the variation is retrieved from the server and delivered to the user’s browser. As a result, no modifications take place on the front-end or browser, so there’s no flickering effect

Endless testing opportunities

Your teams have a lot more options to work with when it comes to server-side testing as it enables you to modify all aspects of your site.

As server-side tests are rendered from the back-end server, you can test more complex dynamic content unlike with client-side tests where it’s not easy to test such content and may damage the user experience if done on the client-side.

In other words, you can build tests that are much more complex that go deeper within your tech stack beyond the scope of UI or cosmetic changes to unlock a whole new world of experimentation.

With such deep experimentation capabilities, teams can thoroughly test all facets of a product to validate its functionality such as evaluating its underlying features, algorithms and back-end logic.

The next section will go over these various use cases where you should consider running server-side testing instead.

When does it make sense to move to server-side testing?

As we’ve mentioned, server-side tests are usually used to run more advanced types of tests and experiment deeply within your tech stack to thoroughly explore how a product works.

Put simply, server-side experimentation solutions allow teams to conduct more robust and secure experiments that are focused on modifying a product’s functions.

Here are some use cases where server side testing is recommended over client-side testing:

Run experiments on your mobile app

As already mentioned, one of the key advantages of server-side testing is enabling omnichannel and cross-platform experimentation.

As client-side solutions rely on Javascript and cookies, it’s not possible to use them to test on native mobile apps and you’re limited to devices that have a default web browser.

This means to run experiments on your mobile app, you will need a more advanced server-side testing solution to handle mobile app technologies, which are more complex and vastly different from web technologies.

Moreover, because server-side testing works on both web applications and mobile apps, you can run the same test serving the same variations irrespective of which channel they’re using. This allows you to compare data from each channel and optimize the user experience across all the different touch points accordingly.

Finally, if you use feature flags to conduct server-side testing on mobile apps, you can bypass the tedious and time-consuming app store approval. Put simply, feature flags enable you to turn functionality on or off remotely without redeploying code to app stores and waiting for its approval or having to wait for all changes to be ready at the same time to release your own changes.

Test your search algorithms

Server-side A/B tests are efficient for testing deeper level modifications related to the backend and the architecture of your website.

This is the case with search algorithms which require modification to your existing code. E-commerce websites usually turn to server-side testing to ensure that customers can easily browse through their website and find the product they’re looking for.

Thus, search algorithms are key in getting customers to find the product they’re looking for by providing a seamless search experience that eventually gets them to purchase.

For example, you can test what is displayed to customers in their search results, whether this should be based on pricing, popularity or reviews or whether you need to prioritize products based on what customers purchased/liked/viewed in the past.

Server-side testing allows you to create such complex scenarios and rules to provide customers with more personalized reocmmendations and optimize their search experience on your site. These are more difficult to test through client-side solutions as search pages are based on the search query and so they are rendered dynamically.

Thus, server-side testing offers more comprehensive testing and allows you to experiment with multiple algorithms by modifying the existing code.

Optimize your product recommendations

Similarly, with server-side tools, you can test multiple product recommendation algorithms typically found at the bottom of product pages to measure which types of layouts generate the most sales or increase average order value. For example, is it better to promote similar products, the most popular products or those that were recently viewed?

Such recommendations are based on a variety of criteria, like browsing history and on your own PIM (Product Information Management) system and other data sources. Server-side testing will allow you to test these multiple algorithms to uncover the best performing selection of recommended products.

Test data-sensitive pages

Server-side testing is great for testing on pages where data security is vital as it ensures that the data remains safe internally within the server without worrying about a security threat.

This is ideal to test on e-commerce payment pages or for banking applications, for example, or any other web pages or apps that contain sensitive data.

You can also use feature flags to test out a new payment method on a subset of users and see how they adapt before rolling it out to everyone else.

Determine the ideal form length

This is especially important for SaaS businesses that rely on free trial and request demo forms to gather information from their visitors. Thus, they need to determine the best length for such forms without risking drop-offs but still being able to gather all the necessary information about a prospect.

Server-side testing is the way to go in this scenario as your forms are directly linked to how your database is structured. If a field is obligatory, you will not be able to hide it using JavaScript because the form’s validation will fail server-side.

As a result, testing the length and complexity of a form that has the highest positive impact on conversion rates should be done on the server-side.

This also applies to other websites that utilize such forms from visitors such as hotel reservation or insurance sites. Note that different sectors will require different and more or less information depending on the type and purpose of the information being gathered.

Test the limit for free shipping

E-commerce businesses should be able to determine the order value over which free shipping is possible. This is important as shipping costs are one of the major causes for cart abandonment.

Therefore, since shipping cost is one of the determining factors in a customer’s purchase decision, companies should test out various cart value thresholds to find out the optimal limit for free shipping to improve transaction rates.

Since shipping costs are usually rendered dynamically from the back-end server, you will need to test them on the server-side. Any modifications made should have an impact on all of the following steps, and should be managed server-side.

Validate your features

Server-side tests allow you to conduct feature testing to validate your product features by rolling out variations of your feature to different groups of users to evaluate its performance before going for a general release.

With the help of feature flags, you can run server-side A/B tests and segment your users by directing them to one variation or the other. If anything goes wrong with a variation, you can easily roll it back by disabling the flag before it further impacts the user experience.

Finally, according to the feedback generated from these users, you can then optimize your features and release them to all your users with the confidence that it matches customers requirements.

Server-side tests and feature flags

The best way to run server-side tests is through feature flags. By decoupling deployment from release, you can conduct server-side A/B tests by rolling out new features to a small group of users. You can then measure their performance on this group of users before rolling out to everyone else.

While server-side testing requires technical expertise and coding skills, it’s not only relevant to tech teams. Often, non-technical staff team up with product teams to define experiments, which are then executed by the engineers. Once implemented, an experiment can usually be controlled, monitored, and analyzed via a dashboard.

With the right feature management solution, all teams across an organization can run server-side tests with the help of an easy-to-use dashboard without facing any technical hassles.

Client- vs server-side testing: Context matters

In the end, it’s important to note that it’s not a question of whether server-side is better than client-side. They’re both complementary approaches and whichever one a company chooses depends on which is better suited for its optimization and larger business goals.

In other words, one type of testing doesn’t replace the other. It’s a matter of looking at the type of experiment you want to run and which type is more suited in that particular context and which teams are looking to run the experiment- marketers tend to favor and use client-side testing more often while product managers and developers usually opt for server-side testing for their experimentation needs. It also depends on the resources businesses have at hand and the maturity of their optimization programs.

To ensure your website is optimized and provides a seamless user experience, having both testing techniques at hand is key to surviving in a competitive digital world.

Both types of testing are indispensable to help you build great products that will bring in maximum revenue. The key is to use both together to heighten productivity and achieve maximum impact.

You might also like...

See all

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Article

3min read

Tag V4: Elevating your Experience Building with an Improved Modification Engine

Julie Dumont

Feb 25, 2025

Article

2min read

New Visual Studio Code Extension: Dev-Friendly Experimentation & Personalization

Julie Dumont

Feb 21, 2025

Subscribe to
our Newsletter

Article

May 22, 2023

7min read

A/B Test Hypothesis Definition, Tips and Best Practices

Emily Healy

Incomplete, irrelevant or poorly formulated A/B test hypotheses are at the root of many neutral or negative tests.

Often we imagine that doing A/B tests to improve your e-commerce site’s performance means quickly changing the color of the “add to cart” button will lead to a drastic increase in your conversion rate, for example. However, A/B testing is not always so simple.

Unfortunately, implementing random changes to your pages won’t always significantly improve your results – there should be a reason behind your web experiments.

This brings us to the following question: how do you know which elements to experiment with and how can you create an effective AB test hypothesis?

Determine the problem and the hypothesis

Far too few people question the true origins of the success (or failure) of the changes they put in place to improve their conversion rate.

However, it’s important to know how to determine both the problem and the hypothesis that will allow you to obtain the best results.

Instead of searching for a quick “DIY” solution, it’s often more valuable in the long term to take a step back and do two things:

Identify the real problem – What is the source of your poor performance? Is it a high bounce rate on your order confirmation page, too many single-page sessions, a low-performing checkout CTA or something more complex?
Establish a hypothesis – This could show the root of the problem. For example, a great hypothesis for A/B testing could be: “Our customers do not immediately understand the characteristics of our products when they read the pages on our e-commerce site. Making the information more visible will increase the clicks on the “add-to-cart” button.”

The second step may seem very difficult because it requires a capacity for introspection and a critical look at the existing site. Nevertheless, it’s crucial for anyone who wants to see their KPIs improve drastically.

If you’re feeling a bit uncomfortable with this type of uncertainty around creating an effective hypothesis, know that you’ve come to the right place.

What is an A/B test hypothesis?

Technically speaking, the word hypothesis has a very simple definition:

“A proposal that seeks to provide a plausible explanation of a set of facts and which must be controlled against experience or verified in its consequences.”

The first interesting point to notice in this definition is “the set of facts to be explained.” In A/B testing, a hypothesis must always start with a clearly identified problem.

A/B tests should not be done randomly, or you risk wasting time.

Let’s talk about how to identify the problem:

Web analytics data – While this data does not explain digital consumers’ behavior exactly, it can highlight conversion problems (identifying abandoned carts, for example) and help prioritize the pages in need of testing.
Heuristic evaluation and ergonomic audit – These analyses allow you to assess the site’s user experience at a lower cost using an analysis grid.
User tests – This qualitative data is limited by the sample size but can be very rich in information that would not have been detected with quantitative methods. They often reveal problems understanding the site’s ergonomics. Even if the experience can be painful given the potential for negative remarks, it will allow you to gather qualified data with precise insights.
Eye tracking or heatmaps – These methods provide visibility into how people interact with items within a page – not between pages.
Customer feedback – As well as analyzing feedback, you can implement tools such as customer surveys or live chats to collect more information.

The tactics above will help you highlight the real problems that impact your site’s performance and save you time and money in the long run.

A/B test hypothesis formula

Initially, making an A/B test hypothesis may seem too simple. At the start, you mainly focus on one change and the effect it produces. You should always respect the following format: If I change this, it will cause that effect. For example:

Changing (the element being tested) from ___________ to ___________ will increase/decrease (the defined measurement).

At this stage, this formula is only a theoretical assumption that will need to be proven or disproven, but it will guide you in solving the problem.

An important point, however, is that the impact of the change you want to bring must always be measurable in quantifiable terms (conversion rate, bounce rate, abandonment rate, etc.).

Here are two examples of hypotheses phrased according to the formula explained above and that can apply to e-commerce:

Changing our CTA from “BUY YOUR TICKETS NOW” to “TICKETS ARE SELLING FAST – ONLY 50 LEFT!” will improve our sales on our e-commerce site.
Shortening the sign-up form by deleting optional fields such as phone and mailing address will increase the number of contacts collected.

In addition, when you think about the solution you want to implement, include the psychology of the prospect by asking yourself the following:

What psychological impact could the problem cause in the digital consumer’s mind?

For example, if your problem is a lack of clarity in the registration process which impacts the purchases, then the psychological impact could be that your prospect is confused when reading information.

With this in mind, you can begin to think concretely about the solution to correct this feeling on the client side. In this case, we can imagine that one fix could be including a progress bar that shows the different stages of registration.

Be aware: the psychological aspect should not be included when formulating your test hypothesis.

Once you have gotten the results, you should then be able to say whether it is true or false. Therefore, we can only rely on concrete and tangible assumptions.

Best practice for e-commerce optimization based on A/B hypotheses

There are many testable elements on your website. Looking into these elements and their metrics can help you create an effective test hypothesis.

We are going to give you some concrete examples of common areas to test to inspire you on your optimization journey:

HOMEPAGE

The header/main banner explaining the products/services that your site offers can increase customers’ curiosity and extend their time on the site.
A visible call-to-action appearing upon arrival will increase the chance visitors will click.
A very visible “about” section will build prospects’ trust in the brand when they arrive on the site.

PRODUCT SECTIONS

Filters save customers a lot of time by quickly showing them what they are looking for.
Highlighting a selection of the most popular products at the top of the sections is an excellent starting point for generating sales.
A “find out more” button or link under each product will encourage users to investigate.

PRODUCT PAGES

Product recommendations create a more personal experience for the user and help increase their average shopping cart
A visible “add to cart” button will catch the prospect’s attention and increase the click rate.
An “add to cart and pay” button saves the customer time, as many customers have an average of one transaction at a time.
Adding social sharing buttons is an effective way of turning the product listing into viral content.

Want to start A/B testing elements on your website? AB Tasty is the best-in-class experience optimization platform to help you convert more customers by leveraging intelligent search and recommendations to create a richer digital experience – fast. From experimentation to personalization, this solution can help you achieve the perfect digital experience with ease.

CART PAGE

The presence of logos such as “Visa certified” enhances customer confidence in the site.
A very visible button/link to “proceed to payment” greatly encourages users to click.

PAYMENT

A single page for payment reduces the exit rate.
Paying for an order without registration is very much appreciated by new prospects, who are not necessarily inclined to share their personal information when first visiting the site.
Having visibility over the entire payment process reassures consumers and will nudge them to finalize their purchase.

These best practices allow you to build your A/B test hypotheses by comparing your current site with the suggestions above and see what directly impacts conversion performance.

The goal of creating an A/B test hypothesis

The end goal of creating an A/B test hypothesis is to identify quickly what will help guarantee you the best results. Whether you have a “winning” hypothesis or not, it will still serve as a learning experience.

While defining your hypotheses can seem complex and methodical, it’s one of the most important ways for you to understand your pages’ performance and analyze the potential benefits of change.

You might also like...

See all

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Article

3min read

Tag V4: Elevating your Experience Building with an Improved Modification Engine

Julie Dumont

Feb 25, 2025

Article

2min read

New Visual Studio Code Extension: Dev-Friendly Experimentation & Personalization

Julie Dumont

Feb 21, 2025

Subscribe to
our Newsletter

Article

Mar 9, 2023

7min read

Google Optimize is Shutting Down: Don’t Wait to Migrate

Rowan Haddad

With Google Optimize’s retirement on the horizon, companies are now faced with the daunting task of finding an alternative tool to carry out their tests and experiments after Google’s announcement that it will be sunsetting its web testing and personalization tool on September 30.

If you haven’t already, it’s time to start your research for a new tool as soon as possible to stay on track with your testing and CRO strategies. September will be here faster than you think, so you need to act soon for a smooth migration post-sunset.

Why the time to find a Google Optimize alternative is now

With roughly 6 months left, teams should already be thinking about the best way for them to carry out their experiments.

You may have to anticipate that some features on Google Optimize will no longer work properly. Not to mention that migrating from one tool to another — and the transferring of all the data — can be a complicated process.

Making an informed decision requires extensive research. Finding a platform that suits your experimentation needs is only the first step. You also need to factor in how long it will take the migration processes to be successfully completed and the learning curve of your new tool. In other words, time is of the essence if you want to make a smooth migration to a new platform.

Therefore, teams need to make sure that they have fully migrated onto their new tool well before the sunset date of September 30.

What’s next? The current and future state of your optimization journey

As a free or low-cost solution, Google Optimize is a great starting point for those at the beginning of their optimization journey.

However, in light of the sunset announcement, organizations should start rethinking their website optimization and experimentation strategies and looking ahead to anticipate their CRO needs.

This should start with evaluating your current and future CRO goals. In other words, you may look into the possibility of investing more resources to optimize your website that will enable you to turn your passive users into active ones by providing a more personalized customer journey.

Consequently, your team may want to delve into other features beyond A/B testing capabilities offered by more advanced solutions. This will enable them to better optimize the website. For example, you may consider venturing beyond surface-level modifications and running more sophisticated tests tied to back-end architecture.

This will ultimately allow teams to provide the best customer experience possible so visitors can turn into customers with the click of a button.

Put simply, engaging your visitors along the entire customer journey up until the point of purchase or conversion should be a central part of your CRO strategy.

Taking into account all these factors will help you understand the current state of your CRO and whether it’s time to take your optimization roadmap to the next level.

How to prepare for a successful migration post-sunset

Selecting the right tool will take considerable time in terms of research and set-up. Therefore, early on, teams will need to follow some crucial steps to take them on the right track to a seamless transition from Google Optimize and to ensure successful implementation of the new tool.

Here’s a checklist for a successful migration:

- Evaluating your experimentation program: Analyzing your CRO strategy and results is important to help you set the requirements for your next testing tool.
- Considering your CRO strategy and budget: This will help you determine what kind of features you need and how scalable it is and the kind of budget you need to execute it.

- Selecting the tool: Evaluating alternative testing tools to suit your budget and needs (you may consider looking into Google-preferred partners for a smoother transition).

- Setting up and installing the tool on your website: This will include migrating all your data and tests from Google Optimize. You should consider if you will need coding experience and if you have sufficient developer resources for that. Otherwise, you should consider opting for low-code/no-code solutions instead. Additionally, you will need to run A/A tests to get acquainted with the new platform and ensure that it’s working as expected when it comes to data accuracy and level of significance.

- Integrating the tool with your stack: Take into account the tools you’re currently using and how they will fit together with the new tool.

- Offering internal training for the new tool: Depending on the kind of support you’ll be receiving from your new tool, you need to make sure that your teams can easily and efficiently use the tool.

Evaluating your experimentation program means taking stock of all the tests you’ve run on Google Optimize to understand what went well with the tool and where it fell short. This will give you an indication of your current situation and how you’d like to evolve when it comes to your testing strategy so you can pick your new tool accordingly.

Each step can take a considerable amount of time to complete so we recommend starting as early as possible.

How AB Tasty fits into your post-Optimize world

As a Google-preferred partner, AB Tasty provides you with best-in-class experience optimization tools to continue optimizing your digital experiences.

While Google Optimize also offered a 360 version with more advanced features, it had its limitations, especially for companies further along their CRO journey and looking for deeper experimentation capabilities.

Here are a few reasons why AB Tasty is the right choice post-Google Optimize to empower you to animate the entire digital consumer journey and take your testing to the next level:

- AB Tasty offers a variety of integrations that will fit seamlessly into your existing tech stack including seamless integrations with Google Analytics and other analytics providers to help you stay on top of your data.
- Explore endless possibilities with a library of widgets to optimize the customer journey. Activate your audience and engage users with banners and pop-ups among other flexible, visually appealing and impactful components.
- Worried about support during your switch? AB Tasty has dedicated CSMs and account managers to provide you with 1:1 support throughout the contract, including transferring your test history and data over from Google Optimize.

Take the next big step with AB Tasty

Are you still on the fence on whether AB Tasty is the right pick for you? Below you will find the answers to all your burning questions about our platform to help make your decision easier:

We’re here to answer your questions

What is AB Tasty’s take on privacy and security?

AB Tasty is GDPR compliant and committed to respecting the principles of this legislation, which consists of regulating data collection.

Can I use AB Tasty to target Google ads campaigns?

Yes, campaigns from Google ads can be easily triggered. With AB Tasty’s granular targeting capabilities, customers can target their visitors based on the campaign source in conjunction with any other events or metrics they see fit to provide the most personalized end-user experience possible.

What type of support do you offer in setting up and migrating to your tool?

Our in-house Customer Success team is on hand to support new and existing clients. If you need assistance setting up new campaigns or transferring existing ones then we can take on the heavy lifting for you to ensure a smooth transition.

Can you link Google Analytics with AB Tasty?

Yes, you can link Google Analytics with AB Tasty to be able to analyze your campaigns or you can have them sent to an in-house tool. You can find more information about our Google Analytics integration here.

How can we set up segmentation and personalization campaigns?

AB Tasty has a wide range of extremely granular targeting capabilities. This allows customers to target their visitors based on any other criteria/events/metrics of their choice to provide a more personalized user experience. This can all be set up in a matter of seconds with no code required.

How does AB Tasty differ from Google Optimize in terms of speed and performance?

AB Tasty has the lightest tag on the market available today that still offers complete functionality. Alongside our detailed performance centre which highlights where improvements can be made, customers can expect greater performance from AB Tasty than they ever had with Google Optimize. Find out more about how we compare here.

How many tests can I run and can they be run concurrently?

With our user-friendly experimentation suite, you can create an unlimited number of A/B tests and you can also run multiple tests simultaneously if needed.

Are you ready to make the move?

The post-Google Optimize world doesn’t have to be bleak.

As one of Google’s top picks as your new A/B testing platform, AB Tasty is a best-in-class A/B testing tool that helps you convert more customers by leveraging experimentation to create a richer digital experience – fast. This experience optimization platform embedded with AI and automation can help you achieve the perfect digital experience with ease.

You might also like...

See all

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Article

3min read

Tag V4: Elevating your Experience Building with an Improved Modification Engine

Julie Dumont

Feb 25, 2025

Article

2min read

New Visual Studio Code Extension: Dev-Friendly Experimentation & Personalization

Julie Dumont

Feb 21, 2025

Subscribe to
our Newsletter

Article

Dec 6, 2022

5min read

Test More, Risk Less: A/B Testing as a Risk Mitigation Tool

Rowan Haddad

Imagine this: You see a continual drop in conversions on your e-commerce website. In particular, customers are abandoning their carts and not completing their purchases.

Based on previous experience, you decide to take action by switching up a few elements on the checkout page.

But it’s still not getting you the results you’ve hoped for; instead, you see your conversions going even further down. What went wrong?

How experiments can come to the rescue

There’s always a certain degree of risk in business and any wrong move can be a potential loss in profit. How can you test your ideas without breaking your website and negatively impacting your business?

Thanks to A/B, A/B/C and multivariate testing, you can manage and even reduce that risk before making a big decision that could hurt your bottom line.

Before we go any further, let’s look at a simple definition of A/B testing.

A/B testing is a marketing technique that involves comparing two variations or versions of a web page or application, randomly presented to users, to see which performs better by evaluating how they impact your KPIs.

The results of such tests will help you assess the risk of your business decision as it presents an opportunity to gather feedback from the people with the most valuable opinions — your customers. Otherwise, you’d make the decision based on personal opinions rather than customer preferences, eventually leading you down the wrong path.

A/B testing helps steer you in the right direction by enabling you to test and learn quickly if something works or not before embedding it into the back-end or permanent coding.

If you don’t take steps to manage risks, you’d never be able to tell if your new ideas will resonate with your customers or if they’re worth the investment. However, when you run A/B tests, you can minimize the risk of any drastic business impact.

For example, you can make a pretty website by changing colors but if this change drives down conversions then you’ll have a nice looking but poor performing website. Thus, A/B testing gives you the green light to go through with an idea by monitoring how it’s affecting your KPIs.

Experimentation is the most efficient way to eliminate rolling out a bad idea for your website or prove the value of a change before investing time and resources. It’s a golden opportunity to learn what really drives conversions so you can use the data extracted to fully commit to any changes in UX.

Even if an experiment doesn’t turn out the way you planned, you can still use it as a learning experience on what your customers don’t want so that you stay on the right track.

To get the most out of your A/B tests, you need to leverage both quantitative and qualitative data when it comes to effective decision-making. In other words, running these experiments is important, but it’s the quick steps you take after based on the results that will make all the difference.

AB Tasty is a great example of an A/B testing tool that allows you to quickly set up tests and gather results to help you mitigate risk with ease. With low code implementation of front-end or UX changes on your web pages, you can gather insights via an ROI dashboard, determine which route will increase your revenue, and achieve the perfect digital experience for your customers.

In summary, always make sure you test new features or changes prior to release to make sure they take your business metrics in the right direction. Even if it seems like it’s a very minor change, it could still have a significant impact on your conversions and revenue.

Steer your A/B test in the right direction

While you’re running an experimentation campaign, there’s also another layer of risk when randomly allocating users to your variations as so often happens to be the case during an A/B test. What if there’s a variation that you notice is performing poorly? How can you quickly turn this off before more traffic is exposed to the variation?

Luckily, there’s a way to further minimize risk during an A/B test and that’s through dynamic allocation, a capability offered by advanced A/B testing solutions such as AB Tasty.

Dynamic allocation seeks to limit loss due to the lowest performing variations so that fewer visitors are sent to the “bad” variations to maximize outcome, as can be seen in the image below.

For example, if you run an A/B test with two variations with the goal to increase conversions on the checkout page and variation B is performing well and has a high conversion rate, then the traffic to that variation is adjusted (and increased) accordingly.

One of the advantages of dynamic allocation is risk mitigation. This will enable you to confidently and safely test new elements. If they don’t work out as predicted, then the traffic allocated will be gradually reduced so fewer users will access it.

Make risk-free and data-driven decisions with A/B testing

Any new changes or releases you have in the works shouldn’t be driven by your gut feeling but rather from a combination of quantitative and qualitative data, the kind of data and insights that you can only obtain from running tests and experiments. This enables you to optimize your website accordingly rather than speed up losses from a misguided UX change based solely on personal opinion.

A/B testing doesn’t have to be a risky endeavor. Features such as dynamic allocation can make all the difference when running A/B tests to avoid any significant loss in conversions from a poorly performing variation.

In the end, it’s a win-win situation: you get valuable insights while managing risk and your customers receive higher-quality products they actually want resulting in enhanced customer satisfaction.

With AB Tasty, you never have to lose a single conversion. Get started building your A/B tests today with our best-in-class software solution to explore how you can get maximum results with minimal effort thanks to our dynamic allocation capability.

You might also like...

See all

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Article

3min read

Tag V4: Elevating your Experience Building with an Improved Modification Engine

Julie Dumont

Feb 25, 2025

Article

2min read

New Visual Studio Code Extension: Dev-Friendly Experimentation & Personalization

Julie Dumont

Feb 21, 2025

Subscribe to
our Newsletter

Article

Nov 28, 2022

7min read

Sample Ratio Mismatch: What Is It and How Does It Happen?

Emily Healy

A/B testing can bring out a few types of experimental flaws.

Yes, you read that right – A/B testing is important for your business, but only if you have trustworthy results. To get reliable results, you must be on the lookout for errors that might occur while testing.

Sample ratio mismatch (SRM) is a term that is thrown around in the A/B testing world. It’s essential to understand its importance during experimentation.

In this article, we will break down the meaning of sample ratio mismatch, how to spot SRM, when it is and is not a problem, why it can happen and how to detect SRM.

Sample ratio mismatch overview

Sample ratio mismatch is an experimental flaw where the expected traffic allocation doesn’t fit with the observed visitor number for each testing variation.

In other words, an SRM is evidence that something went wrong.

Sample ratio mismatch is crucial to be aware of in A/B testing.

Now that you have the basic idea, let’s break this concept down piece by piece.

What is a “sample”?

The “sample” portion of SRM refers to the traffic allocation.

Traffic allocation refers to how the traffic is split toward each test variation. Typically, the traffic will be split equally (50/50) during an A/B test. Half of the traffic will be shown the new variation and the other half will go toward the control version.

This is how an equal traffic allocation will look for a basic A/B test with only one variant:

A/b testing equal traffic allocation

If your test has two variants or even three variants, the traffic will still be allocated equally to each test to ensure that each version receives the same amount of traffic. An equal traffic allocation in an A/B/C test will be split into 33/33/33.

For both A/B and A/B/C tests, traffic can be manipulated in different ways such as 60/40, 30/70, 20/30/50, etc. Although this is possible, it is not a recommended practice to get accurate and trustworthy results from your experiment.

Even by following this best practice guideline, equally allocated traffic will not eliminate the chance of an SRM. This type of mismatch is something that can still occur and must be calculated no matter the circumstances of the test.

Define sample ratio mismatch (SRM)

Now that we have a clear picture of what the “sample” is, we can build a better understanding of what SRM means:

SRM happens when the ratio of the sample does not match the desired 50/50 (or even 33/33/33) traffic allocation
SRM occurs when the observed traffic allocation to each variant does not match the allocation chosen for the test
The control version and variation receive undesired mismatched samples

Whichever words you choose to describe SRM, we can now understand our original definition with more confidence:

“Sample ratio mismatch is an experimental flaw where the expected traffic allocation doesn’t fit with the observed visitor number for each testing variation.”

sample ratio mismatch

Is SRM always a problem?

To put it simply, SRM occurs when one test version receives a noticeably different amount of visitors than what was originally expected.

Imagine that you have set up a classic A/B test: Two variations with 50/50 traffic allocation. You notice at one point that version A receives 10,000 visitors and version B receives 10,500 visitors.

Is this truly a problem? What exactly happened in this scenario?

The problem is that while conducting an A/B test, an extremely strict respect of the allocation scheme is not always 100% possible since it must be random. The small difference in traffic that is noted in the example above is something we would typically refer to as a “non-problem.”

If you are seeing a similar traffic allocation on your A/B test in the final stages, there is no need to panic.

A randomly generated traffic split has no way of knowing exactly how many visitors will stumble upon the A/B test during the given time frame of the test. This is why toward the end of the test duration, there may be a smaller difference in the traffic allocation while the majority (+95%) of traffic is correctly allocated.

When is SRM a problem?

Some tests may have SRM due to experimental setup.

When the SRM is a big problem, there will be a noticeable difference in traffic allocation.

If you see 1,000 directed to one variant and 200 directed to the other — this is an issue. Sometimes, spotting SRM does not require a particular mathematical formula dedicated to calculating SRM as it is evident enough on its own.

However, an extreme difference in traffic allocation can be very rare. Therefore, it’s essential to check the visitor counts in an SRM test before each test analysis.

Does SRM occur frequently?

Sample ratio mismatch can happen more often than we think. According to a study done by Microsoft & Booking, about 6% of experiments experience this problem.

Furthermore, if the test includes a redirect to an entirely new page, SRM can be even more likely.

Since we heavily rely on tests and trust their conclusions to make strategic business decisions, it’s important that you are able to detect SRM as early as possible when it happens during your A/B test.

Can SRM still affect tests using Bayesian?

The reality is that everyone needs to be on the lookout for SRM, no matter what type of statistical test they are running. This includes experiments using the Bayesian method.

There are no exemptions to the possibility of experiencing a statistically significant mismatch between the observed and expected results of a test. No matter the test, if the expected assumptions are not met, the results will be unreliable.

Sample ratio mismatch: why it happens

Sample ratio mismatch can happen due to a variety of different root causes. Here we will discuss three common examples that cause SRM.

One common example is when the redirection to one variant isn’t working properly for poorly connected visitors.

Another classic example is when the direct link to one variant is spread on social media, which brings all users who click on the link directly to one of the variants. This error does not allow the traffic to be properly distributed among the variants.

In a more complex case, it’s also possible that a test including JS code is crashing a variant and therefore some of the visitor configurations. In this situation, some visitors that are being sent to the crashing variant won’t be collected and indexed properly, which leads to SRM.

All of these examples have a selection bias: some non-random visitors are excluded. The non-random visitors are arriving directly from a link shared on social media, have a poor connection, or are visiting a crashing variant.

In any case, when these issues occur, the SRM is an indication that something went wrong and you cannot trust the numbers and the test conclusion.

Checking for SRM in your A/B tests

Something important to be aware of when doing an SRM check is that the priority metric when checking needs to be “users” and not “visitors.” Users are the specific people that are allocated to each variation, meanwhile, the visitors metric is counting the number of sessions that each user makes.

It’s important to differentiate between users and visitors because results may be skewed if a visitor comes back to their variation multiple times. SRM detected with “visitors” may not be the most reliable metric, but using the “users” metric is evidence of a problem.

SRM in A/B testing

Testing for sample ratio mismatch may seem a bit complicated or unnecessary at first glance. In reality, it’s quite the opposite.

Understanding what SRM is, why it happens, and how it can affect your results is crucial in A/B testing. Running an A/B test to help make key decisions is only helpful for your business if you have reliable data from those tests.

Want to get started on A/B testing for your website? AB Tasty is a great example of an A/B testing tool that allows you to quickly set up tests with low code implementation of front-end or UX changes on your web pages, gather insights via an ROI dashboard, and determine which route will increase your revenue.

You might also like...

See all

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Article

3min read

Tag V4: Elevating your Experience Building with an Improved Modification Engine

Julie Dumont

Feb 25, 2025

Article

2min read

New Visual Studio Code Extension: Dev-Friendly Experimentation & Personalization

Julie Dumont

Feb 21, 2025

Subscribe to
our Newsletter

Article

Nov 17, 2022

6min read

How to solve real user problems with a CRO strategy

AB Tasty

Catch up on the previous installment of our Customer-Centric Data Series, How to Become a Data-Centric Company, or read the series introduction.

In the next installment of our series on a data-driven approach to customer-centric marketing, we spoke with our partner Raoul Doraiswamy, Founder & Managing Director of Conversionry to understand the flow of a customer-centric experimentation process, and why it is critical to tap into insights from experimentation processes to make better decisions.

What do you find is the biggest gap in the marketing & growth knowledge among brands right now?

Many brands today have the right set of tools such as technology investments, or the right people with marketing expertise. However, brands often face the issue of not knowing how to meet customer needs/how to give their customers what they want whether on their website, app or through digital advertising on the website, app or digital advertising – in other words, how can these brands increase conversions? Raoul identifies the lack of customer understanding to be at the core of this gap and suggests that brands should adopt a customer-centric, customer-driven process that enables a flow of customer insights, complemented by experimentation.

Which key activities deliver the best insights into customer problems?

Raoul believes that to start a strategy that puts the customers at the core, it is important to have the right data-gathering approach to get insights. It’s the foundation of any experimentation program, but can be applied to all marketing channels.

“Imagine you are an air traffic controller. You have multiple screens constantly feeding you where the planes are, or when they might crash into each other. From all these constant insights, the person in front of the screens will have to make the right decisions,” he shares. “However, there are also inconsequential insights such as baggage holders being full – and it is up to the decision-makers to pick out the critical data and make use of them.”

Raoul provides this analogy to liken it to the role of marketing decision-makers, who normally have a dashboard with metrics like revenue, conversion rate, abandoned cart and more. An insights dashboard helps marketers better understand their customers, combining this real-time data with customer feedback from sources like analytics, heatmaps, session recordings, social media comments and user testing. Solid research can be done through a critical analysis of session recordings and user poll forms, and the main takeaways can be fed to this dashboard. How empowering is that for a marketing decision-maker?

Where are the best sources for experimentation ideas?

Raoul asserts that a combination of quantitative and qualitative analysis is key. Heuristic analysis and competitor analysis are also gold when coming up with experimentation ideas. He continues, “Don’t limit yourself to looking at competitors, look at other industries too. For example, for a $90M trade tools client we had to solve the problem of increasing user sign-ins to their loyalty program. By researching Expedia and Qantas, we got the idea to show users points instead of cash to pay for items.” Raoul shares, “Do heat map analysis, look at session recordings, user polls, run surveys to email databases, and user testing. User testing is critical in understanding the full picture.”

After distilling customer problems and coming up with some rough experimentation ideas, the next step is to flesh out your experiment ideas fully. “Going back to the analogy of the Air Traffic Controller, one person on the team is seeing a potential crash but might have limited experience in dealing with this situation. That’s when more perspectives can be brought in by, let’s say, a supervisor, to make a more well-rounded decision. In the same way, when you are ideating, you do not want to just limit it to yourself but rather have a workshop where you discuss ideas with your internal team. If you are working with an agency, you can still have a workshop with both the agency and the client present, or have your CRO team and product team come together to share ideas. This way, you can get multiple stakeholders involved, each of them being able to provide expertise based on their experience with customers,” says Raoul.

Is there value in running data-gathering experiments (as opposed to improving conversion / driving a specific metric)?

“Yes, absolutely,” replies Raoul. “Aligning growth levers with clients every quarter while working with CRO and Experimentation teams on the experimentation process is important. When working towards the goal of increasing conversions, there are KPIs and predictive models to project the goals.

“On the other hand, if the focus of the program is on product feature validation or reducing the risk of revenue due to untested features, there will be a separate metric for that,” he continues. “It is key to have specific micro KPIs for the tests that are running to generate a constant flow of insights, which then allows us to make better decisions.”

In running data-gathering experiments, features such as personalization can be applied which can have a positive impact on the conversions on the website.

What do brands need to get started?

“To begin, you need to start running experiments. Every day without a test is a day lost in revenue!” heeds Raoul. “For marketing leaders who have yet to start running experiments, you can start by pinpointing customer problems, and the flow of insights. To get the insights, you can gather them from Google Analytics, more specifically, by looking at your funnel. Through these insights, identify the drop-off point and observe the Next Page Path, to see where users go next.

“Take for example an eCommerce platform. If the users are dropping off at the product page instead of adding to the cart and moving on to the shipping page, this shows that they are confused about the shipping requirements. This alone can tell you what goes through the user’s mind. Look at heat maps and session recordings to understand the customer’s problems. The next step then is to solve the issue and to do that, you will need an A/B testing platform. Use the A/B testing platform to build tests and launch them as quickly as possible.”

As for established marketing teams who are already doing some testing, Raoul recommends gathering insights and customer problems as they come in every month. “Then to make sense of the data you’ve collected, you need conversion optimization analysts like our experts at Conversionry who are experienced in distilling data down to problems.”

Identifying customer problems is key. If some of the issues your customers encounter stay unaddressed, it could lead to the initiatives flatlining despite months of experimentation. Instead by keeping customer feedback top of mind, you can start designing, development, testing, speak to experience optimization platforms like AB Tasty to build the experiments, then gather insights, and repeat the cycle to see what wins and what doesn’t.

Get started building your A/B tests today with the best-in-class software solution, AB Tasty. With embedded AI and automation, this experimentation and personalization platform creates richer digital experiences for your customers, fast.

You might also like...

See all

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Article

3min read

Tag V4: Elevating your Experience Building with an Improved Modification Engine

Julie Dumont

Feb 25, 2025

Article

2min read

New Visual Studio Code Extension: Dev-Friendly Experimentation & Personalization

Julie Dumont

Feb 21, 2025

Subscribe to
our Newsletter

Article

Jul 26, 2022

6min read

Using Failed A/B Test Results to Drive Innovation

Maddie Ostrander

“Failure” can feel like a dirty word in the world of experimentation. Your team spends time thinking through a hypothesis, crafting a test, and finally when it rolls out … it falls flat. While it can feel daunting to see negative results from your a/b tests, you have gained valuable insights that can help you make data-driven, strategic decisions for your next experiment. Your “failure” becomes a learning opportunity.

Embracing the risk of negative results is a necessary part of building a culture of experimentation. On the first episode of the 1,000 Experiments Club podcast, Ronny Kohavi (formerly of Airbnb, Microsoft, and Amazon) shared that experimentation is a time where you will “fail fast and pivot fast.” As he learned while leading experimentation teams for the largest tech companies, your idea might fail. But it is your next idea that could be the solution you were seeking.

“There’s a lot to learn from these experiments: Did it work very well for the segment you were going after, but it affected another one? Learning what happened and why will lead to developing future strategies and being successful,” shares Ronny.

In order to build a culture of experimentation, you need to embrace the failures that come with it. By viewing negative results as learning opportunities, you build trust within your team and encourage them to seek creative solutions rather than playing it safe. Here are just a few benefits to embracing “failures” in experimentation:

Encourage curiosity: With AB Tasty, you can test your ideas quickly and easily. You can bypass lengthy implementations and complex coding. Every idea can be explored immediately and if it fails, you can get the next idea up and running without losing speed, saving you precious time and money.
Eliminate your risks without a blind rollout: Testing out changes on a few pages or with a small audience size can help you gather insights in a more controlled environment before planning larger-scale rollouts.
Strengthen hypotheses: It’s easy to fall prey to confirmation bias when you are afraid of failure. Testing out a hypothesis with a/b testing and receiving negative results confirms that your control is still your strongest performer, and you’ll have data to support the fact that you are moving in the right direction.
Validate existing positive results: Experimentation helps determine what small changes can drive a big impact with your audience. Comparing negative a/b test results against positive results for similar experiments can help to determine if the positive metrics stand the test of time, or if an isolated event caused skewed results.

In a controlled, time-limited environment, your experiment can help you learn very quickly if the changes you have made are going to support your hypothesis. Whether your experiment produces positive or negative results, you will gain valuable insights about your audience. As long as you are leveraging those new insights to build new hypotheses, your negative results will never be a “failure.” Instead, the biggest risk would be allowing a status quo continuing to go unchecked.

“Your ability to iterate quickly is a differentiation,” shares Ronny. “If you’re able to run more experiments and a certain percentage are pass/fail, this ability to try ideas is key.”

Below are some examples of real-world a/b tests and the crucial learnings that came from each experiment:

Lesson learned: Removing “Add to Basket” CTAs decreased conversion

In this experiment, our beauty/cosmetics client tested removing the “Add to Basket” CTA from their product pages. The idea behind this was to test if users would be more interested in clicking through to the individual pages, leading to a higher conversion rate. The results? While there was a 0.4% increase in visitors clicking “Add to Basket,” conversions were down by 2%. The team took this as proof that the original version of the website was working properly, and they were able to reinvest their time and effort into other projects.

Beauty client add to basket use case

Lesson learned: Busy form fields led to decreased leads

A banking client wanted to test if adjusting their standard request form would drive passage to step 2 and ultimately increase the number of leads from form submissions. The test focused on the mandatory business identification number field, adding a pop-up explaining what the field meant in the hopes of reducing form abandonment. The results? They saw a 22% decrease in leads as well as a 16% decrease in the number of visitors continuing to step 2 of the form. The team’s takeaways from this experiment were that in trying to be helpful and explain this field, their visitors were overwhelmed with information. The original version was the winner of this experiment, and the team saved themselves a huge potential loss from hardcoding the new form field.

Lesson learned: Product availability couldn’t drive transactions

The team at this beauty company designed an experiment to test whether displaying a message about product availability on the basket page would lead to an increase in conversions by appealing to the customer’s sense of FOMO. Instead, the results proved inconclusive. The conversion rate increased by 1%, but access to checkout and the average order value decreased by 2% and 0.7% respectively. The team determined that without the desired increase in their key metrics, it was not worth investing the time and resources needed to implement the change on the website. Instead, they leveraged their experiment data to help drive their website optimization roadmap and identify other areas of improvement.

Beauty client availability use case

Despite negative results, the teams in all three experiments leveraged these valuable insights to quickly readjust their strategy and identify other places for improvement on their website. By reframing the negative results of failed a/b tests into learning opportunities, the customer experience became their driver for innovation instead of untested ideas from an echo chamber.

Jeff Copetas, VP of E-Commerce & Digital at Avid, stresses the importance of figuring out who you are listening to when building out an experimentation roadmap. “[At Avid] we had to move from a mindset of ‘I think …’ to ‘let’s test and learn,’ by taking the albatross of opinions out of our decision-making process,” Jeff recalls. “You can make a pretty website, but if it doesn’t perform well and you’re not learning what drives conversion, then all you have is a pretty website that doesn’t perform.”

Through testing you are collecting data on how customers are experiencing your website, which will always prove to be more valuable than never testing the status quo. Are you seeking inspiration for your next experiment? We’ve gathered insights from 50 trusted brands around the world to understand the tests they’ve tried, the lessons they’ve learned, and the successes they’ve had.

You might also like...

See all

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Article

3min read

Tag V4: Elevating your Experience Building with an Improved Modification Engine

Julie Dumont

Feb 25, 2025

Article

2min read

New Visual Studio Code Extension: Dev-Friendly Experimentation & Personalization

Julie Dumont

Feb 21, 2025

Subscribe to
our Newsletter

Article

Apr 22, 2022

9min read

How to Create a Modern Data Foundation for Experimentation

AB Tasty

Staying ahead of the game to deliver seamless brand experiences for your customers is crucial in today’s experience economy. Today we’ll dip our toe into the “how” by looking at the underlying foundation upon which all of your experiences, optimization and experimentation efforts will be built: data.

Data is the foundation experimentation is built on (Source)

Data is the technology that can power the experiences you build for your customers by first understanding what they want and how it’ll best serve your business to deliver this. It’s the special sauce that helps connect the dots between your interpretation of existing information and trends, and the outcomes that you hypothesize will address customer needs (and grow revenue).

If you’ve ever wondered whether the benefits of a special offer are sufficiently enticing for your customer or why you have so many page hits and so few purchases, then you’ve asked the questions the marketing teams of your competitors are both asking and actively working to answer. Data and experimentation will help you take your website to the next level, better understand your customers’ preferences, and optimize their purchasing journey to drive stronger business outcomes.

So, the question remains: Where do you start? In the case of e-commerce, A/B testing is a great way to use data to test hypotheses and make decisions based on information rather than opinions.

A/B testing helps brands make decisions based on data (Source)

“The idea behind experimentation is that you should be testing things and proving the value of things before seriously investing in them,” says Jonny Longden, head of the conversion division at agency Journey Further. “By experimenting…you only do the things that work and so you’ve already proven [what] will deliver value.”

Knowing and understanding your data foundation is the platform upon which you’ll build your knowledge base and your experimentation roadmap. Read on to discover the key considerations to bear in mind when establishing this foundation.

Five things to consider when building your data foundation

Know what data you’re collecting and why
Knowing what you’re dealing with when it comes to slicing and dicing your data also requires that you understand the basic types and properties of the information to which you have access. Firstly, let’s look at the different types of data:
- First-party data is collected directly from customers, site visitors and followers, making it specific to your products, consumers and operations.
- Second-party data is collected by a secondary party outside of your company or your customers. It’s usually obtained through data-sharing agreements between companies willing to collaborate.
- Third-party data is collected by entirely separate organizations with no consideration for your market or customers; however, it does allow you to draw on increased data points to broaden general understanding.
Data also has different properties or defining characteristics: demographic data tells you who, behavioral data tells you how, transactional data tells you what, and psychographic data tells you why. Want to learn more? Download our e-book, “The Ultimate Personalization Guide”!

‎‎ㅤ‎‎

Gathering and collating a mix of this data will then allow you to segment your audience and flesh out a picture of who your customers are and how to meet their needs, joining the dots between customer behavior and preferences, website UX and the buyer journey.

Chad Sanderson, head of product – data platform at Convoy, recommends making metrics your allies to ensure data collection and analysis are synchronized. Knowing what your business leaders care about, and which metrics will move the business forward, will ensure that your data foundation is relevant and set up for success.

‎ㅤ
Invest in your data infrastructure
Data is everywhere, in its myriad of forms and gathered from a multitude of sources. Even so, if you’re going to make use of it, you need a robust system for gathering, storing and analyzing it in order to best put it to work. Start by understanding how much first-party data you have the capacity to gather by evaluating your current digital traffic levels. How many people are visiting your site or your app? You can get this information using Google Analytics or a similar platform, and this will help you understand how sophisticated your data-leveraging practices can be and identify gaps where you might need to source supplementary data (second- and third-party).
Next, you’ll need to evaluate your infrastructure. Companies that are further on their data analytics journey will invest in customer data platforms (CDPs) that allow them to collect and analyze data – gathered from a variety of sources and consolidated into a central database – at a more granular level. Stitching together this data via a CDP helps you bring all the pieces together to form a complete picture of your customers and identify any gaps. This is a critical step before you leap into action. Chad Sanderson concurs. “[Start] with the business and what the business needs,” he advises. “Tailoring your… solution to that – whatever that is – is going to be a lot more effective.”‎
ㅤ
Get consent to build consumer trust
Data security is rightly of foremost concern to consumers. The very users from whom you want to gather that first-party data want to ensure that their private information remains secure. Getting their consent and being transparent about the inherent benefit to them if they agree to your request – be it through giveaways, exclusive offers, additional information or services – will give you the best chance of success. Demonstrating that you adhere to, and take seriously, various data compliance laws (such as GDPR) and good governance will also build trust in your brand and give you the opportunity to make it worth their while through improved UX and personalized experiences.

Build trust in your brand by respecting your users’ private information (Source)

‎ㅤ
Collect and discover insights to upgrade your customer strategy
We’ve already covered the fact that data is everywhere. As Chad Sanderson highlighted above, identifying immediate business needs and priorities – as well as focusing on quick wins and low-lift changes that can have a quick and high-level impact – can help you navigate through this minefield. It’s best to think of this section as a four-step process:
ㅤㅤ• Collect data as it flows into your CDP
ㅤㅤ• Transform or calibrate your data so that it can be compared in a
ㅤ ㅤlogical manner
ㅤㅤ• Analyze the data by grouping and categorizing it according to
ㅤ ㅤthe customer segments you’ve identified and benchmarking
ㅤ ㅤagainst business priorities
ㅤㅤ• Activate your insights by pushing the learnings back into
ㅤ ㅤyour platforms and/or your experimentation roadmap and really
ㅤ ㅤput this data to work
ㅤ
Turn your data into actions
It’s crunch time (no pun about numbers intended)! We’ve examined the different types of data and where to source them, how to be responsible with data collection and how to set up the infrastructure needed to consolidate data and generate insights. We’ve also covered the need to understand business priorities and core strategy to drive data collection, analysis and activation in the same direction. Now we need to put that data and those insights to work.
In the experience economy, where constant evolution is the name of the game, innovation and optimization are the key drivers of experimentation. Taking the data foundation that you’ve built and using it to fuel and nourish your experimentation roadmap will ensure that none of the hard work of your tech, marketing and product teams is in vain. Testing allows you to evaluate alternatives in real time and make data-driven decisions about website UX. It also ensures that business metrics are never far from reach, where conversion and revenue growth take center stage.
Use the data you’ve gathered to fuel your experimentation roadmap (Source)

Invest in a solid data foundation to maximize and scale

At AB Tasty, we apply the Bayesian approach to interpreting data and test results because in A/B testing, this method not only shows whether there is a difference between the tested options but also goes beyond that by calculating a measure of that difference. Being able to identify what that variance is allows you to best understand what you will gain by adopting a permanent change.

Collecting and analyzing data, and then leveraging the insights that you glean, are key to unlocking the next level of experience optimization for your customers and your business. An experimentation roadmap grounded in real-time responsiveness and long-term, server-side improvements will have a solid data foundation approach at its core, where understanding who you want to target and how to act drives success. Furthermore, if you invest in your data foundation – and the five core drivers we’ve explored above – you’ll be equipped to scale your experimentation and allow optimization to become a key business maximizer.