Author: Emily Healy

Article

Jun 7, 2023

8min read

10 Generative AI Ideas for Your Experimentation Roadmap

Emily Healy

Artificial intelligence has been a recurring theme for decades. However, it’s no longer science fiction – it’s a reality.

Since OpenAI launched its own form of generative AI, ChatGPT, in November 2022, the world has yet to stop talking about its striking capabilities. It’s particularly fascinating to see just how easy it is to get results after interacting with this bot which is comprised of deep-learning algorithms for natural language processing.

Even Google quickly followed by launching a new and experimental project, Gemini, to revolutionize its own Search. By harnessing the power of generative AI and the capacity of large language models, Google is seeking to take its search process to the next level.

Given the rapid growth of this technological advancement over the past few months, it’s time that we talk about generative AI in the context of A/B testing and experimentation.

Whether you’re curious about how AI can impact your experiments or are ready for inspiration we’ll discuss some of our ideas around using AI for A/B testing, personalization, and conversion rate optimization.

What is generative AI?

Generative AI is a type of artificial intelligence that doesn’t have programming limitations, which allows it to generate new content (think ChatGPT). Instead of following a specific, pre-existing dataset, generative AI learns from indexing extensive data, focusing on patterns and using deep learning techniques and neural networks to create human-like content based on its learnings.

The way algorithms capture ideas is similar to how humans gather inspiration from previous experiences to create something unique. Based on the large amounts of data used to craft generative AI’s learning abilities, it’s capable of outputting high-quality responses that are similar to what a human would create.

However, some concerns need to be addressed:

Biased information: Artificial intelligence is only as good as the datasets used to train it. Therefore if the data used to train it has biases, it may create “ideas” that are equally biased or flawed.
Spreading misinformation: There are many concerns about the ethics of generative AI and sharing information directly from it. It’s best practice to fact-check any content written by AI to avoid putting out false or misleading information.
Content ownership: Since content generated with AI is not generated by a human, can you ethically can claim it as your own idea? In a similar sense, the same idea could potentially be generated elsewhere by using a similar prompt. Copywriting and ownership are then called into question here.
Data and privacy: Data privacy is always a top-of-mind concern. With the new capabilities of artificial intelligence, data handling becomes even more challenging. It’s always best practice to avoid using sensitive information with any form of generative AI.

By keeping these limitations in mind, generative AI has the potential to streamline processes and revolutionize the way we work – just as technology has always done in the past.

10 generative AI uses for A/B testing

In the A/B testing world, we are very interested in how one can harness these technological breakthroughs for experimentation. We are brainstorming a few approaches to re-imagine the process of revolutionizing digital customer experiences to ultimately save time and resources.

Just like everyone else, we started to wonder how generative AI could impact the world of experimentation and our customers. Here are some ideas, some of them concrete and some more abstract, as to how artificial intelligence could help our industry:

DISCLAIMER: Before uploading information into any AI platform, ensure that you understand their privacy and security practices. While AI models strive to maintain a privacy standard, there’s always the risk of data breaches. Always protect your confidential information.

1. Homepage optimization

Your homepage is likely the first thing your visitors will see so optimization is key to staying ahead of your competitors. If you want a quick comparison of content on your homepage versus your competitors, you can feed this information into generative AI to give it a basis for understanding. Once your AI is loaded with information about your competitors, you can ask for a list of best practices to employ to make new tests for your own website.

2. Analyze experimentation results

Reporting and analyzing are crucial to progressing on your experimentation roadmap, but it’s also time-consuming. By collecting a summary of testing logs, generative AI can help highlight important findings, summarize your results, and potentially even suggest future steps. Ideally, you can feed your A/B test hypothesis as well as the results to show your thought process and organization. After it recognizes this specific thought process and desired results, it could aid in generating new test hypotheses or suggestions.

3. Recommend optimization barriers

Generative AI can help you prioritize your efforts and identify the most impactful barriers to your conversion rate. Uploading your nonsensitive website performance data gathered from your analytics platforms can give AI the insight it needs into your performance. Whether it suggests that you update your title tags or compress images on your homepage, AI can quickly spot where you have the biggest drop-offs to suggest areas for optimization.

4. Client reviews

User feedback is your own treasure trove of information for optimization. One of the great benefits of AI that we already see is that it can understand large amounts of data quickly and summarize it. By uploading client reviews, surveys and other consumer feedback into the database, generative AI can assist you in creating detailed summaries of your users’ pain points, preferences and levels of satisfaction. The more detailed your reviews – the better the analysis will be.

5. Chatbots

Chatbots are a popular way to communicate with website visitors. As generative AI is a large language model, it can quickly generate conversational scripts, prompts and responses to reduce your brainstorming time. You can also use AI to filter and analyze conversations that your chatbot is already having to determine if there are gaps in the conversation or ways to enhance its interaction with customers.

6. Translation

Language barriers can limit a brand that has a presence in multiple regions. Whether you need translations for your chatbot conversations, CTAs or longer form copy, generative AI can provide you with translations in real time to save you time and make your content accessible to all zones touched by your brand.

7. Google Adwords

Speed up brainstorming sessions by using generative AI to experiment with different copy variations. Based on the prompts you provide, AI can provide you with a series of ideas for targeting keywords and creating copy with a particular tone of voice to use with Google Adwords. Caution: be sure to double-check all keywords proposed to verify their intent.

8. Personalization

Personalized content can be scaled at speed by leveraging artificial intelligence to produce variations of the same messages. By customizing your copy, recommendations, product suggestions and other messages based on past user interactions and consumer demographics, you can significantly boost your digital consumer engagement.

9. Product Descriptions

Finding the best wording to describe why your product is worth purchasing may be a challenge. With generative AI, you can get more ambitious with your product descriptions by testing out different variations of copy to see which version is the most promising for your visitors.

10. Predict User Behavior

Based on historical data from your user behavior, generative AI can predict behavior that can help you to anticipate your next A/B test. Tailoring your tests according to patterns and trends in user interaction can help you conduct better experiments. It’s important to note that predictions will be limited to patterns interpreted by past customer data collected and uploaded. Using generative AI is better when it’s used as a tool to guide you in your decision-making process rather than to be the deciding force alone.

The extensive use of artificial intelligence is a new and fast-evolving subject in the tech world. If you want to leverage it in the future, you need to start familiarizing yourself with its capabilities.

Keep in mind that it’s important to verify the facts and information AI generates just as you carefully verify data before you upload. Using generative AI in conjunction with your internal experts and team resources can assist in improving ideation and efficiency. However, the quality of the output from generative AI is only as good as what you put in.

Is generative AI a source of competitive advantage in A/B testing?

The great news is that this technology is accessible to everyone – from big industry leaders like Google to start-ups with a limited budget. However, the not-so-great news is that this is available to everyone. In other words, generative AI is not necessarily a source of competitive advantage.

Technology existing by itself does not create more value for a business. Rather, it’s the people driving the technology who are creating value by leveraging it in combination with their own industry-specific knowledge, past experiences, data collection and interpretation capabilities and understanding of customer needs and pain points.

While we aren’t here to say that generative AI is a replacement for human-generated ideas, this technology can definitely be used to complement and amplify your already-existing skills.

Leveraging generative AI in A/B testing

From education to copywriting or coding – all industries are starting to see the impact that these new software developments will have. Leveraging “large language models” is becoming increasingly popular as these algorithms can generate ideas, summarize long forms of text, provide insights and even translate in real-time.

Proper experimentation and A/B testing are at the core of engaging your audience, however, these practices can take a lot of time and resources to accomplish successfully. If generative AI can offer you ways to save time and streamline your processes, it might be time to use it as your not-so-secret weapon. In today’s competitive digital environment, continually enhancing your online presence should be at the top of your mind.

Want to start optimizing your website? AB Tasty is the best-in-class experience optimization platform that empowers you to create a richer digital experience – fast. From experimentation to personalization, this solution can help you activate and engage your audience to boost your conversions.

You might also like...

See all

Article

3min read

Experiment Health Check: Proactive Monitoring for Reliable Experimentation

Emily Healy

Jul 1, 2025

Article

9min read

Heatmaps: Your Team’s Secret Weapon for Uncovering Website Gold

AB Tasty

Jun 24, 2025

Article

6min read

9 AI Features that Transform How Digital Teams Test, Learn, and Grow

Emily Healy

Jun 20, 2025

Subscribe to
our Newsletter

Article

May 30, 2023

8min read

Digital Consumer Engagement: Activate Your Mid-Market Audience

Emily Healy

One of the biggest challenges for mid-market businesses is keeping your website visitors engaged. At this point, you’ve grown your brand enough to have a significant amount of traffic, but are in search of the right strategy to nudge your passive visitors into paying customers.

When you have a significant number of passive visitors, enticing them to take action is crucial to growing your commercial activity, achieving your business objectives and creating loyalty.

After working diligently to get traffic on your website, how do you engage these visitors?

In this article, we will answer that burning question with some practical strategies to ultimately improve your conversions.

What are passive users?

Passive users are visitors on your website that do not take any action.

These individuals may browse through some content on your site, but they leave before impacting your conversion rates in a positive way.

They are simply just browsing your web pages instead of filling out forms, subscribing to newsletters, engaging with CTAs, or making a purchase.

How to identify passive users

To engage passive users, you have to identify them.

Website analytics tools, such as Google Analytics, can help you identify these particular users by recognizing patterns when you’re analyzing your users’ behavior.

With analytics tools, you can see:

Which pages are more likely to attract passive visitors
The number of passive visitors on your website
Which pages attract active users
The actions taken on each page
The amount of time visitors spend on each page
Where your visitors come from
And more

Note: Many analytics tools provide more metrics than you can manage. In other words, it’s easy to get lost in the sea of data and metrics. It’s important to identify key performance indicators to measure your user behavior consistently.

This information can help you understand user behavior and recognize what motivates visitors to take action if you analyze it correctly. Once you can identify these motivators, you can begin to create a strategy to employ similar tactics on other pages for consistency in your engagement.

Identifying and understanding your digital consumers’ behavior is the first step to reaching your conversion goals.

Why you need to engage your audience

Converting passive visitors into paying customers is crucial for success. A higher conversion rate means more revenue and growth opportunities.

To recap: in order to engage your audience, you will need to understand them based on the data you collect from their behavior. This collection of information will help you develop best practices and develop guidelines for your future endeavors. Not to mention, you will also see what doesn’t work.

Mastering this first step to engaging your audience will, at the same time, help you to create a more seamless and attractive digital customer experience.

Whether your goal is to convert more leads on your landing page, increase your sign-up form conversions, increase brand awareness or nudge visitors to add items to their shopping cart, you must capture your audience to make these objectives a reality.

As your company increases conversions, your brand reputation will grow in parallel. A great reputation attracts new customers and creates further brand loyalty.

Now the question is, how do you engage your users in the mid-market?

How to engage visitors with widgets

Unfortunately, maximizing your digital consumer engagement won’t happen overnight. Based on your consumer research on your website, you will need to create a plan to optimize your digital customer journey.

Offering incentives such as discounts, free trials or exclusive offers is one great way to engage passive website visitors. By doing this you are providing a reason for users to take action.

How can you achieve such engagement at a mid-market level? Leveraging widgets.

With little to no-code needed, widgets are the best tools you can use for your website optimization strategy. Deploying flexible, visually appealing, and impactful components, such as widgets, is a foolproof way to increase digital consumer engagement.

Here are a few ideas and advice on how you can implement widgets to boost your engagement:

Alert and notify with a pop-in

An exit-intent pop-in is one popular widget to set up on your page that can have a real impact on retaining your visitors before they leave. By displaying important information that your visitor might have missed, you are giving them another opportunity to benefit from your message.

Le Slip Français, a French clothing manufacturer, displayed a seasonal exit pop-in with free delivery during Valentine’s Day, leading to a 22% increase in clicks.

Leverage geotargeting banners

Having multiple regions requires you to differentiate your communication. With geo-targeted banners, you can maximize your chances of a conversion by sending the right message to the right digital consumer in the right location.

This type of widget can be used in a variety of ways such as creating a customizable banner to offer discounts or services at the closest physical store to the user, which could entice them to go to your location.

Be aware of overuse

It’s important to think of the right time and place for your information. Put yourself in the shoes of your digital audience and ask yourself what information would be enticing and what would be overwhelming.

Creating space with your messages is essential to avoid burdening visitors with too many pop-ins and banners. Even if you’re offering discounts for various reasons, be sure not to go overboard.

Test what works and what doesn’t

Every audience is unique. What works for one company may not work for the next.

This is why A/B testing your new ideas is important to find out what your audience likes. Without testing, a bias based on your personal likes and dislikes can get in the way of data-driven decision-making. Experimentation is the only way to find what works for your customers without making prior assumptions.

Want to start enticing your customers with widgets and leveraging A/B testing on your website? AB Tasty is the best-in-class experience optimization platform that empowers you to create a richer digital experience – fast. From experimentation to adding widgets, this solution can help you activate and engage your audience to boost your conversions.

Benefits of engaging digital consumers with widgets

Providing your visitors with additional opportunities to engage with your website can bring a number of benefits to your business, which include:

Drive conversions

Your visitors have already communicated their interest in your products by visiting your webpage. Now they need to be convinced that your products or service will be worth it to nudge them down the funnel.

Find out what influences your digital consumers’ decision-making with different tactics:

A countdown timer creates excitement for digital consumers. Alerting visitors that there is a timely discount helps create a sense of urgency that will help them make a decision, fast.
Adding social proof to your product pages is a popular scarcity and urgency tactic used by many marketers to help draw attention to certain products. By letting visitors know that their peers are also interested in the same product, they will feel a sense of comfort knowing they aren’t alone.

As an example, NYX cosmetics leveraged AB Tasty’s social proof banner widget on their product pages to create desire. Without any coding knowledge, NYX created this test and saw a 2x increase in their transaction rate compared to the original version.

Increase basket size

Another benefit of engaging users on your website is increasing the average basket size.

By adding widgets that propose complimentary products to your consumer’s purchases, you are actively encouraging them to add more to their basket while highlighting benefits that they may receive.

Let non-subscribers know they are eligible for a discount if they upgrade or give visibility to a free shipping discount by adding a progress bar.

Decathlon, a French sporting goods retailer, implemented a progress bar using AB Tasty’s easy-to-use widgets. By offering this special “gift” of free shipping to customers who filled the progress bar, they saw a 10% increase in the transaction rate.

Build loyalty

Building loyalty is an ever-growing challenge with the growing market saturation. Post-purchase, you should focus your attention on building customer loyalty and creating new brand ambassadors.

One strategy to increase loyalty is by implementing widgets to collect information about their experience in the form of a pop-up survey. An exit pop-in is a great way to capture your customers’ attention while the transaction process is still fresh in their minds.

Unicef, an agency of the United Nations responsible for providing humanitarian and developmental aid to children worldwide, used an iframe widget to collect information about their digital consumer’s experience quickly and saw over 300 answers in a short period of time.

unicef survey

Engaging your digital audience

When you have fewer interactive elements, there’s nothing in place to entice visitors to stay on your website. This means ultimately ending up with more passive customers.

Meanwhile, placing easy-to-use widgets is a great way to grab your customers’ attention and give them the nudge they need to engage more with your website. Getting your digital consumers to make the switch from passive to active will eventually make a positive impact on the KPIs that matter to you most.

If you’re looking to drive conversions, increase your basket size or drive loyalty – widgets may be the perfect addition to your optimization roadmap.

You might also like...

See all

Article

3min read

Experiment Health Check: Proactive Monitoring for Reliable Experimentation

Emily Healy

Jul 1, 2025

Article

9min read

Heatmaps: Your Team’s Secret Weapon for Uncovering Website Gold

AB Tasty

Jun 24, 2025

Article

6min read

9 AI Features that Transform How Digital Teams Test, Learn, and Grow

Emily Healy

Jun 20, 2025

Subscribe to
our Newsletter

Article

May 22, 2023

7min read

A/B Test Hypothesis Definition, Tips and Best Practices

Emily Healy

Incomplete, irrelevant or poorly formulated A/B test hypotheses are at the root of many neutral or negative tests.

Often we imagine that doing A/B tests to improve your e-commerce site’s performance means quickly changing the color of the “add to cart” button will lead to a drastic increase in your conversion rate, for example. However, A/B testing is not always so simple.

Unfortunately, implementing random changes to your pages won’t always significantly improve your results – there should be a reason behind your web experiments.

This brings us to the following question: how do you know which elements to experiment with and how can you create an effective AB test hypothesis?

Determine the problem and the hypothesis

Far too few people question the true origins of the success (or failure) of the changes they put in place to improve their conversion rate.

However, it’s important to know how to determine both the problem and the hypothesis that will allow you to obtain the best results.

Instead of searching for a quick “DIY” solution, it’s often more valuable in the long term to take a step back and do two things:

Identify the real problem – What is the source of your poor performance? Is it a high bounce rate on your order confirmation page, too many single-page sessions, a low-performing checkout CTA or something more complex?
Establish a hypothesis – This could show the root of the problem. For example, a great hypothesis for A/B testing could be: “Our customers do not immediately understand the characteristics of our products when they read the pages on our e-commerce site. Making the information more visible will increase the clicks on the “add-to-cart” button.”

The second step may seem very difficult because it requires a capacity for introspection and a critical look at the existing site. Nevertheless, it’s crucial for anyone who wants to see their KPIs improve drastically.

If you’re feeling a bit uncomfortable with this type of uncertainty around creating an effective hypothesis, know that you’ve come to the right place.

What is an A/B test hypothesis?

Technically speaking, the word hypothesis has a very simple definition:

“A proposal that seeks to provide a plausible explanation of a set of facts and which must be controlled against experience or verified in its consequences.”

The first interesting point to notice in this definition is “the set of facts to be explained.” In A/B testing, a hypothesis must always start with a clearly identified problem.

A/B tests should not be done randomly, or you risk wasting time.

Let’s talk about how to identify the problem:

Web analytics data – While this data does not explain digital consumers’ behavior exactly, it can highlight conversion problems (identifying abandoned carts, for example) and help prioritize the pages in need of testing.
Heuristic evaluation and ergonomic audit – These analyses allow you to assess the site’s user experience at a lower cost using an analysis grid.
User tests – This qualitative data is limited by the sample size but can be very rich in information that would not have been detected with quantitative methods. They often reveal problems understanding the site’s ergonomics. Even if the experience can be painful given the potential for negative remarks, it will allow you to gather qualified data with precise insights.
Eye tracking or heatmaps – These methods provide visibility into how people interact with items within a page – not between pages.
Customer feedback – As well as analyzing feedback, you can implement tools such as customer surveys or live chats to collect more information.

The tactics above will help you highlight the real problems that impact your site’s performance and save you time and money in the long run.

A/B test hypothesis formula

Initially, making an A/B test hypothesis may seem too simple. At the start, you mainly focus on one change and the effect it produces. You should always respect the following format: If I change this, it will cause that effect. For example:

Changing (the element being tested) from ___________ to ___________ will increase/decrease (the defined measurement).

At this stage, this formula is only a theoretical assumption that will need to be proven or disproven, but it will guide you in solving the problem.

An important point, however, is that the impact of the change you want to bring must always be measurable in quantifiable terms (conversion rate, bounce rate, abandonment rate, etc.).

Here are two examples of hypotheses phrased according to the formula explained above and that can apply to e-commerce:

Changing our CTA from “BUY YOUR TICKETS NOW” to “TICKETS ARE SELLING FAST – ONLY 50 LEFT!” will improve our sales on our e-commerce site.
Shortening the sign-up form by deleting optional fields such as phone and mailing address will increase the number of contacts collected.

In addition, when you think about the solution you want to implement, include the psychology of the prospect by asking yourself the following:

What psychological impact could the problem cause in the digital consumer’s mind?

For example, if your problem is a lack of clarity in the registration process which impacts the purchases, then the psychological impact could be that your prospect is confused when reading information.

With this in mind, you can begin to think concretely about the solution to correct this feeling on the client side. In this case, we can imagine that one fix could be including a progress bar that shows the different stages of registration.

Be aware: the psychological aspect should not be included when formulating your test hypothesis.

Once you have gotten the results, you should then be able to say whether it is true or false. Therefore, we can only rely on concrete and tangible assumptions.

Best practice for e-commerce optimization based on A/B hypotheses

There are many testable elements on your website. Looking into these elements and their metrics can help you create an effective test hypothesis.

We are going to give you some concrete examples of common areas to test to inspire you on your optimization journey:

HOMEPAGE

The header/main banner explaining the products/services that your site offers can increase customers’ curiosity and extend their time on the site.
A visible call-to-action appearing upon arrival will increase the chance visitors will click.
A very visible “about” section will build prospects’ trust in the brand when they arrive on the site.

PRODUCT SECTIONS

Filters save customers a lot of time by quickly showing them what they are looking for.
Highlighting a selection of the most popular products at the top of the sections is an excellent starting point for generating sales.
A “find out more” button or link under each product will encourage users to investigate.

PRODUCT PAGES

Product recommendations create a more personal experience for the user and help increase their average shopping cart
A visible “add to cart” button will catch the prospect’s attention and increase the click rate.
An “add to cart and pay” button saves the customer time, as many customers have an average of one transaction at a time.
Adding social sharing buttons is an effective way of turning the product listing into viral content.

Want to start A/B testing elements on your website? AB Tasty is the best-in-class experience optimization platform to help you convert more customers by leveraging intelligent search and recommendations to create a richer digital experience – fast. From experimentation to personalization, this solution can help you achieve the perfect digital experience with ease.

CART PAGE

The presence of logos such as “Visa certified” enhances customer confidence in the site.
A very visible button/link to “proceed to payment” greatly encourages users to click.

PAYMENT

A single page for payment reduces the exit rate.
Paying for an order without registration is very much appreciated by new prospects, who are not necessarily inclined to share their personal information when first visiting the site.
Having visibility over the entire payment process reassures consumers and will nudge them to finalize their purchase.

These best practices allow you to build your A/B test hypotheses by comparing your current site with the suggestions above and see what directly impacts conversion performance.

The goal of creating an A/B test hypothesis

The end goal of creating an A/B test hypothesis is to identify quickly what will help guarantee you the best results. Whether you have a “winning” hypothesis or not, it will still serve as a learning experience.

While defining your hypotheses can seem complex and methodical, it’s one of the most important ways for you to understand your pages’ performance and analyze the potential benefits of change.

You might also like...

See all

Article

3min read

Experiment Health Check: Proactive Monitoring for Reliable Experimentation

Emily Healy

Jul 1, 2025

Article

9min read

Heatmaps: Your Team’s Secret Weapon for Uncovering Website Gold

AB Tasty

Jun 24, 2025

Article

6min read

9 AI Features that Transform How Digital Teams Test, Learn, and Grow

Emily Healy

Jun 20, 2025

Subscribe to
our Newsletter

Article

Mar 20, 2023

8min read

Great Value Proposition Examples

Emily Healy

A value proposition is a clear statement of the fundamentals that define your brand.

To create a value proposition, you need to clearly define your product or service and showcase how you are different from competitors. This is often referred to as a Unique Selling Point (USP).

So, how can you define and present your company’s USP? By crafting a value proposition canvas.

A value proposition canvas is a visual display that maps out your product or service and how it can meet the specific needs of your customers.

It’s essentially an extension of the frequently-used business model canvas and has become a widely popular method of presentation in all corners of the business world.

In this article, we will dive into the specifics of a value proposition, give you tips for creating your own and advice on how to develop it further with a value proposition canvas.

What is a value proposition?

At first glance, a value proposition seems like a very simple and broad concept. However, a particularly effective value proposition is exceptionally complex and loaded with details.

Here are some common questions at the heart of each value proposition:

Why should a customer use your service or buy your product over a competitor’s?
What are you offering that no one else can?
How can you serve the needs of each individual customer?

A value proposition is not a positioning statement designed to appeal to as many customers as possible, rather it’s very specific. To meet the specific needs of a specific customer, you need to be as niche as possible.

For example, a business might be looking for software that comes with customer support packages that are more encompassing than usual due to the fact that they have little experience with a particular product.

Suppose a software company highlights its top-rated customer service team and affordable ongoing support options that their competitors are unable to provide. In that case, it will likely attract businesses that will profit from (and be in need of) this type of support.

Let’s look at another example: Uber – a travel service provider that meets low-cost and on-demand needs. Immediacy, convenience and simplicity are at the heart of their value proposition.

Uber value proposition

Think about the unique benefit that you are providing. If you can define this simply, concisely and accurately, you have a value proposition of worth to your audience.

How to create a value proposition

Now that we’ve defined what a value proposition is, let’s move on to how you can start brainstorming to transform your thoughts and ideas into a clear statement.

Before we get too intense, start by drafting up a few phrases with this template: We help (Customer) to (Goal) by (Unique value proposition).

We should also keep in mind that value propositions don’t need to be subtle. In fact, they can be rather blunt instruments used to draw in new customers or clients.

Value propositions should not be loaded with jargon or complexities. They should be thought of more like a well-researched advertising jingle than a dissertation.

Let’s discuss a few points to keep in mind when developing your value proposition:

Define your target audience
Know your competition
Tell your customers how you meet their needs
Be prepared to combat stereotypes

Define your target audience

As value propositions are highly dependent on those that are being sold to, it’s essential to define your target audience.

This requires a great deal of research, but will undoubtedly pay off in the long run. Age, demographics, marital status, economic status and personal interests can all inform how a value proposition is designed.

Your message needs to be clearly personalized for your audience.

Know the competition

It’s hard to stand out from the crowd if you have little concept of what the crowd is offering.

If a larger competitor offers the same USP (Unique Selling Points), it’s likely that they will get a hold of your potential customers since they will rely on your differences: brand size and reputation.

This requires extensive research to define new and fertile ground. You need to know your competition well – not only to be aware of your similarities but also to let your audience know how you stand out.

Tell your customer how you meet their needs

To start with, it’s best to define exactly what your product or service offers. Then, you can move into what makes it unique:

Customer service?
Affordability?
A simpler design than the competitors?

Once you have defined what you will highlight, it’s time to get more specific with your word choice. Remember: specificity does not mean complexity.

If the customer service is highly rated, why?
Affordability is great, but does the quality of the product match or even improve on your competitor?
Simplicity is admirable, but how does this improve customer satisfaction?

If you can define this in simple, but effective terms, you will have a highly effective value proposition.

Be prepared to combat stereotypes

Every sector has its own negative stereotype that can be harmful even before they get started.

Delivery services are known for not turning up when agreed upon, car salesmen are known for being notoriously pushy and fast food is known for being convenient but unhealthy. These are all examples of initial reactions, or cognitive biases, that most people have before using a product or service.

One of the most powerful kinds of value propositions are those that go against the grain of expectations.

If your particular sector is known for a negative practice, ensuring that your brand is above the stereotype can provide a concise and enticing positioning statement.

Value proposition canvas

A value proposition canvas is a focused way of structuring the main components of the value proposition in order to provide the most concise solutions for potential customers.

Value proposition canvas segments

The value proposition canvas is made up of two segments: the customer segment and the value proposition.

Inside each segment are three sections. The customer segment (shown below in red) has the specific needs of the customer, while the value proposition segment (shown in green) has the corresponding solutions.

value proposition canvas

The three sections of the customer segment (red) in the value proposition canvas are:

Customer jobs – what tasks can your customer accomplish by using your product or service?
Customer pains – what problems, risks or negative experiences can the customer eliminate when using your product or service?
Customer gains – what unique benefit does the customer gain when
using your product or service?

The value proposition map (green) on the other side of the canvas seeks to address these factors while presenting unique solutions and unexpected gains.

Gain creators – this is where you detail the unique improvements and benefits that your product or service offers to your customers
Pain relievers – this is the area to address elements that eliminate a current frustration or pain point (such as undesired costs, accessibility, or negative stereotypes)
Products and services – this is the spot to list all of your products and service

How to create a value proposition canvas

Let’s start with the customer segment.

The first thing to consider when creating a value proposition canvas is the customer. The customer is the central part of the entire premise. Be sure to speak carefully to your specific audience and avoid making generalizations.

Customer pains are generally more straightforward; therefore, pain relievers are often easier to define.

It’s tempting to focus solely on the functional factors, such as how to get from A to B. However, while the function is important, the emotional and social aspects also need deep consideration when developing your value proposition canvas.

Customer gains are a more complex element of the value proposition canvas as they can be slightly more difficult to define. In essence, these cannot be discovered without a substantial understanding of the uniqueness of the business itself. Not only will you list what the customer will gain from your product, but also what the customer will gain by choosing you over the competition.

Now let’s move on to the value proposition portion of the canvas.

Listing the potential jobs, or tasks, that customers want to accomplish with the help of your product or service is a good place to start.

Once completed, you will move on to the pain relievers and gains.

The pain relievers will specifically target the pains listed in the customer segment and should remain consistent. Meanwhile, the gain creators will show where your product or service adds value.

It’s important to remember that gains aren’t the opposite of pains. They are different factors that ultimately encourage consumers to adopt the product or service.

In the gains section, you will address the lesser-known desires of the customers – these are things that the customer themselves may not even be aware of. The lesser-known customer needs can be considered the “bonuses” that your brand offers potential customers.

Let’s go back to Uber as an example. Uber’s customers need quick, accessible and affordable transportation. These are the customer’s first thoughts. In addition, their customers also need safety during their rides. Uber offers insurance during trips to their drivers and passengers to cover them in the case of an accident. Furthermore, they can provide passengers with more comfort by including driver ratings from previous passengers. Customers may not immediately think of these factors, but they address very important human needs.

After development, each part of the right side of the canvas should be balanced with solutions and gains on the left. Ultimately, a value proposition canvas seeks perfect alignment.

Finding your fit in the market

Creating a value proposition canvas is a great tool to help understand your product, where you fit in the market, how you stand against competitors and the best way to market your brand successfully.

Drawing conclusions from this canvas and analyzing your value proposition can help you improve your strategies, messaging and overall product. A unique and profound value proposition is the core of a strong business.

You might also like...

See all

Article

3min read

Experiment Health Check: Proactive Monitoring for Reliable Experimentation

Emily Healy

Jul 1, 2025

Article

9min read

Heatmaps: Your Team’s Secret Weapon for Uncovering Website Gold

AB Tasty

Jun 24, 2025

Article

6min read

9 AI Features that Transform How Digital Teams Test, Learn, and Grow

Emily Healy

Jun 20, 2025

Subscribe to
our Newsletter

Article

Mar 14, 2023

10min read

Digital Customer Journey: Insights and Optimization Tips

Emily Healy

In a highly competitive digital marketplace, optimizing your website for a unique and seamless digital customer journey is no longer just a competitive advantage — it’s a necessity.

It’s important to remember that the digital customer journey does not begin and end with a purchase – it’s a web of all customer interactions and touchpoints with your brand.

AB Tasty has mapped out seven customer phases that we consider crucial in the journey. To craft unique experiences, you’ll need to differentiate these seven phases customers pass through and understand how to animate their digital journey.

Once you have a better understanding of these phases, you will be better equipped to set your business goals and properly optimize your website for growth and impact.

Click to view the full-sized infographic in new tab

How exactly can you optimize each phase of the digital customer journey? Let’s dive right in and take a look at some examples.

Phase 1: Awareness

When visitors land on your website for the first time, a great first impression is crucial.

Your page needs to be both visually appealing and intuitive. A dynamic above the fold design is a great place to start.

In this first phase, it’s important to let your best ideas shine to capture and keep your visitors’ attention. You can accomplish this by creating personalized welcome messages for first-time visitors, displaying your value proposition and organizing high-impact elements for better visibility.

Let’s take a look at Just Over The Top’s experiment to modify the layout of their homepage. They used AB Tasty’s experience optimization platform to test if their users responded better seeing a layout with product categories rather than individual products.

Original:

Individual product display - Just Over The Top

Variation: Product category display - Just Over The Top

After creating a test variation to run against the original layout, they saw a 17.5% click increase on the three blocks below the hero image. This brought many more users into the second phase of their customer journey.

Phase 2: Discovery

When consumers reach the second phase, they’ve already discovered your brand and they’re getting curious.

To accommodate visitors during this phase, your website should be optimized for an excellent browsing experience. Whether this means making your search bar more visible, creating dynamic filters while searching, or using a virtual assistant to get to know your visitors’ interests with a series of questions, an easy browsing experience with intelligent search is key.

In this example, Claudie Pierlot focused on optimizing the customer browsing experience by testing the search bar visibility. In their variation, the small search icon was made more visible by adding the word “recherche” (or search in English) in the top right-hand corner.

Original:

Claudie Pierlot- before

Variation:

Claudie Pierlot - recherche

This clear above the fold design made it easier for visitors to identify the search bar to begin their browsing experience.

With this simple A/B test, they saw a 47% increase in search bar clicks and a 7% increase in conversion rates coming directly from the search bar.

In another example, Villeroy & Boch, a ceramic manufacturing company, wanted to leverage intelligent search on their website. With the help of AB Tasty, they implemented an AI search algorithm to navigate online shoppers.

With our solution, they designed a new and intuitive navigation complete with filters and a comprehensive autosuggestion feature.

By changing their search functions, Villeroy & Boch saw a 33% increase in search results clicks and a 20% increase in sales through the search function.

Phase 3: Consideration

Now is the time when your visitors are considering your brand and which products they are interested in. Showcasing your product pages in their best light during the consideration phase might be exactly what your visitor needs to continue moving down the funnel.

Let’s look at how Hanna Anderson optimized their product pages during this phase.

The clothing retail company wanted to experiment with the images on their product listing pages. Previously, their toddler line had only images of clothing sizes for an older child. They were convinced there was room for improvement and decided to run a test by changing their images to include toddler sizes.

Original:

Hanna Anderson - original Variation:

Hanna Anderson - toddler product images - variation

After implementing age-appropriate clothing images, the results were clear. During this test, the clicks on PLPs increased by almost 8% and the purchase rate on those items skyrocketed by 22%.

Phase 4: Intent

During the intent phase, your visitors are on the verge of becoming customers but need to be convinced to make a purchase.

Social proof, urgency messaging and bundling algorithms are a few ideas to lightly nudge visitors to add to cart or add more to cart.

Let’s take a look at the impact that urgency messaging can have: IZIPIZI, an eyewear retailer, decided to add a special message flag next to their product description to show viewers how many people have purchased this product. The idea of this message is to show viewers that this product is popular and to encourage them to take action.

IZIPIZI - social proof

With this simple sentence of social proof to validate a product’s desirability, they saw a 36% increase in add-to-basket rate.

In another scenario, you can see that adding a progress bar is a simple way to upsell. With a progress par, you are showing your customer how close they are to earning free shipping, which entices them to add more to their cart.

Product bar - Vanessa Bruno

Vanessa Bruno experimented with this additive with the help of AB Tasty and saw a 3.15% increase in transactions and a €6 AOV uplift.

Phase 5: Purchase

Purchase frustration is real. If customers experience friction during checkout, you risk losing money.

Friction refers to any issues the visitors may encounter such as unclear messaging during the payment (did the payment actually go through?), confusing or expensive shipping options, discounts not working, double authentication check-in delays, difficult sign-in and more.

Optimizing your checkout sequence for your audience with rollouts and KPI-triggered rollbacks can help you find a seamless fit for your website.

Let’s look at an example for this phase: Galeries Lafayette, the French luxury department store, saw an opportunity to optimize their checkout by displaying default payment methods that do not require double authentication.

Payment options

During this test, they saw a €113,661 increase in profit, a €5 uplift in average order value, and a 38% increase in the conversion rate by adding the CB (bank card) option for a quicker checkout.

Phase 6: Experience

Optimizing the buyer experience doesn’t end after the purchase. Now is the time to grow your customer base and stop churn in its tracks. So, how do you keep your customers interested? By maintaining the same level of quality in your messages and personalization.

Let’s look at how Envie de Fraise, a French boutique, leveraged their user information to transform a normal post-purchase encounter into a personalized experience.

One of their customers had just purchased a maternity dress and had been browsing multiple maternity dresses prior to their purchase. By knowing this information, they experimented with using the “you will love these products” algorithm to gently nudge their customer to continue shopping.

products you will love algorithim

With a customized recommendation like this, Envie de Fraise saw a €127K increase in their potential profit.

As your customer spends more time with your brand, you will learn more about their habits and interests. The more time they spend with you, the more personalized you can make their experience.

Phase 7: Loyalty

In the final step of your customer’s journey, they move into the loyalty phase. To turn customers into champions of your brand, it’s important to remind them that you value their loyalty.

This can be done by sending emails with individual offers, social proof, product suggestions or incentives for joining a loyalty program to earn rewards or complete product reviews.

Another example of this is sending a personalized email displaying items that are frequently bought together that align with their purchase. This will remind the customer about your brand and give them recommendations for future purchases.

Why Optimizing the Digital Customer Journey is Essential to Boost Conversions

The fierce competition in the e-commerce marketplace is undeniable. In order to attract and retain customers, you have to focus on crafting personalized user experiences to turn passive visitors into active buyers.

Understanding their needs in each phase and optimizing your digital space is your best solution to nudge visitors down the purchasing funnel.

By personalizing the experience of your customers during each phase of the digital customer journey, you can ensure an optimal shopping experience, boost purchases, increase customer satisfaction and see more repeat customers.

You might also like...

See all

Article

3min read

Experiment Health Check: Proactive Monitoring for Reliable Experimentation

Emily Healy

Jul 1, 2025

Article

9min read

Heatmaps: Your Team’s Secret Weapon for Uncovering Website Gold

AB Tasty

Jun 24, 2025

Article

6min read

9 AI Features that Transform How Digital Teams Test, Learn, and Grow

Emily Healy

Jun 20, 2025

Subscribe to
our Newsletter

Article

Jan 25, 2023

12min read

Understanding Bounce Rate to Improve It

Emily Healy

Bounce rate is one of the most important metrics used to understand how well your website is performing. It’s a type of web analytics metric that measures the behavior of visitors to a website or page within the website.

In this article, we will cover everything about bounce rates: how they are calculated, bounce rates vs exit rates, what is considered a good bounce rate, how to use web analytics for tracking and tips for improving this metric.

Let’s get started.

What is bounce rate on a website?

The definition of bounce rate is relatively easy to understand compared to other web analytics concepts. However, that is not to say that the data is somehow superficial or unimportant, as there are many insights that a bounce rate analysis can provide.

The bounce rate yields information on the behavior of a website’s visitors and how well the website is engaging them.

To “bounce” from a website simply means to leave before interacting with the page in some way such as leaving a comment, clicking on something, scrolling or visiting another page on the site.

In other words, to enter and to leave without engaging beyond the initial entrance of the website is a bounce. However, a bounce is not always a bad thing or a sign that the website is not performing well.

Each visitor to a website can be seen as a drop of water with the website presenting a surface that is either porous or waterproof. The goal of a website is to be as porous as possible, absorbing each visitor to the site with as relevant and interesting information as possible. Non-porous websites “bounce” visitors immediately, often indicating that the website is not performing as required and flagging issues that can be addressed with further web analytics.

How to calculate bounce rate

A bounce rate analysis is a very straightforward formula that can be summed up in a simple equation.

The number of visitors who leave a website after only visiting the landing page (the page that led them to the website) and not interacting in any way, is divided by the total number of visitors to the site.

Bounce Rate Formula

For example, if 40 visitors leave without being “absorbed” into further interaction with a site and there have been 100 visitors overall, the bounce rate will be 40%.

There are a few main ways that a visitor may bounce. For example:

Clicking on a link to another website
Clicking the back arrow that takes them to the previous page
Entering a new URL and hitting enter
Closing the browser or tab

One other way in which a visitor may bounce is if they stop interacting entirely, causing the session to time out.

Anything over half an hour of idle time is considered a bounce. Any further interaction after this time, even if it occurs within the site, is considered a new session.

Bounce rate vs exit rate

Bounce rate and exit rate are often thought to be synonyms, or at the very least that they ultimately provide the same data.

This is a huge error and can lead to false bounce rate analysis and poor decision-making.

The confusion is understandable, as both concepts seemingly measure similar things on the surface. However, the difference is quite profound and requires an understanding of what both metrics measure.

As previously described, bounce rate measures the percentage of visitors that leave a website before interacting with it in any way.

Exit rates provide information on specific pages of the website, measuring the percentage of visitors that left the site after viewing a specific page, no matter how many pages they have visited before in the session.

In other words, all bounces are exits, but not all exits are bounces.

Bounce rates are based on only sessions that start and end with one page. While exit rates calculate the last page visited in the user’s journey, regardless of the number of pages a user has visited during one session.

Bounce Rate vs Exit Rate

For example, let’s say a site has three pages named Page X, Page Y and Page Z. From Monday to Friday, the interaction might look something like this:

Monday – Page Y > Exit
Tuesday – Page Y >Page Z > Page X >Exit
Wednesday – Page X > Page Z > Page Y >Exit
Thursday – Page Z >Exit
Friday – Page Y > Page X > Page Z > Exit

The analysis would show:

Page X has an exit rate of 33% and a bounce rate of 0%.

Three sessions included Page X and one session exited from Page X.
There was no single-page session for Page X.
One session began on Page X, but it was not a single-page session, so the bounce rate is zero.
No visitor entered and left without any other interaction on this page.

Page Y has an exit rate of 50% and a bounce rate of 33%.

Four sessions included Page Y.
Two of those four total sessions exited directly from Page Y, so the exit rate is 50%.
The bounce rate is less than the exit rate because three sessions started with Page Y and only the single-page session led to a bounce.

Page Z has an exit rate of 50% and a bounce rate of 100%.

Just like Page Y, the exit rate of Page Z is 50% because four sessions included Page Z, and two sessions exited from Page Z.
The bounce rate is 100% because the only session that started with Page Z was a single-page session that led to a bounce.

High exit rates of some pages can be a good sign. For example, within e-commerce, leaving a page after completing a purchase is a good sign as it points towards a satisfactory outcome of a transaction.

Define average bounce rates

Defining average bounce rates can be a complex task. This is largely due to the fact that a high bounce rate for some pages, might be considered low for others.

Each case can be highly subjective due to the industry and purpose of the webpage. Certainly, bounce rate can only provide some information about a website’s performance and it remains highly important to use other metrics to fully understand visitor behavior.

Industry-standard bounce rates consider anything over 50% as high and anything between 20% and 50% as low, but it’s essential to go beyond this broad definition of what is an acceptable bounce rate.

Anything under 20% is likely an error and should be looked into.

What is a healthy bounce rate?

Rather than being concerned with an all-encompassing good bounce rate, a better goal would be finding a healthy bounce rate for each specific site and its unique goals.

The goal of 20% to 50% is not without merit, but it can be a superficial reading of behavior or almost impossible to attain depending on your type of page.

For an e-commerce landing page, browsing is often encouraged because it can lead to sales later on. Lower bounce rates will indicate that visitors decided to continue to look around rather than leave.

For other sites, like online recipe sites or information-specific sites, browsing is not likely. Therefore, a high bounce rate on these types of pages can be an indication that the visitor is satisfied having found the information they needed and bounced after.

Some websites are designed to be only one page, so it’s impossible to know what a “bad” bounce rate is when the design itself ensures it remains at 100%. In these circumstances, and those where a lot of information is presented on the landing page, other metrics can provide more relevant insights.

For example, it is always a bad sign if the visitor leaves within a matter of seconds. However, if they remain on the page for multiple minutes, the bounce rate might not be the best indicator that the site is not performing as desired.

To determine a high or low bounce rate, it’s important to consider the site’s purpose, the average bounce rate in your sector and typical user behavior. From there it will be easier to gain the appropriate insight that the data is providing.

Want to lower bounce rates by testing different aspects of your website? AB Tasty’s best-in-class customer experience optimization platform offers you an A/B testing tool that allows you to quickly set up tests with low code implementation of front-end or UX changes on your web pages. Go further by gathering insights via an ROI dashboard, and determining which route will attract customers and ultimately increase your revenue.

How to check the bounce rate of your website

Once you have defined what you consider to be your desired website bounce rate, the next step is to understand where to find the data.

Thankfully this data is very easy to find and comprehend in Google Analytics, which also provides a multitude of other data to help understand how well your website is performing.

Not being particularly familiar with Google Analytics should not deter you from using it. It is fairly self-explanatory once you have a grasp of the terms and purpose of the data.

Where to find the bounce rate on Google Analytics?

First, sign into the website’s Google Analytics page. Once there, select the “Audience Overview” tab, which also provides a variety of other metrics that can be of great use. From here you will need to choose whether you want to read the entire website’s or individual pages’ bounce rate.

For the entire website, simply click on the bounce rate metric, which will also provide a graph for the defined time period. The time period can be changed as required by the calendar at the top right of the screen and includes a customizable option.

For individual pages, click “Behavior” followed by “All Pages”, which will then provide a list of your website’s pages and their specific bounce rate. Much like understanding the website’s unique bounce rate context, this is also the case for individual pages.

Do all analytic software measure bounce rates the same way?

Google Analytics measures a bounce as a single-page session that has a duration of 0 seconds. Clearly, a visitor will not make any other actions during a 0-second time frame.

If you’re using another analytics software to do your website tracking, keep in mind that they may score it slightly differently.

Some analytics software do not have a time frame that they use to count a bounce. Rather, they rely strictly on the interaction of the visitor during their session. Both ways will calculate a bounce rate for your website, but it’s important to know the specifics.

The key takeaway here is not to compare results from one analytic software to another. Consistency is key to tracking your performance.

How to reduce website bounce rate

Once you have an understanding of the concept of bounce rate, the next question is “Why is my bounce rate so high?” And more importantly, “how do I lower the bounce rate?”

There are many methods that can help reduce your bounce rate, including simpler things like seeing which page is performing best and implementing some of its elements on other pages that have higher bounce rates.

Some other ideas will become self-explanatory, while others might require a little trial and error.

Here are a few ideas to try out:

Improve content

One of the simplest ways to decrease the bounce rate of a page or the entire site is to improve the content itself.

In some ways this should be self-explanatory: the more interesting and higher quality the content, the more likely the reader will stick around to explore what else the site has to offer.

Relevancy is also key. If your website is primarily about camping, publishing unrelated content, such as politics, is likely to be met with disinterest and repel visitors instantly. This will undoubtedly lead to a high bounce rate.

Content requires some planning and forethought if a site is going to “absorb” visitors.

Readability

Content isn’t just related to interesting posts or entertaining media, readability is also key.

Text-heavy websites might be full of amazing information, but unless the page is formatted to draw the reader in, the first impression might be a bit overwhelming.

Be sure to use some mixture of imagery, bullet points, subheadings, and bolded words. This creates a balance to attract and retain visitors.

Avoid the use of excessive pop-ups

Nothing cheapens the feel of a site like the dreaded pop-up, and this isn’t just the case for ads.

Pop-ups are distracting, appear pushy and ruin the flow of the experience for a user. There is nothing quite as infuriating as having to click away one or several flashing online forms while trying to read something else.

Pop-ups do have their place, and when used at a minimum. They can help grow a subscriber list with a degree of effectiveness. If you like the idea of pop-ups but want to try something different, there are more effective measures such as email campaigns or personalized banners. By implementing one of these campaigns, visitors have the option to interact with an element after they’ve had a chance to engage with the content.

Targeted keywords

Lowering the bounce rate will always require a mixture of tactics, but one of the most reliable ways to do it is by targeting keywords.

The key to using keywords effectively is relevancy, so it is not just a matter of throwing as many keywords at the problem as possible. There is nothing more likely to lead to a high bounce rate than keywords that draw in visitors that are not interested in a website’s content.

The best place to begin is the use of high-value traffic/low-competition keywords. But, how do you find out what these are for your site? Well, it isn’t as complex as you might think. Google Keyword Planner is the perfect tool to implement targeted keywords for your site, and it’s free to use for Google account holders.

Lowering bounce rate with meta descriptions

Meta descriptions are the information that appears under the title of a website in a Google search. They add some depth to the description of a website presence on a search page and are therefore essential in garnering relevant traffic.

Meta descriptions do not rank by keyword, but they filter out traffic that will likely bounce, while also drawing in those more likely to interact. Remember that meta descriptions only show up to 155 characters. Target them to get the right information across to users who aren’t as likely to bounce.

Understanding bounce rates

As you can see, bounce rates can provide a lot of insight into your website’s performance.

Although there is no magic number to indicate a “good” or “bad” webpage bounce rate performance, you can determine your own baseline metrics based on each page’s purpose and past performance.

You might also like...

See all

Article

3min read

Experiment Health Check: Proactive Monitoring for Reliable Experimentation

Emily Healy

Jul 1, 2025

Article

9min read

Heatmaps: Your Team’s Secret Weapon for Uncovering Website Gold

AB Tasty

Jun 24, 2025

Article

6min read

9 AI Features that Transform How Digital Teams Test, Learn, and Grow

Emily Healy

Jun 20, 2025

Subscribe to
our Newsletter

Article

Nov 28, 2022

7min read

Sample Ratio Mismatch: What Is It and How Does It Happen?

Emily Healy

A/B testing can bring out a few types of experimental flaws.

Yes, you read that right – A/B testing is important for your business, but only if you have trustworthy results. To get reliable results, you must be on the lookout for errors that might occur while testing.

Sample ratio mismatch (SRM) is a term that is thrown around in the A/B testing world. It’s essential to understand its importance during experimentation.

In this article, we will break down the meaning of sample ratio mismatch, how to spot SRM, when it is and is not a problem, why it can happen and how to detect SRM.

Sample ratio mismatch overview

Sample ratio mismatch is an experimental flaw where the expected traffic allocation doesn’t fit with the observed visitor number for each testing variation.

In other words, an SRM is evidence that something went wrong.

Sample ratio mismatch is crucial to be aware of in A/B testing.

Now that you have the basic idea, let’s break this concept down piece by piece.

What is a “sample”?

The “sample” portion of SRM refers to the traffic allocation.

Traffic allocation refers to how the traffic is split toward each test variation. Typically, the traffic will be split equally (50/50) during an A/B test. Half of the traffic will be shown the new variation and the other half will go toward the control version.

This is how an equal traffic allocation will look for a basic A/B test with only one variant:

A/b testing equal traffic allocation

If your test has two variants or even three variants, the traffic will still be allocated equally to each test to ensure that each version receives the same amount of traffic. An equal traffic allocation in an A/B/C test will be split into 33/33/33.

For both A/B and A/B/C tests, traffic can be manipulated in different ways such as 60/40, 30/70, 20/30/50, etc. Although this is possible, it is not a recommended practice to get accurate and trustworthy results from your experiment.

Even by following this best practice guideline, equally allocated traffic will not eliminate the chance of an SRM. This type of mismatch is something that can still occur and must be calculated no matter the circumstances of the test.

Define sample ratio mismatch (SRM)

Now that we have a clear picture of what the “sample” is, we can build a better understanding of what SRM means:

SRM happens when the ratio of the sample does not match the desired 50/50 (or even 33/33/33) traffic allocation
SRM occurs when the observed traffic allocation to each variant does not match the allocation chosen for the test
The control version and variation receive undesired mismatched samples

Whichever words you choose to describe SRM, we can now understand our original definition with more confidence:

“Sample ratio mismatch is an experimental flaw where the expected traffic allocation doesn’t fit with the observed visitor number for each testing variation.”

sample ratio mismatch

Is SRM always a problem?

To put it simply, SRM occurs when one test version receives a noticeably different amount of visitors than what was originally expected.

Imagine that you have set up a classic A/B test: Two variations with 50/50 traffic allocation. You notice at one point that version A receives 10,000 visitors and version B receives 10,500 visitors.

Is this truly a problem? What exactly happened in this scenario?

The problem is that while conducting an A/B test, an extremely strict respect of the allocation scheme is not always 100% possible since it must be random. The small difference in traffic that is noted in the example above is something we would typically refer to as a “non-problem.”

If you are seeing a similar traffic allocation on your A/B test in the final stages, there is no need to panic.

A randomly generated traffic split has no way of knowing exactly how many visitors will stumble upon the A/B test during the given time frame of the test. This is why toward the end of the test duration, there may be a smaller difference in the traffic allocation while the majority (+95%) of traffic is correctly allocated.

When is SRM a problem?

Some tests may have SRM due to experimental setup.

When the SRM is a big problem, there will be a noticeable difference in traffic allocation.

If you see 1,000 directed to one variant and 200 directed to the other — this is an issue. Sometimes, spotting SRM does not require a particular mathematical formula dedicated to calculating SRM as it is evident enough on its own.

However, an extreme difference in traffic allocation can be very rare. Therefore, it’s essential to check the visitor counts in an SRM test before each test analysis.

Does SRM occur frequently?

Sample ratio mismatch can happen more often than we think. According to a study done by Microsoft & Booking, about 6% of experiments experience this problem.

Furthermore, if the test includes a redirect to an entirely new page, SRM can be even more likely.

Since we heavily rely on tests and trust their conclusions to make strategic business decisions, it’s important that you are able to detect SRM as early as possible when it happens during your A/B test.

Can SRM still affect tests using Bayesian?

The reality is that everyone needs to be on the lookout for SRM, no matter what type of statistical test they are running. This includes experiments using the Bayesian method.

There are no exemptions to the possibility of experiencing a statistically significant mismatch between the observed and expected results of a test. No matter the test, if the expected assumptions are not met, the results will be unreliable.

Sample ratio mismatch: why it happens

Sample ratio mismatch can happen due to a variety of different root causes. Here we will discuss three common examples that cause SRM.

One common example is when the redirection to one variant isn’t working properly for poorly connected visitors.

Another classic example is when the direct link to one variant is spread on social media, which brings all users who click on the link directly to one of the variants. This error does not allow the traffic to be properly distributed among the variants.

In a more complex case, it’s also possible that a test including JS code is crashing a variant and therefore some of the visitor configurations. In this situation, some visitors that are being sent to the crashing variant won’t be collected and indexed properly, which leads to SRM.

All of these examples have a selection bias: some non-random visitors are excluded. The non-random visitors are arriving directly from a link shared on social media, have a poor connection, or are visiting a crashing variant.

In any case, when these issues occur, the SRM is an indication that something went wrong and you cannot trust the numbers and the test conclusion.

Checking for SRM in your A/B tests

Something important to be aware of when doing an SRM check is that the priority metric when checking needs to be “users” and not “visitors.” Users are the specific people that are allocated to each variation, meanwhile, the visitors metric is counting the number of sessions that each user makes.

It’s important to differentiate between users and visitors because results may be skewed if a visitor comes back to their variation multiple times. SRM detected with “visitors” may not be the most reliable metric, but using the “users” metric is evidence of a problem.

SRM in A/B testing

Testing for sample ratio mismatch may seem a bit complicated or unnecessary at first glance. In reality, it’s quite the opposite.

Understanding what SRM is, why it happens, and how it can affect your results is crucial in A/B testing. Running an A/B test to help make key decisions is only helpful for your business if you have reliable data from those tests.

Want to get started on A/B testing for your website? AB Tasty is a great example of an A/B testing tool that allows you to quickly set up tests with low code implementation of front-end or UX changes on your web pages, gather insights via an ROI dashboard, and determine which route will increase your revenue.

You might also like...

See all

Article

3min read

Experiment Health Check: Proactive Monitoring for Reliable Experimentation

Emily Healy

Jul 1, 2025

Article

9min read

Heatmaps: Your Team’s Secret Weapon for Uncovering Website Gold

AB Tasty

Jun 24, 2025

Article

6min read

9 AI Features that Transform How Digital Teams Test, Learn, and Grow

Emily Healy

Jun 20, 2025

Subscribe to
our Newsletter

Article

Oct 4, 2022

7min read

The ROI of Experimentation

Emily Healy

When you hear ‘A/B Testing’, do you think straight away of revenue gain? Uplift? A dollars and cents outcome?

According to David Mannheim, CEO of the Conversion Rate Optimization (CRO) agency User Conversion, you probably do – and shouldn’t. Here’s why:

Unfortunately, it’s just not that simple.

Experimentation is more than just a quick strategy to uplift your ROI.

In this article we will discuss why we experiment, the challenges of assessing return on investment (ROI), prioritization, and what A/B testing experimentation is really about.

Why do we experiment?

Technically speaking, experimentation is performed to support or reject a hypothesis. Experimentation provides you with valuable insights into cause-and-effect relationships by determining the outcome of a certain test when different factors are manipulated in a controlled setting.

In other words, if there is no experiment, there is no way to refute a hypothesis and reduce the risk of losing business or negatively impacting metrics.

Experimentation is about prioritization, minimizing risk and learning from the outcome. The tests you choose to implement should be developed accordingly. It’s not necessarily about making the “right” or “wrong” decision, experimentation helps you make better decisions based on data.

In visual terms, experimentation will look something like this:

ROI frustration backlog

Online experiments in the business world must be carefully designed to learn, accomplish a specific purpose, and/or measure a key performance indicator that may not have an immediate financial effect.

However, far too often it’s the key stakeholders (or HIPPOs) who decide what tests get implemented first. Their primary concern? The amount of time it will take to see a neat revenue uplift.

This tendency leads us to the following theory:

The ROI of experimentation is impossible to achieve because the industry is conditioned to think that A/B testing is only about gain.

Frustrations and challenges of ROI expectations

You may be asking yourself at this point, What’s so bad about expecting revenue uplift from A/B tests? Isn’t it normal to expect a clear ROI?

It is normal, however, the issue isn’t just that simple.

We’ve been conditioned to expect a neat formula with a clean-cut solution: “We invested X, we need to get Y.”

This is a misleading CRO myth that gets in the way.

Stakeholders have come to erroneously believe that every test they run should function like this – which has set unrealistic ROI expectations for conversion optimization practitioners.

As you can imagine, this way of thinking creates frustration for those implementing online experimentation tests.

Experiment backlog example

What people often overlook is the complexity of the context in which they are running their experimentation tests and assessing their ROI.

It’s not always possible to accurately measure everything online, which makes putting an exact number on it next to impossible.

Although identifying the impact of experiments can be quite a challenge due to the complexity of the context, there are some online tools that exist to measure your ROI efforts as accurately as possible.

AB Tasty is an example of an A/B testing tool that allows you to quickly set up tests with low-code implementation of front-end or UX changes on your web pages, gather insights via an ROI dashboard, and determine which route will increase your revenue.

Aside from the frustration that arises from the ingrained ROI expectation to be focused on immediate financial improvement, three of the biggest challenges of the ROI of experimentation are forecasting, working with averages, and multiple tests at once.

Challenge #1: Forecasting

The first challenge with assessing the ROI of experimentation is forecasting. A huge range of factors impacts an analyst’s ability to accurately project revenue uplift from any given test, such as:

Paid traffic strategy
Online and offline marketing
Newsletters
Offers
Bugs
Device traffic evolution
Season
What your competitors are doing
Societal factors (Brexit)

In terms of estimating revenue projection for the following year from a single experiment– it’s impossible to predict an exact figure. It’s only possible to forecast an ROI trend or an expected average.

Expecting a perfectly accurate and precise prediction for each experiment you run just isn’t realistic – the context of each online experimentation test is too complex.

Challenge #2: Working with averages

The next challenge is that your CRO team is working with averages – in fact, the averages of averages.

Let’s say you’ve run an excellent website experiment on a specific audience segment – and you experienced a high uplift in conversion rate.

If you then take a look at your global conversion rate for your entire site, there’s a very good chance that this uplift will be swallowed up in the average data.

Your revenue wave will have shrunk to an undetectable ripple. And this is a big issue when trying to assess overall conversion rate or revenue uplift – there are just too many external factors to get an accurate picture.

With averages, the bottom line is that you’re shifting an average. Averages make it very difficult to get a clear understanding.

On average, an average customer, exposed to an average A/B test will perform… averagely.

Challenge #3: Multiple tests

The third challenge of ROI expectations happens when you want to run multiple online experiments at one time and try to aggregate the results.

Again, it’s tempting to run simple math equations to get a clear-cut answer for your gain, but the reality is more complicated than this.

Grouping together multiple experiments and the results of each experiment will provide you will blurred results.

This makes ROI calculations for experimentation a nightmare for those simultaneously running tests. Keeping experiments and their respective results separate is the best practice when running multiple tests.

Should it always be “revenue first”?

Is “revenue first” the best mentality? When you step back and think about it, it doesn’t make sense for conversion optimizers to expect revenue gain, and only revenue gain, to be the primary indicator of success driving their entire experimentation program.

What would happen if all businesses always put revenue first?

That would mean no free returns for an e-commerce site (returns don’t increase gain!), no free sweets in the delivery packaging (think ASOS), the most inexpensive product photographs on the site, and so on.

If you were to put immediate revenue gain first – as stakeholders so often want to do in an experimentation context – the implications are even more unsavory.

Let’s take a look at some examples: you would offer the skimpiest customer service to cut costs, push ‘buy now!’ offers unendingly, discount everything, and forget any kind of brand loyalty initiatives. Need we go on?

In short, focusing too heavily on immediate, clearly measurable revenue gain inevitably cannibalizes the customer experience. And this, in turn, will diminish your revenue in the long run.

What should A/B testing be about?

One big thing experimenters can do is work with binomial metrics.

Avoid the fuzziness and much of the complexity by running tests that aim to give you a yes/no, black or white answer.

binomial metrics examples

Likewise, be extremely clear and deliberate with your hypothesis, and be savvy with your secondary metrics: Use experimentation to avoid loss, minimize risk, and so on.

But perhaps the best thing you can do is modify your expectations.

Instead of saying, experimentation should unfailingly lead to a clear revenue gain, each and every time, you might want to start saying, experimentation will allow us to make better decisions.

Good experimentation model

These better decisions – combined with all of the other efforts the company is making – will move your business in a better direction, one that includes revenue gain.

The ROI of experimentation theory

With this in mind, we can slightly modify the original theory of the ROI of experimentation:

The ROI of experimentation is difficult to achieve and should be contextualized for different stakeholders and businesses. We should not move completely away from a dollar sign way of thinking, but we should deprioritize it. “Revenue first” is not the best mentality in all cases- especially in situations as complex as calculating the ROI of experiments.