Tag: EN

Article

May 22, 2023

7min read

A/B Test Hypothesis Definition, Tips and Best Practices

Emily Healy

Incomplete, irrelevant or poorly formulated A/B test hypotheses are at the root of many neutral or negative tests.

Often we imagine that doing A/B tests to improve your e-commerce site’s performance means quickly changing the color of the “add to cart” button will lead to a drastic increase in your conversion rate, for example. However, A/B testing is not always so simple.

Unfortunately, implementing random changes to your pages won’t always significantly improve your results – there should be a reason behind your web experiments.

This brings us to the following question: how do you know which elements to experiment with and how can you create an effective AB test hypothesis?

Determine the problem and the hypothesis

Far too few people question the true origins of the success (or failure) of the changes they put in place to improve their conversion rate.

However, it’s important to know how to determine both the problem and the hypothesis that will allow you to obtain the best results.

Instead of searching for a quick “DIY” solution, it’s often more valuable in the long term to take a step back and do two things:

Identify the real problem – What is the source of your poor performance? Is it a high bounce rate on your order confirmation page, too many single-page sessions, a low-performing checkout CTA or something more complex?
Establish a hypothesis – This could show the root of the problem. For example, a great hypothesis for A/B testing could be: “Our customers do not immediately understand the characteristics of our products when they read the pages on our e-commerce site. Making the information more visible will increase the clicks on the “add-to-cart” button.”

The second step may seem very difficult because it requires a capacity for introspection and a critical look at the existing site. Nevertheless, it’s crucial for anyone who wants to see their KPIs improve drastically.

If you’re feeling a bit uncomfortable with this type of uncertainty around creating an effective hypothesis, know that you’ve come to the right place.

What is an A/B test hypothesis?

Technically speaking, the word hypothesis has a very simple definition:

“A proposal that seeks to provide a plausible explanation of a set of facts and which must be controlled against experience or verified in its consequences.”

The first interesting point to notice in this definition is “the set of facts to be explained.” In A/B testing, a hypothesis must always start with a clearly identified problem.

A/B tests should not be done randomly, or you risk wasting time.

Let’s talk about how to identify the problem:

Web analytics data – While this data does not explain digital consumers’ behavior exactly, it can highlight conversion problems (identifying abandoned carts, for example) and help prioritize the pages in need of testing.
Heuristic evaluation and ergonomic audit – These analyses allow you to assess the site’s user experience at a lower cost using an analysis grid.
User tests – This qualitative data is limited by the sample size but can be very rich in information that would not have been detected with quantitative methods. They often reveal problems understanding the site’s ergonomics. Even if the experience can be painful given the potential for negative remarks, it will allow you to gather qualified data with precise insights.
Eye tracking or heatmaps – These methods provide visibility into how people interact with items within a page – not between pages.
Customer feedback – As well as analyzing feedback, you can implement tools such as customer surveys or live chats to collect more information.

The tactics above will help you highlight the real problems that impact your site’s performance and save you time and money in the long run.

A/B test hypothesis formula

Initially, making an A/B test hypothesis may seem too simple. At the start, you mainly focus on one change and the effect it produces. You should always respect the following format: If I change this, it will cause that effect. For example:

Changing (the element being tested) from ___________ to ___________ will increase/decrease (the defined measurement).

At this stage, this formula is only a theoretical assumption that will need to be proven or disproven, but it will guide you in solving the problem.

An important point, however, is that the impact of the change you want to bring must always be measurable in quantifiable terms (conversion rate, bounce rate, abandonment rate, etc.).

Here are two examples of hypotheses phrased according to the formula explained above and that can apply to e-commerce:

Changing our CTA from “BUY YOUR TICKETS NOW” to “TICKETS ARE SELLING FAST – ONLY 50 LEFT!” will improve our sales on our e-commerce site.
Shortening the sign-up form by deleting optional fields such as phone and mailing address will increase the number of contacts collected.

In addition, when you think about the solution you want to implement, include the psychology of the prospect by asking yourself the following:

What psychological impact could the problem cause in the digital consumer’s mind?

For example, if your problem is a lack of clarity in the registration process which impacts the purchases, then the psychological impact could be that your prospect is confused when reading information.

With this in mind, you can begin to think concretely about the solution to correct this feeling on the client side. In this case, we can imagine that one fix could be including a progress bar that shows the different stages of registration.

Be aware: the psychological aspect should not be included when formulating your test hypothesis.

Once you have gotten the results, you should then be able to say whether it is true or false. Therefore, we can only rely on concrete and tangible assumptions.

Best practice for e-commerce optimization based on A/B hypotheses

There are many testable elements on your website. Looking into these elements and their metrics can help you create an effective test hypothesis.

We are going to give you some concrete examples of common areas to test to inspire you on your optimization journey:

HOMEPAGE

The header/main banner explaining the products/services that your site offers can increase customers’ curiosity and extend their time on the site.
A visible call-to-action appearing upon arrival will increase the chance visitors will click.
A very visible “about” section will build prospects’ trust in the brand when they arrive on the site.

PRODUCT SECTIONS

Filters save customers a lot of time by quickly showing them what they are looking for.
Highlighting a selection of the most popular products at the top of the sections is an excellent starting point for generating sales.
A “find out more” button or link under each product will encourage users to investigate.

PRODUCT PAGES

Product recommendations create a more personal experience for the user and help increase their average shopping cart
A visible “add to cart” button will catch the prospect’s attention and increase the click rate.
An “add to cart and pay” button saves the customer time, as many customers have an average of one transaction at a time.
Adding social sharing buttons is an effective way of turning the product listing into viral content.

Want to start A/B testing elements on your website? AB Tasty is the best-in-class experience optimization platform to help you convert more customers by leveraging intelligent search and recommendations to create a richer digital experience – fast. From experimentation to personalization, this solution can help you achieve the perfect digital experience with ease.

CART PAGE

The presence of logos such as “Visa certified” enhances customer confidence in the site.
A very visible button/link to “proceed to payment” greatly encourages users to click.

PAYMENT

A single page for payment reduces the exit rate.
Paying for an order without registration is very much appreciated by new prospects, who are not necessarily inclined to share their personal information when first visiting the site.
Having visibility over the entire payment process reassures consumers and will nudge them to finalize their purchase.

These best practices allow you to build your A/B test hypotheses by comparing your current site with the suggestions above and see what directly impacts conversion performance.

The goal of creating an A/B test hypothesis

The end goal of creating an A/B test hypothesis is to identify quickly what will help guarantee you the best results. Whether you have a “winning” hypothesis or not, it will still serve as a learning experience.

While defining your hypotheses can seem complex and methodical, it’s one of the most important ways for you to understand your pages’ performance and analyze the potential benefits of change.

You might also like...

See all

Article

6min read

Which Statistical Model is Best for A/B Testing: Bayesian, Frequentist, CUPED, or Sequential?

Emily Healy

Jul 15, 2025

Article

7min read

Is Your Average Order Value (AOV) Misleading You?

Hubert Wassner

Jul 11, 2025

Article

5min read

Why AB Tasty Delivers 4x Faster

Leo Wiel

Jul 7, 2025

Subscribe to
our Newsletter

Article

May 16, 2023

6min read

How to Leverage Disruption in Experimentation | Ben Labay

Rowan Haddad

Ben Labay outlines essential frameworks for a more strategic, tactical and disruptive approach to experimentation

With two degrees, in Evolutionary Behavior and Conservation Research Science, Ben Labay spent a decade in academia with a wide-ranging background in research and experimentation dealing with technical data work.

Now as CEO of experimentation and conversion optimization agency Speero, Ben describes his work in experimentation as his “geek-out” area which is customer experience research and dealing with customer data.

At Speero, Ben works to scope and run research and test program strategies for companies including Procter & Gamble, ADP, Codecademy, MongoDB, Toast and many others around the world.

AB Tasty’s VP Marketing Marylin Montoya spoke with Ben on how to create mechanisms for companies to not only optimize but also be more disruptive when it comes to web experimentation to drive growth.

Here are some of the key takeaways from their conversation.

Consider a portfolio way of management in experimentation

Inspired by Jim Collins’ and Jerry I. Porras’ book “Built to Last”, Ben discusses a framework that the book provides on the ways a company can grow based on the best practices from 18 successful companies.

He identifies one big pillar that many organizations are often neglecting: experimentation. To tackle this, Ben suggests taking a portfolio management way of doing experimentation made up of three portfolio tags which provide a solution spectrum around iterative changes for optimization.

The first level consists of making small tweaks or changes to a website based on customer feedback such as improving layouts and the second which includes more substantial types of changes such as new content pieces.

But there’s a bigger third level which Ben refers to as more “disruptive” and “innovative” such as a brand new product or pricing model that can serve as a massive learning experience.

With three different levels of change, it’s important to set a clear distribution of time spent on each level and have alignment among your team.

In the words of Ben, “Let’s put 20% of our calories over into iterating, 20% onto substantial and 20/30/ or 40% over on disruptive. And that map – that framework has been really healthy to use as a tool to get teams on the same page.”

For Ben, applying such a framework is key to getting all teams on the same page as it helps ensure companies are not under-resourcing disruptive and “big needle movers”. Velocity of work is important, he argues, but so is quality of ideas.

Let your goal tree map guide you

Every A/B test or personalization campaign needs to be fed with good ingredients which determine the quality of the hypothesis.

“Every agency, every in-house company researches. We do research. We collect data, we have information, we get insights and then they test on insights. But you can’t stop there.” Ben says.

The trick is not to stop at the insights part but to actually derive a theme based on those insights. This will allow companies to pick underlying strengths and weaknesses to map them into their OKRs.

For example, you may have a number of insights like a page underperforming, users are confused about pricing and social proof gets skipped over. The key is to conduct a thematic analysis and look for patterns based on these different insights.

Consequently, it’s important for companies to create a goal tree map to help them understand how things cascade down and to become more tactical and SMART about their goals and set their OKRs accordingly to organize and make sense of the vast amount of data.

When the time comes to set up a testing program, teams will have a strategic testing roadmap for a particular theme that links to these OKRs. This helps transform the metrics into more actionable frameworks.

And at the end of each quarter, companies can evaluate their performance based on this scorecard of metrics and how the tests they ran during the quarter impacted these metrics.

Build engagement and efficiency into your testing program strategy

The main value prop of testing centers around making profit but Ben advocates for a second value prop which revolves around how a business operates. This requires shifting focus to efficiency and how different teams across an organization work together.

Ben parallels the A/B testing industry with Devops as it strives to bring in elements from the DevOps cultural movement when we refer to a culture of experimentation and being data-driven. In many ways, this echoes the DevOps methodology, which is focused on breaking down silos between development and operation teams to enhance collaboration and efficiency between these teams. “The whole idea is to optimize the efficiency of a big team working together”, Ben says.

This means organizations should take a hard look at their testing program and the components that make up the program which includes getting the right people behind it. It’s also about becoming more customer-centric and embracing failure.

Ben refers to this as the “programmatic side” of the program which serves as the framework or blueprint for decision making. It helps to answer questions like “how do I organize my team structure?” or “what is my meeting cadence with the team?”

Ultimately, it’s about changing and challenging your current process and transforming your culture internally by engaging your team within testing your program and the way you’re using data to make decisions.

What else can you learn from our conversation with Ben Labay?

Ways to get out of a testing rut
How to structure experimentation meetings to tackle roadblocks
How experimentation relates to game theory
The importance of adopting a actionable framework for decision making

About Ben Labay

Ben Labay combines years of academic and statistics training with customer experience and UX knowledge. Currently, Ben is the CEO at Speero. With two degrees in Evolutionary Behavior and Conservation Research Science (resource management), Ben started his career in academia, working as a staff researcher at the University of Texas focused on research and data modeling. This helped form the foundation for his current passion and work at Speero, which focuses on helping organizations make decisions using customer data.

About 1,000 Experiments Club

The 1,000 Experiments Club is an AB Tasty-produced podcast hosted by Marylin Montoya, VP of Marketing at AB Tasty. Join Marylin and the Marketing team as they sit down with the most knowledgeable experts in the world of experimentation to uncover their insights on what it takes to build and run successful experimentation programs.

You might also like...

See all

Article

6min read

Which Statistical Model is Best for A/B Testing: Bayesian, Frequentist, CUPED, or Sequential?

Emily Healy

Jul 15, 2025

Article

7min read

Is Your Average Order Value (AOV) Misleading You?

Hubert Wassner

Jul 11, 2025

Article

5min read

Why AB Tasty Delivers 4x Faster

Leo Wiel

Jul 7, 2025

Subscribe to
our Newsletter

Article

May 11, 2023

17min read

AB Tasty’s JavaScript Tag Performance and Report Analysis

AB Tasty

Hello! I am Léo, Product Manager at AB Tasty. I’m in charge, among several things, of our JavaScript tag that is currently running on thousands of websites for our clients. As you can guess, my roadmap is full of topics around data collection, privacy and… performance.

In today’s article, we are going to talk about JavaScript tag performance, open-data monitoring and competition. Let’s go!

Performance investigation

As performance has become a big and hot topic during the past few years, mainly thanks to Google’s initiative to deploy their Core Web Vitals, my team and I have been focused a lot on that. We’ve changed a lot of things, improved many parts of our tag and reached excellent milestones. Many of our users have testified of their satisfaction around that. I have already made a (long) series of blog articles about that here. Sorry though, it’s only in French. 😊🥖

From time to time, we get tickled by competitors about a specific report around performance that seems to show us as underperforming based on some metrics. Some competitors claim that they are up to 4 times faster than us! And that’s true, I mean, that’s what the report shows.

You can easily imagine how devastating this can be for the image of my company and how hard it could be for our sales team when a client draws this card. This is especially demoralizing for me and my team after all the work we’ve pushed through this topic during the last few years.

Though it was the first feeling I got when seeing this report, I know for a fact that our performance is excellent. We’ve reached tremendous improvements after the release of several projects and optimizations. Today all the benchmarks and audits I run over our customers’ websites show very good performance and a small impact on the famous Core Web Vitals.

Also, it’s very rare that a customer complains about our performance. It can happen, that’s for sure, but most of the time all their doubts disappear after a quick chat, some explanations and hints about optimization best practices.

But that report is still there, right? So maybe I’m missing something. Maybe I’m not looking at the correct metric. Maybe I’ve only audited customers where everything is good, but there’s a huge army of customers that don’t complain that our tag is drastically slowing their website down.

One easy way to tackle that would be to say that we are doing more with our tag than our competitors do.

Is CRO the same as analytics?

On the report (I promise I will talk about it in depth below 😄), we are grouped in the Analytics Category. However, Conversion Rate Optimization isn’t the same as Analytics. An analytics tool only collects data while we activate campaigns, run personalizations, implement widgets, add pop-ins and more. In this sense, our impact will be higher.

Let’s talk about our competitors: Even though we have the best solution out there (😇), our competitors do more or less the same things as us by using the same technics with the same limits and issues. Therefore, it’s legit to compare us with the same metrics. It might be true that we do a bit more than they do, but in the end, this shouldn’t explain a 4x difference in our performance.

Back then, and before digging into the details, I took the results of the report with humility. Therefore, my ambition was to crawl the data, analyze websites where their tag is running and try to find what they do better than us. We call that retro-engineering, and I find it healthy as it would help to have a faster website for everyone.

My engagement with my management was to find where we had a performance leak and solve it to be able to decrease our average execution time and get closer to our competitors.

But first, I needed to analyze the data. And, wow, I wasn’t prepared for that.

The report

The report is a dataset that is being monthly generated by The HTTP Archive. Here is a quote from their About Page:

“Successful societies and institutions recognize the need to record their history – this provides a way to review the past, find explanations for current behavior, and spot emerging trends. In 1996, Brewster Kahle realized the cultural significance of the Internet and the need to record its history. As a result he founded the Internet Archive which collects and permanently stores the Web’s digitized content.”

“In addition to the content of web pages, it’s important to record how this digitized content is constructed and served. The HTTP Archive provides this record. It is a permanent repository of web performance information such as size of pages, failed requests, and technologies utilized. This performance information allows us to see trends in how the Web is built and provides a common data set from which to conduct web performance research.”

Every month, they run a Lighthouse audit on millions of websites and generate a dataset containing the raw results.

As it is open-source and legit, it can be used by anyone to draw data visualization and ease access to this type of data.

That’s what the inventor of Google Lighthouse, Patrick Hulce, has done. Through his website, GitHub, he provides a nice visualization of this huge dataset and allows anyone to dig into details through several categories such as Analytics, Ads, Social Media and more. As I said, you’ll find the CRO tools in the Analytics category.

The website is fully open-source. The methodology is known and can be accessed.

So, what’s wrong with the report?

Well, there’s nothing technically wrong with it. We could find it disappointing that the dataset isn’t automatically updated every month, but the repository is open-source, so anyone motivated could do it.

However, this is only displaying the data in a fancy manner and not providing any insights or deep analysis of it. Any flaw or inconsistency will remain hidden and it could lead to a situation where a third party is seen as having bad performance compared to others when it is not necessarily the case.

One issue though, not related to the report itself, is the flaw an average could bring with it. That’s also something we are all aware of but that we tend to forget. If you take 10 people, 9 of them earn 800€ a month but one is earning 12 million euros a month, then we could conclude that everyone earns 1.2 million euros per month. Statistically right, but sounds a bit wrong, doesn’t it? More on that in a minute.

Knowing that, it was time to get my hands a bit dirty. With my team, we downloaded the full dataset from February 2023 to run our own audit and understand where we had performance leaks.

Note that downloading the full dataset is something we have been doing regularly for about one and a half years to monitor our trend. However, this time I decided to dig into the February 2023 report in particular.

The analysis

On this dataset, we could find the full list of websites running AB Tasty that have been crawled and the impact our tag had on them. To be more accurate, we have the exact measured execution time of our tag, in milliseconds.

This is what we extracted. The pixellated column is the website URL. The last column is the execution time in milliseconds.

With the raw data, we were able to calculate a lot of useful metrics.

Keep in mind that I am not a mathematician or anything close to a statistics expert. My methodology might sound odd, but it’s adequate for this analysis.

Average execution time

This is the first metric I get — the raw average for all the websites. That’s probably very close, if not equal, to what is used by the thirdpartyweb.today website. We already saw the downside of having an average, however, it’s still an interesting value to monitor.

Mean higher half and mean lower half

Then, I split the dataset in half. If I have 2000 rows, I create two groups of 1000 rows. The “higher” one and the “lower” one. It helps me have a view of the websites where we perform – the worst compared to the best. Then, I calculate the average of each half.

The difference between the two halves

The difference between the two halves is important as it shows the disparity within the dataset. The closer it is, the less extreme values we have.

The number of websites with a value above 6k ms

It’s just an internal metric we follow to give us a mid-term goal of having 0 websites above this value.

The evolution of the last dataset

I compute the evolution between the last dataset I have and the current. It helps me see if we get better in general, as well as how many websites are leaving or entering the chart.

The results

These are the results that we have:

Here are their corresponding graphs:

This is the evolution between October 2022 and February 2023:

Watch out: Logarithmic scale! Sorted by February 2023 execution time from left to right.

The figures say it all. But, if I can give a global conclusion, it’s that we made tremendous improvements in the first six months and staled a bit after with finer adjustments (the famous 80/20 of Pareto’s).

However, after the initial fall, two key figures are important.

First of all, the difference between the two halves is getting very close. This means that we don’t have a lot of potential performance leaks anymore (features that lead to an abnormal increase in the execution time). This is our first recent win.

Then, the evolution shows that in general, and except for the worst cases, it is steady or going down. Another recent win.

Digging into the details

What I have just shared is the raw results without having a look at the details of each row and each website that is being crawled.

However, as we say, the devil is in the details. Let’s dig in a bit.

Let’s focus on the websites where AB Tasty takes more than six seconds to execute.

Six seconds might sound like a lot (and it is), but don’t forget that the audit simulates a low-end CPU which is not representative of the average device. Instead, it shows the worst-case scenario.

In the February 2023 report, there are 33 of them. This is an average execution time of 19877 ms. I quickly identified that:

27 of them are from the same AB Tasty customer
One of them is abtasty.com and the total execution of resources coming from *abtasty.com on this website is very high 😊
Two others are also coming from one singular AB Tasty customer

In the end, we have only 5 customers on this list (but still 33 websites, don’t get me wrong).

Let’s now try to group up these two customers with duplicates to see the impact on the average. The customer with 27 duplicates also has websites that are below the 6k ms mark, but I’m going to ignore it for now (and to ease things up).

For each of the two customers with duplicates, I’m going to compute the average of all their duplicates. For the first one, the result is 21671 ms. For the second, the result is 14708 ms.

I’m also going to remove abtasty.com, which is not relevant.

With the new list, I went from 1223 ms for the full list average to 1005 ms. I just improved our average by more than 200 ms! 🎉

Wait, what? But you’re just removing the worst websites. Obviously, you are getting better…

Yep, that’s true. That’s cheating for sure! But, the point of this whole article is to demonstrate that data doesn’t say it all.

Let’s talk first about what is happening with this customer that has 27 duplicates.

The same tag has been deployed on more than 50 very different websites! You might not be very familiar with AB Tasty, so let me explain why this is an issue.

You might have several websites which have the same layout (that’s often the case when you have different languages). It makes sense to have the same tag on these different domains to be able to deploy the same personalizations on all of them at once. That’s not the most optimal way of doing it, but as of today, that’s the easiest way to do it with our tool.

However, if your websites are all different, there is absolutely no point in doing that. You are going to create a lot of campaigns (in this case, hundreds!) that will almost never be executed on the website (because it’s not the correct domain) but are still at least partially included in the tag. So our tag is going to spend its time checking hundreds of campaigns that have no chance to execute as the URL is rarely going to be valid.

Though we are working on a way to block this behavior (as we have alternatives and better options), it will take months before it disappears from the report.

Note: If you start using AB Tasty, you will not be advised to do that. Furthermore, the performance of your tag will be far better than that.

Again, I didn’t take the time to group all the duplicated domains as it is pointless, the goal was to demonstrate that it is easy to show better performance if we exclude anomalies that are not representative. We can imagine that we would improve more than 200+ ms by keeping only one domain.

I took the most obvious case, but a quick look at the rest of the dataset showed me some other examples.

The competitors’ figures

Knowing these facts and how our score might look worse than it is because of one single anomaly, I started looking into our competitors’ figures to see if they have the same type of issue.

I’m going to say it again: I’m not trying to say that we are better (or worse) than any of our competitors here, that’s not my point. I’m just trying to show you why statistics should be deeply analyzed to avoid any interpretation mistakes.

Let’s start by comparing AB Tasty’s figures for February 2023 with the same metrics for one of them.

Competitor's figures

In general, they look a bit better, right? Better average and even the means for each half is better (and the lower half by a lot!).

However, between the two halves, the factor is huge: 24! Does it mean that depending on your usage, the impact of their tag might get multiplied by 24?

If I wanted to tease them a little bit, I would say that when testing the tag on your website, you might find excellent performance but when starting to use it intensely you might face serious performance drops.

But, that would be interpreting a very small part of what the data said.

Also, they have more than twice the number of websites that are above the 6k ms mark (again: this mark is an AB Tasty internal thing). And that is by keeping the duplicates in AB Tasty’s dataset that we discussed just before! They also have duplicates, but not as many as we do.

A first (and premature) conclusion is that they have more websites with a big impact on performance but at the same time, their impact is lower in general.

Now that I know that in our case we have several customers that have duplicates, I wanted to check if our competitors have the same. And this one does – big time.

Among the 2,537 websites that have been crawled, 40% of them belong to the same customer. This represents 1,016 subdomains of the same domain.

How does this impact their score?

Well, their customer wasn’t using the solution at the moment the data was collected (I made sure of it by visiting some of the subdomains). This means that the tag wasn’t doing anything at all. It was there, but inactive.

The average execution time of these 1,016 rows in the dataset is 59 ms!! 😭 It also has a max value of 527 ms and a min value of 25 ms.

I don’t need to explain why this “anomaly” interestingly pulls down their average, right?

The 1,016 subdomains are not fake websites at all. I’m not implying that this competitor cheated on purpose to look better- I’m sure they didn’t. It is just a very nice coincidence for them, whether they are aware of it or not.

To finish, let’s compare the average of our two datasets after removing these 1,016 subdomains.

AB Tasty is at 1223 ms (untouched list) when this competitor is now at… 1471 ms.

They went from 361 ms better to 248 ms worse. I told you that I can let the figures say whatever I want. 🙂

I would have a lot of other things to say about these datasets, but I didn’t run all the analysis that could have been done here. I already spent too much time on it, to be honest.

Hopefully, though, I’ve made my point of showing that the same dataset can be interpreted in a lot of different manners.

What can we conclude from all of this?

The first thing I want to say is: TEST IT.

Our solution is very easy to implement. You simply put the tag on your website and run an audit. To compare, you can put another tool’s tag on your website and run the same audit. Run it several times with the same conditions and compare. Is the second tool better on your website? Fine, then it will probably perform better for your specific case.

Does a random report on the web says that one solution is better than another? Alright, that’s one insight, but you should either crunch the data to challenge it or avoid paying too much attention to it. Just accepting the numbers as it is displayed (or worse: advertised…) might make you miss a big part of the story.

Does AB Tasty have a bad performance?

No, it doesn’t. Most of our customers never complained about performance and some are very grateful for the latest improvements we’ve released on this topic.

So, some customers are complaining?

Yes. This is because sometimes AB Tasty can have a lower performance depending on your usage. But, we provide tools to help you optimize everything directly from our platform. We call this the Performance Center. It is a full section inside the platform and is dedicated to showing you which campaign is impacting your performance and what you can do to improve it. Just follow the guidelines and you’ll be good. It’s a very innovative and unique feature in the market, and we are very proud of it.

Though, I must admit that a few customers (only a few) have unrealistic expectations about performance. AB Tasty is a JS tag that is doing DOM manipulations, asynchronous checks, data collection and a lot of fancy stuff. Of course, it will impact your website more than a simple analytics tool will. The goal for you is to make sure that the effect of optimizing your conversions is higher than what it costs you in terms of performance. And it will be the same, whatever the CRO tool you are using, except if you use a server-side tool like Flagship by AB Tasty, for example.

I am convinced that we should aim towards a faster web. I am very concerned about my impact on the environment, and I’m trying to keep my devices as long as possible. My smartphone is 7 years old (and I’m currently switching to another one that is 10 years old) and my laptop isn’t very recent either. So, I know that a slow website can be a pain.

Final Remarks

Let me assure you that at AB Tasty we are fully committed to improving our performance because our customers are expecting us to do it, because I am personally motivated to do it, and because that is a very fun and interesting challenge for the team (and also because my management asks me to do it 😅 ).

Also, kudos to HTTP Archive which does very important work in gathering all this data and especially sharing it with everyone. Kudos to Patrick Hulce who took the time to build a very interesting website that helps people have a visual representation of HTTP Archive’s data. Kudos to anyone that works to build a better, faster and more secure web, often for free and because that’s what they believe in.

Want to test our tool for yourself? AB Tasty is the complete platform for experimentation, content personalization, and AI-powered recommendations equipped with the tools you need to create a richer digital experience for your customers — fast. With embedded AI and automation, this platform can help you achieve omnichannel personalization and revolutionize your brand and product experiences.

You might also like...

See all

Article

6min read

Which Statistical Model is Best for A/B Testing: Bayesian, Frequentist, CUPED, or Sequential?

Emily Healy

Jul 15, 2025

Article

7min read

Is Your Average Order Value (AOV) Misleading You?

Hubert Wassner

Jul 11, 2025

Article

5min read

Why AB Tasty Delivers 4x Faster

Leo Wiel

Jul 7, 2025

Subscribe to
our Newsletter

Article

May 10, 2023

11min read

Product Manager vs Product Owner: What’s the Difference?

AB Tasty

What’s the difference between a product manager and a product owner or are these roles actually one and the same?

If you’re wondering, you’re not alone. The terms are often used interchangeably and not without reason as sometimes the two roles’ responsibilities may overlap especially in small to medium-sized companies.

However, Product Manager (PM) and Product Owner (PO) are two distinct roles. Although they do share a common goal, delivering products users love, the scope of their responsibilities is actually not identical.

Generally, a product manager is responsible for the why. They shape the product roadmap based on users’ needs and desires. They are focused on business metrics and on whether the product is going in the right direction on a larger scale.

The product owner, on the other hand, is responsible for creating and managing the product backlog. They are operational and on a deadline. For them, it’s all about Scrum, back and forth with developers, and getting stuff done.

However, the lines between the two do often blur. Depending on where you work, you might be used to a variety of setups.

What are the differences in responsibilities of a product manager and a product owner?

The short answer is: it depends. Product Manager Experience

Now, I’m sure you’re thinking ‘of course! Isn’t that true for everything?’

But in this case, it really does depend on a lot of different factors. A chunk of this article will go over them in detail but first we will discuss each role to highlight the responsibilities of each one on its own to be able to then understand their differences.

The product owner role

The term “product owner” comes from Scrum- an Agile framework that helps teams structure and manage their work and solve complex problems effectively by adopting a set of values and principles. A typical Scrum team consists of Product Owner, a Scrum Master and Developers with specific accountabilities.

According to the “Scrum Guide”, the product owner is responsible for “maximizing the value of the product resulting from the work of the Scrum Team”. Consequently, this role is usually found in organizations that have adopted an Agile methodology.

Among the key responsibilities of the product owner is creating user stories for the development team to implement and ensuring that these stories meet customer requirements. In other words, the product owner advocates for customer needs and represents the voice of the customer to the development team by making sure that the right product is being built.

Some other responsibilities of the product owner include:

Manage product backlog by creating and communicating the backlog items and prioritize them accordingly to maximize value
Define and manage the product vision
Understand market and customer needs and turn them into actionable user stories
Collaborate closely with cross-functional teams such as developers to ensure that products meet customer needs and requirements. They also make sure that goals are clear and business objectives are aligned with the product vision
Act as the primary point of contact for all stakeholders and ensure they have proper buy-in on all major decisions and strategies

The product manager role

As we’ve seen, product owners are more tactical in that their focus is on maximizing value through creating and managing the product backlog. Additionally, product owners are more detail-oriented and have more of a short- to mid-term focus.

Meanwhile, product managers play a more strategic role, which means their focus is more on creating the long-term vision of a product as well as aligning the product roadmap with larger organizational goals.

They then build the product strategy in order to build a concrete product based on its overall vision that spans across the entire roadmap. Put simply, a product manager manages the entire product life cycle and oversees every aspect of its development from the early stages of user research to the moment of product launch.

The responsibilities of the product manager may vary from one organization to the next but they usually include the following tasks:

Conduct user research to determine customer needs to determine product vision
Create and align teams around the product roadmap
Decide what features to build next
Supervise teams and projects to ensure the successful launch of the product

A product manager role is usually more outward-facing in that they speak to customers to define the requirements of the product to be built while a product owner is more inward-facing as they work closely with development teams to ensure the product is being built according to these requirements.

We can look at product owners as an extension of product managers as they apply the strategies outlined by product managers and transform it into an actionable backlog.

In other words, the product manager decides what products or features to build next while the product owner oversees that developers build these products.

Do organizations need both roles?

The simple answer to this question would be it depends.

Various factors will need to be taken into account when it comes to deciding whether you need one or the other or both.

In an ideal scenario, there would be a product manager managing the product strategy and vision and a product owner responsible for executing that strategy. On the surface, it may seem that the responsibilities of the two roles may sometimes overlap as they’re working towards common goals but in reality, their day-to-day tasks differ significantly.

However, not all companies may have the sufficient resources to have two different people dedicated to each of these roles.

Indeed, many small companies may find that having two distinct roles is not necessary and the responsibilities of the product development process can be carried out solely by the product manager. However, processes may become more complex in larger companies and could require a product manager to manage the product life cycle and outline the overall strategy and a product owner overseeing the development process to ensure the tactical execution of that strategy.

It’s important to also keep in mind that the product owner role is tightly linked with the Scrum development approach. Therefore, it’s typically found in teams practicing Scrum. Product managers, for their part, exist in organizations that aren’t necessarily following a specific approach or methodology. They can operate within any framework as part of the product team while product owners in highly Agile businesses come as part of a Scrum team.

Nonetheless, many organizations struggle with the decision of whether to have one or the other or both.

The most important consideration when making this decision is focusing on outcomes and not on the titles. Organizations need to examine their objectives and any weaknesses and bottlenecks in their current processes or structures that may hinder them from achieving these objectives. Such insights are key to building your winning team.

When to consider a product manager vs product owner

Let’s not forget, there are products of all kinds: industrial products, agricultural products, services, chemical products, fashion and software products.

In the SaaS martech space, the thinking goes that all products are built the same way and that companies are all structured the same way. They tend to think there is always a Product Manager and a Product Owner behind each feature.

We’ve already mentioned some things to consider when you’re deciding whether to have both roles. In this section, we look in further detail at some of these factors that determine the right setup for your organization.

1. Momentum

The most important criteria is ‘the momentum of the company.’ What does this mean?

How new/old is the organization?

This question is an important one. It is not directly related to the size of the company or to the number of clients, either.

But rather, the right question here would be: How far or close is the company in relation to a specific “key moment”?

product owner vs product manager ab tasty

What we mean by ‘key moment’ are key phases or events in the life cycle of a company. Some examples include:

the creation of the company
its sale
new funding
tax inspection
a competitor’s release
a crisis
replacement of the CEO
… etc

All these different key stages will have an impact on how the product development is executed.

And therefore, they affect what a product manager does and what a product owner does.

For example, just after a company is founded, the CEO is presumably the best person to take care of the product roles. They are very often (at least in scalable SaaS companies) the one who wears the hat of PO, PM, QA & Support. It is not a choice or a strategy. It is just what the moment implies.

Also, in general, the way a company’s momentum is handled defines how “well” a company is doing.

Has the moment happening right now been anticipated, or is it just the consequence of the reaction of the last moment? The more a company reacts, the more “Agile” you can consider it to be. The more moments are anticipated, the more “visionary” you can consider the company. It’s challenging to be both.

2. Size matters

Size here doesn’t necessarily refer to the number of employees a company has. The size of a company could also refer to the number of:

products
models
features
scopes
price grids
or markets they address.

Cutting the products in the right “dimensions” and deciding if we split per market, per range or per topic can be a lifelong job, because it changes very often and the faster the company evolves, the more it has to be changed.

Size can also be defined by the number of users, clients, partners, retailers, languages, unit systems,… But also and more frequently, a combination of all the above will tell how many PMs, POs and Heads of Product your organization will need.

For example, in many non-profits, product managers are based on the donor personas. You have a PM for small donation amounts (say $10 to $500), one for larger ones ($500 to $10,000) and one for those that are even bigger. Why? Because expectations from each persona type differ radically, and being able to cater to each will help the non-profit grow in the long run.

In short, the size of the company (its client portfolio, its product catalogue…) may define what the PM does, what the PO does, and how many of each there will be.

3. The remaining criteria

But these are not the only factors that define how many PMs, POs or any other product-facing people you will need. Many other factors can also be taken into account, including:

number of developers
number of designers
shipping velocity
technical stacks
tooling
product lifecycle
individual wishes
growth potential
existing need coverage ratio
legacy ratio
human resources policy
the market you address
the level of politics installed in the company.

There is some kind of global agreement (at least in SaaS businesses) on the fact that the split between PM and PO is based on: vision vs. operations, projection vs. immediacy, value vs. metrics. But both are sides of the same coin.

In short, no two organizations are identical. In other words, there’s no one set answer to the question of what a PM vs. PO should do. Some organizations will need many, some will need none. Enabling people to work successfully together is hard enough. Copying and pasting a method won’t help – you have to find your own.

What skills are needed to be a good product manager and/or product owner?

Both product managers and product owners need a good balance of hard and soft skills to carry out their tasks efficiently.

The necessary skills for product managers include:

Excellent communication and collaboration skills
A good amount of technical expertise
Prioritization skills
Good business acumen
Analytical skills

Meanwhile, the necessary skills for product owners include:

Problem solving
Project management skills
Great communication and strong storytelling skills
Deep understanding of user data and analytics

The skills necessary for product managers and product owners may overlap especially in smaller companies where a product manager could also be a product owner and vice versa.

What KPIs are used to measure success for product owners and product managers?

It depends and may vary from one organization to the other depending on their objectives

However, if we tie them into their individual responsibilities, we can give a general overview of the kind of KPIs that can be used to measure their performance.

A product manager, for example, has a wide range of responsibilities including the creation and prioritization of the product roadmap along with cross-team collaboration to ensure alignment around the roadmap.

In that sense, a product manager’s performance is primarily related to product success in line with the overall business goals. Thus, usually product managers’ KPIs should be based on business metrics such as growth, revenue, churn rate and costs.

Meanwhile, product owners have a narrower role and based on the responsibilities we outlined above, their success is measured using KPIs based on delivery, quality, and internal team satisfaction.

Conclusion

A product owner and manager are great assets to have but having both or two separate people taking on these roles is not a must for every organization.

At the end of the day, the question is not whether you should have both a product manager and product owner but whether having both of those roles is right for your organization. You will have to look inward to understand what your business actually needs.

How you decide to structure your teams will depend on the processes you have in place and the kind of outcomes you’re hoping to achieve based on business objectives, customer and company needs.

While we give an overall idea of each role, these responsibilities are not set in stone and they could widely vary on a case by case basis.

What really matters is that you have the right people with the right skills working towards a shared goal, which is building a product customers will love and meets their requirements and needs.

You might also like...

See all

Article

6min read

Which Statistical Model is Best for A/B Testing: Bayesian, Frequentist, CUPED, or Sequential?

Emily Healy

Jul 15, 2025

Article

7min read

Is Your Average Order Value (AOV) Misleading You?

Hubert Wassner

Jul 11, 2025

Article

5min read

Why AB Tasty Delivers 4x Faster

Leo Wiel

Jul 7, 2025

Subscribe to
our Newsletter

Article

Apr 27, 2023

13min read

How to Deal with Low Traffic in CRO

Hubert Wassner

If your website traffic numbers aren’t as high as you may hope for, that’s no reason to give up on your conversion rate optimization (CRO) goals.

By now you must have noticed that most CRO advice is tailored for high-traffic websites. Luckily, this doesn’t mean you can’t optimize your website even if you have lower traffic.

The truth is, any website can be optimized – you just need to tailor your optimization strategy to suit your unique situation.

In this article, we will cover:

The most common complaint about CRO (95% threshold and where it comes from)
An appropriate threshold for low traffic
Ideas on how to optimize your website with a smaller traffic flow
The CUPED testing technique
When CUPED works and doesn’t work

CRO analogy

In order to make this article easier to understand, let’s start with an analogy. Imagine that instead of measuring two variants and picking a winner, we are measuring the performance of two boxers and placing bets on who will win the next 10 rounds.

So, how will we place our bet on who will win?

Imagine that boxer A and boxer B are both newbies that no one knows. After the first round, you have to make your choice. In the end, you will most likely place your bet on the boxer who won the first round. It might be risky if the winning margin is small, but in the end, you have no other way to base your decision.

Imagine now that boxer A is known to be a champion, and boxer B is a challenger that you don’t know. Your knowledge about boxer A is what we would call a prior – information you have before that influences your decision.

Based on the prior, you will be more likely to bet on boxer A as the champion for the next few rounds, even if boxer B wins the first round with a very small margin.

Furthermore, you will only choose boxer B as your predicted champion if they win the first round by a large margin. The stronger your prior, the larger the margin needs to be in order to convince you to change your bet.

Are you following? If so, the following paragraphs will be easy to grasp and you will understand where this “95% threshold” comes from.

Now, let’s move on to tips for optimizing your website with low traffic.

1. Solving the problem: “I never reach the 95% significance”

This is the most common complaint about CRO for websites with lower traffic and for lower traffic pages on bigger websites.

Before we dig into this most common problem, let’s start by answering the question, where does this 95% “golden rule” come from?

The origin of the 95% threshold

Let’s start our explanation with a very simple idea: What if optimization strategies were applied from day one? If two variants with no previous history were created at the same time, there would be no “original” version challenged by a newcomer.

This would force you to choose the best one from the beginning.

In this setting, any small difference in performance could be measured for decision-making. After a short test, you will choose the variant with the higher performance. It would not be good practice to pick the variant that had lower performance and furthermore, it would be foolish to wait for a 95% threshold to pick a winner.

But in practice, optimization is done well after the launch of a business.

So, in most real-life situations, there is a version A that already exists and a new challenger (version B) that is created.

If the new challenger, version B, comes along and the performance difference between the two variants is not significant, you will have no issues declaring version B “not a winner.”

Statistical tests are symmetric. So if we reverse the roles, swapping A and B in the statistical test will tell you that the original is not significantly better than the challenger. The “inconclusiveness” of the test is symmetric.

So, why do you set 100% of traffic toward the original at the end of an inconclusive test, implicitly declaring A as a winner? Because you have three priors:

Version A was the first choice. This choice was made by the initial creator of the page.
Version A has already been implemented and technically trusted. Version B is typically a mockup.
Version A has a lot of data to prove its value, whereas B is a challenger with limited data that is only collected during the test period.

Points 1 & 2 are the bases of a CRO strategy, so you will need to go beyond these two priors. Point 3 explains that version A has more data to back its performance. This explains why you trust version A more than version B. Version A has data.

Now you understand that this 95% confidence rule is a way of explaining a strong prior. And this prior mostly comes from historical data.

Therefore, when optimizing a page with low traffic, your decision threshold should be below 95% because your prior on A is weaker due to its traffic and seniority.

The threshold should be set according to the volume of traffic that went through the original from day one. However, the problem with this approach is that we know that the conversion rates are not stable and can change over time. Think of seasonality – i.e. black Friday rush, vacation days, Christmas time increases in activity, etc. Because of the seasonal changes, you can’t compare performances in different periods.

This is why practitioners only take into account data for version A and version B taken at the same period of time and set a high threshold (95%) to accept the challenger as a winner in order to formalize a strong prior toward version A.

What is the appropriate threshold for low traffic?

It’s hard to suggest an exact number to focus on because it depends on your risk acceptance.

According to the hypothesis protocol, you should structure a time frame for the data collection period in advance.

This means that the “stop” criteria of a test are not a statistical measure or based on a certain number. The “stop” criteria should be a timeframe coming to an end. Once the period is over, then you should look at the stats to make an appropriate decision.

AB Tasty, our customer experience optimization and feature management software, uses the Bayesian framework which produces a “chances to win” index which encourages a direct interpretation instead of a p-value, which has a very complex meaning.

In other words, the “chances to win index” is the probability for a given variation to be better than the original.

Therefore, a 95% “chance to win” means that there is a 95% probability that the given variation will be the winner. This is assuming that we don’t have any prior knowledge or specific trust for the original.

The 95% threshold itself is also a default compromise between the prior you have on the original and a given level of risk acceptance (it could have even been a 98% threshold).

Although it is hard to give an exact number, let’s make a rough scale for your threshold:

New A & B variations: If you have a case where variation A and variation B are both new, the threshold could be as low as 50%. If there is no past data on the variations’ performance and you must make a choice for implementation, even a 51% chance to win is better than 49%.
New website, low traffic: If your website is new and has very low traffic, you likely have very little prior on variation A (the original variation in this case). In that case, setting 85% as a threshold is reasonable. Since it means that if you put aside the little you know about the original you still have 85% to pick up the winner and only 15% to pick a variation that is equivalent to the original, and a lesser chance that it performs worse. So depending on the context, such a bet can make sense.
Mature business, low traffic: If your business has a longer history, but still lower traffic, 90% is a reasonable threshold. This is because there is still little prior on the original.
Mature business, high traffic: Having a lot of prior, or data, on variation A suggests a 95% threshold.

The original 95% threshold is far too high if your business has low traffic because there’s little chance that you will reach it. Consequently, your CRO strategy will have no effect and data-driven decision-making becomes impossible.

By using AB Tasty as your experimentation platform, you will be given a report that includes the “chance to win” along with other statistical information regarding your web experiments. A report from AB Tasty would also include the confidence interval on the estimated gain as an important indicator. The boundaries around the estimated gain are also computed in a Bayesian way, which means it can be interpreted as the best and the worst scenario.

The importance of Bayesian statistics

Now you understand the exact meaning of the well-known 95% “significance” level and are able to select appropriate thresholds corresponding to your particular case.

It’s important to remember that this approach only works with Bayesian statistics since frequentist approaches give statistical indices (such as p-Values and confidence intervals that have a totally different meaning and are not suited to the explained logic).

2. Are the stats valid with small numbers?

Yes, they are valid as long as you do not stop the test depending on the result.

Remember the testing protocol says once you decide on a testing period, the only reason to stop a test is when the timeframe has ended. In this case, the stat indices (“chances to win” & confidence interval) are true and usable.

You may be thinking: “Okay, but then I rarely reach the 95% significance level…”

Remember that the 95% threshold doesn’t need to be the magic number for all cases. If you have low traffic, chances are that your website is not old. If you refer back to the previous point, you can take a look at our suggested scale for different scenarios.

If you’re dealing with lower traffic as a newer business, you can certainly switch to a lower threshold (like 90%). The threshold is still higher because it’s typical to have more trust in an original rather than a variant because it’s used for a longer time.

If you’re dealing with two completely new variants, at the end of your testing period, it will be easier to pick the variant with the higher conversions (without using a stat rest) since there is no prior knowledge of the performance of A or B.

3. Go “upstream”

Sometimes the traffic problem is not due to a low-traffic website, but rather the webpage in question. Typically, pages with lower traffic are at the end of the funnel.

In this case, a great strategy is to work on optimizing the funnel closer to the user’s point of entry. There may be more to uncover with optimization in the digital customer journey before reaching the bottom of the funnel.

4. Is the CUPED technique real?

What is CUPED?

Controlled Experiment Using Pre-Experiment Data is a newer buzzword in the experimentation world. CUPED is a technique that claims to produce up to 50% faster results. Clearly, this is very appealing to small-traffic websites.

Does CUPED really work that well?

Not exactly, for two reasons: one is organizational and the other is applicability.

The organizational constraint

What’s often forgotten is that CUPED means Controlled experiment Using Pre-Experiment Data.

In practice, the ideal period of “pre-experiment data” is two weeks in order to hope for a 50% time reduction.

So, for a 2-week classic test, CUPED claims that you can end the test in only 1 week.

However, in order to properly see your results, you will need two weeks of pre-experiment data. So in fact, you must have three weeks to implement CUPED in order to have the same accuracy as a classic 2-week test.

Yes, you are reading correctly. In the end, you will need three weeks time to run the experiment.

This means that it is only useful if you already have two weeks of traffic data that is unexposed to any experiment. Even if you can schedule two weeks of no experimentations into your experimentation planning to collect data, this will be blocking traffic for other experiments.

The applicability constraint

In addition to the organizational/2-week time constraint, there are two other prerequisites in order for CUPED to be effective:

CUPED is only applicable to visitors browsing the site during both the pre-experiment and experiment periods.
These visitors need to have the same behavior regarding the KPI under optimization. Visitors’ data must be correlated between the two periods.

You will see in the following paragraph that these two constraints make CUPED virtually impossible for e-commerce websites and only applicable to platforms.

Let’s go back to our experiment settings example:

Two weeks of pre-experiment data
Two weeks of experiment data (that we hope will only last one week as there is a supposed 50% time reduction)
The optimization goal is a transaction: raising the number of conversions.

Constraint number 1 states that we need to have the same visitors in pre-experiment & experiment, but the visitor’s journey in e-commerce is usually one week.

In other words, there is very little chance that you see visitors in both periods. In this context, only a very limited effect of CUPED is to be expected (up to the portion of visitors that are seen in both periods).

Constraint number 2 states that the visitors must have the same behavior regarding the conversion (the KPI under optimization). Frankly, that constraint is simply never met in e-commerce.

The e-commerce conversion occurs either during the pre-experiment or during the experiment but not in both (unless your customer frequently purchases several times during the experiment time).

This means that there is no chance that the visitors’ conversions are correlated between the periods.

In summary: CUPED is simply not applicable for e-commerce websites to optimize transactions.

It is clearly stated in the original scientific paper, but for the sake of popularity, this buzzword technique is being misrepresented in the testing industry.

In fact, and it is clearly stated in scientific literature, CUPED works only on multiple conversions for platforms that have recurring visitors performing the same actions.

Great platforms for CUPED would be search engines (like Bing, where it has been invented) or streaming platforms where users come daily and do the same repeated actions (playing a video, clicking on a link in a search result page, etc).

Even if you try to find an application of CUPED for e-commerce, you’ll find out that it’s not possible.

One may say that you could try to optimize the number of products seen, but the problem of constraint 1 still applies: a very little number of visitors will be present on both datasets. And there is a more fundamental objection – this KPI should not be optimized on its own, otherwise you are potentially encouraging hesitation between products.
You cannot even try to optimize the number of products ordered by visitors with CUPED because constraint number 2 still holds. The act of purchase can be considered as instantaneous. Therefore, it can only happen in one period or the other – not both. If there is no visitor behavior correlation to expect then there is also no CUPED effect to expect.

Conclusion about CUPED

CUPED does not work for e-commerce websites where a transaction is the main optimization goal. Unless you are Bing, Google, or Netflix — CUPED won’t be your secret ingredient to help you to optimize your business.

This technique is surely a buzzword spiking interest fast, however, it’s important to see the full picture before wanting to add CUPED into your roadmap. E-commerce brands will want to take into account that this testing technique is not suited for their business.

Optimization for low-traffic websites

Brands with lower traffic are still prime candidates for website optimization, even though they might need to adapt to a less-than-traditional different approach.

Whether optimizing your web pages means choosing a page that’s higher up in the funnel or adopting a slightly lower threshold, continuous optimization is crucial.

Want to start optimizing your website? AB Tasty is the best-in-class experience optimization platform that empowers you to create a richer digital experience – fast. From experimentation to personalization, this solution can help you activate and engage your audience to boost your conversions.

You might also like...

See all

Article

6min read

Which Statistical Model is Best for A/B Testing: Bayesian, Frequentist, CUPED, or Sequential?

Emily Healy

Jul 15, 2025

Article

7min read

Is Your Average Order Value (AOV) Misleading You?

Hubert Wassner

Jul 11, 2025

Article

5min read

Why AB Tasty Delivers 4x Faster

Leo Wiel

Jul 7, 2025

Subscribe to
our Newsletter

Article

Apr 25, 2023

10min read

Why You Should Slot Feature Flags into Your Agile Roadmap

AB Tasty

It’s easy to lose your way when building an Agile roadmap.

If you get too detailed with your planning, you end up building a roadmap that is Agile in name alone but looks more like a traditional Waterfall roadmap. If you don’t perform enough planning, then you’ll produce a skeleton of a roadmap that sends you running in multiple directions without ever arriving anywhere meaningful.

The correct approach lies somewhere in the middle. You keep things loose, nimble, and iterative but you also set a beacon that will guide each of your sprints to an impactful destination.

From our experience, one “beacon” that will keep your Agile product roadmap grounded, and your products moving in the right direction, is a simple function— the feature flag.

It isn’t fancy. It isn’t flashy. And it doesn’t look overly strategic. If you use feature flags properly, then they will keep your Agile roadmap focused on the outcomes that matter most without forcing you down a fixed path. Here’s why.

First principles: The real benefit of Agile over Waterfall

It feels like a given these days: if you work as a Product Manager (especially in the tech sector) then you’re going to follow some kind of Agile methodology. Depending on your work history, you may never have worked with a Waterfall roadmap, let alone developed one, in your entire career.

If that’s the case, it might even feel confusing why Waterfall was ever developed. The methodology is slow. It’s rigid. It’s opaque. On the surface, it looks inferior to Agile in every way. But once you dig into it a little, there is one area where waterfall trumps Agile. Waterfall is a better fit within a traditional corporate context than Agile.

While Agile and Waterfall are popular in software development, each one is best suited for different types of projects.

For example, a Waterfall approach makes sense when a software project has clearly defined requirements with low probability that any changes will occur halfway through.

Waterfall fits really well into that broader corporate world’s standard operating procedures. It collects business requirements in a standard one-off phase and then sets them in stone with a concrete project. Waterfall adopts a more linear way of working so that development phases flow in one direction just like the flow of a waterfall, hence the name and tends to occur over a long period of time.

It breaks that project into a clear, crisply defined plan and each step must be completed before moving onto the next phase. In the end, the project’s success will be defined by how well its leaders completed the milestones in the project’s plan, and if they delivered to the project’s requirements on-time and on-budget.

Waterfall methodology isn’t really about trying to create the most effective, efficient, or accountable system. It’s about having the product developers and managers operate in a way that makes sense to a large, lumbering corporation.

A new approach—Agile— was only possible because it was developed outside of this legacy corporate context. In fact, Agile is an iterative approach that came about as a response and alternative to Waterfall’s rigid and linear structure.

And here’s what they came up with: product management would deliver a greater impact if it stopped lining up to what a corporation wanted, and if it instead lined up to what actual real-world users want.

In an Agile approach, which introduces flexibility, teams work on multiple phases at the same time with the goal to enable faster software delivery for the purpose of collecting customer feedback. It does this by breaking down the software development life cycle into sprints, which could last from one to four weeks, that include regular feedback loops.

Incremental releases means teams can build better features much quicker that offer more value and optimize and iterate these features based on the feedback received. It aligns the product not only with the product vision but also with customer needs.

This is the real innovation of an Agile roadmap over a Waterfall one. It isn’t the increased speed & efficiency that everyone fixates on. It’s the simple but powerful fact that an Agile roadmap re-aligns the product manager’s focus onto the user.

Here are some of the advantages of an Agile methodology:

Faster feedback loops
Higher customer satisfaction
Reduced time-to-market
Increased flexibility with more room for innovation
Enhanced productivity by breaking down projects into smaller, more manageable chunks

And most of Agile methodology’s core user-alignment activities occur during Feature Release Management and are brought to life by the right feature flag tool.

A quick caveat: Yes, business impact still matters in Agile

Before we move on, let’s make one point very clear.

When we say Waterfall aligns well to corporate context, we mean corporate operational context. But we don’t mean a Waterfall approach offers the best way to deliver results.

Most often, these big Waterfall projects deliver poor results because they can take months—or even years—between their initial requirements collection and their project’s completion. During this time, the project’s alignment, and even its viability to its users, often shifts, reducing its chances of producing any meaningful business impact.

By contrast, a properly developed and managed Agile roadmap will maintain alignment with its users throughout its entire lifecycle and deliver concrete, measurable, and accountable results.

Feature release management, and feature flags, can also drive this tight connection between user-centered development and KPI improvement. We’ll get to how in just a minute.

Feature release management: The heart of any effective Agile roadmap

From a user-alignment perspective, feature releases are the key point that differentiates an Agile roadmap from a Waterfall roadmap.

Agile looks different from Waterfall in many areas of activity.

In Waterfall, new products and features are released to all users at once, in a single big bang, after a very long development cycle. In an Agile roadmap, new products and features can be—and should be—released at a much faster rate.

This is the key functional difference that makes Agile more user-centered than Waterfall. Rapid and effective feature release management lets you:

Keep your users top-of-mind at all times.
Regularly collect your users’ data and feedback.
Use up-to-date feedback to guide your development cycles.
Repeat the cycle, to make sure you correctly incorporated user feedback in your next round of features and product updates.

If you want to keep your development user-centered then feature release management is critical to effectively incorporate into your Agile product roadmap. Here’s how.

The 5 key elements to include in your Agile release planning

Agile release planning is key to building customer-centric products by allowing you to prioritize and release product requirements as needed. In other words, it allows you to plan your product’s incremental releases- your features- and helps ensure your project is headed in the right direction and following the Agile methodology.

It differs from a product roadmap in that release planning focuses on one sprint at a time (on short-term goals) while a product roadmap looks further ahead in the future and focuses on long-term objectives.

Put simply, the goal of a release plan is to help you prioritize features of your product and focus on releasing specific features in less time to improve the customer experience. Thus, teams use this kind of planning when they’re dividing a project into short sprints or increments instead of planning for one major product release.

It is a unique approach to planning as it takes into account the flexible nature of software development by leaving room for any necessary adjustments as you go through the development lifecycle to incorporate customer (and stakeholder) feedback.

The idea is to be open to prioritizing tasks to provide improved value to your customers.

Here are the key elements to include in each of your feature releases that will turn them into a critical, recurring touchpoint between you and your users.

1. User segmentation

At a basic level, you need to carefully select which user audiences you will first release (and test) new features and products to.

At a deeper level, user segmentation can flow throughout every step of feature release management. You can personalize the experience of your new products and features to each segment you test them with. In other words, you try out different versions of each new product or feature with different segments.

During testing, you can rapidly toggle features off for segments who are not responding well to them. And you can even guide the ongoing development of your products and features depending on which user segments respond the best to them.

2. KPIs measurement

However you measure product or feature success, you must quantify it, and measure those metrics in real-time during each release.

Doing so serves two purposes: First, it gives you the ability to produce an accurate, objective measure about which products and features are succeeding with which segment (and whether or not you are actually improving their performance during each development sprint).

Second, they let you demonstrate concrete, measurable, and accountable results for your business—to both report on the success of your most recent development, and to create meaningful justifications for more robust rollouts.

3. Governance

You need some formalized way to make decisions off the data that you produce. When do you toggle a feature on or off and for who? When do you roll out the product or feature to new segments? When is a product or feature ready to deploy to your entire user community?

To make these decisions, you must have established definitions for success (see “KPIs”), and defined procedures for monitoring and acting on release performance data both in real-time and during post-release recaps.

4. A/B testing

Any time you are segmenting audiences, testing multiple variations on products and features, and collecting copious amounts of real-world user data, then you are setting the stage for multiple A/B tests.

By performing comprehensive A/B tests during each of your feature releases, you will eliminate multiple developmental dead ends and narrow the list of viable “next steps” for your next sprint.

5. Automation

If you incorporate these four elements, then your feature release management process will get pretty complex, pretty quickly. But if you select the right tool to automate as many of these elements and their internal processes,as possible, then you can let go of most operational elements. Afterwards, you would simply sit back during feature releases and focus on making informed decisions before, during, and after each of your releases.

By incorporating each of these five elements into your feature release process, you will ensure that each of these critical touch points brings you and keeps you as close as possible to your users.

And, thankfully, there is one single function that incorporates each of these elements and makes them a practical and effortless habit in your Agile roadmap— feature flags.

Bringing it all home: Feature flags

At their core, the goal of feature flags is to enable you to toggle features on or off, with a single click on your dashboard, without having to adjust your codebase.

That may seem very basic at first glance but buried in this simplicity is a lot of depth, and a lot of room to easily deliver on each of the above elements of user-centered feature release management.

With the right feature flag tool, you can:

Perform sophisticated real-time control over which user segments get new products and features.
Attach core KPIs to your releases and immediately kill products and features that are not performing well while immediately expanding the release of those that are knocking it out of the park.
Monitor your results (and take action) in real-time.
Easily manage and act on complex A/B tests.
Bundle feature flags in with a complete suite of feature release functionality to wrap the whole exercise up in a single, highly-automated platform.

We kept each of these functions in mind when we built our own Feature Flag function, and release management platform.

If you’d like to take it for a test run and see how easy you can incorporate the core actions of Feature Flagging, feature release management, and user-centered Agile product management into your roadmap, drop us a line!

You might also like...

See all

Article

6min read

Which Statistical Model is Best for A/B Testing: Bayesian, Frequentist, CUPED, or Sequential?

Emily Healy

Jul 15, 2025

Article

7min read

Is Your Average Order Value (AOV) Misleading You?

Hubert Wassner

Jul 11, 2025

Article

5min read

Why AB Tasty Delivers 4x Faster

Leo Wiel

Jul 7, 2025

Subscribe to
our Newsletter

Article

Apr 12, 2023

4min read

AB Tasty Unleashes New GA4 Integration for Next-Level Experimentation

John Hughes

We know how important data is to our clients. Understanding what people are doing on your website, app, or mobile site, is vital for businesses to create the best digital experiences. To meet customer demands for a more personalized experience, brands turn to AB Tasty to optimize at every stage of the digital customer journey.

And now we’re taking your data-driven optimization strategy even further with our GA4 integration.

What this integration means for marketing teams

By connecting AB Tasty and Google Analytics 4 (GA4), marketing teams have a clearer vision of how visitors interact with their site via advanced analytics on CPA, conversion rate, bounce rate, SEO and traffic. This integration means you can use data from either tool to better understand the effects of your experimentation during or post-rollout and drive more innovative ideas with data-backed hypotheses.

With this simplified integration, you can seamlessly connect Analytics with AB Tasty enabling you to leverage the robust reporting and intelligence features of Analytics, while taking advantage of our digital experience software.

The AB Tasty and Google collaboration offers businesses a powerful toolkit for optimizing their digital experiences and driving better results. By combining advanced analytics capabilities with sophisticated testing and personalization features, businesses can gain a competitive edge and deliver more value to their customers, deepen their understanding of users’ behavior and preferences, and use that knowledge to create more effective and engaging experiences.

With the ability to test and personalize every element of their digital experiences, businesses can ensure that their websites, apps, and campaigns are optimized for maximum engagement and conversion, ultimately leading to increased revenue and customer loyalty. In short, the AB Tasty and Google integration is essential for businesses looking to stay ahead of the curve in today’s fast-paced digital landscape.

AB Tasty partners with Google Cloud

As AB Tasty strengthens its partnership with Google, we are delighted to announce that we are an official Google Cloud Partner.

This means that clients and partners can access AB Tasty’s digital experience tools and Google’s secure platform. Engage and activate your clients through AB Tasty’s suite of products, testing and experimentation, personalization, server-side testing, feature flags, search and product recommendations, to create resilient digital products, make exceptional customer experiences, and deliver strong business results.

A continued partnership

This strategic alliance is crucial for businesses looking to offer exceptional products and services to their customers. By harnessing the advanced capabilities of AB Tasty’s platform and Google Cloud’s cutting-edge technologies in AI and ML, this partnership promises to empower companies in delivering optimized solutions that drive superior outcomes for their consumers.

Businesses with an annual commitment to Google Cloud can easily add AB Tasty solutions to their packages and quotes with ease. The partnership will also enable customers to access existing vendor relationships with Google Cloud, streamlining the deployment of AB Tasty on the platform.

Google Optimize Sunset

AB Tasty is excited to be a launch partner for Google Analytics newly released experimentation dimension and to help Google Optimize customers transition to new software.

By making the move to AB Tasty, you can gain access to top-of-the-line experience optimization tools to elevate your digital experiences to the next level. For companies that have progressed further in their CRO journey and require more extensive experimentation capabilities, AB Tasty provides a superior solution.

Discover the perks of our advanced solution that delivers a range of integrations with diverse providers, personalized features, pre-built widgets for optimizing the customer journey, and expert CSMs and account managers to offer personalized support throughout the entire contract period. This includes seamless transfer of your test history and data from Google Optimize.

Improving each touchpoint for your customers and learning from every product and feature rollout can have a big impact on your ROI.

The GA4 and AB Tasty integration levels up your ability to create highly personalized experiences for each step of the customer journey with a wide range of audience targeting options and in-depth data on user behavior.

Are you ready to make the move?

The post-Google Optimize world doesn’t have to be bleak.

As one of Google’s top picks as your new A/B testing platform, AB Tasty is a best-in-class A/B testing tool that helps you convert more customers by leveraging experimentation to create a richer digital experience – fast. This experience optimization platform embedded with AI and automation can help you achieve the perfect digital experience with ease.

You might also like...

See all

Article

6min read

Which Statistical Model is Best for A/B Testing: Bayesian, Frequentist, CUPED, or Sequential?

Emily Healy

Jul 15, 2025

Article

7min read

Is Your Average Order Value (AOV) Misleading You?

Hubert Wassner

Jul 11, 2025

Article

5min read

Why AB Tasty Delivers 4x Faster

Leo Wiel

Jul 7, 2025

Subscribe to
our Newsletter

Article

Apr 6, 2023

11min read

How Can Teams Use Feature Flags in Mobile App Deployment?

Rowan Haddad

In the digital age, companies can no longer only focus their efforts on optimizing for desktop especially as more and more consumers are using their mobile devices to visit websites and make purchases through apps.

However with millions of apps out there, competition and consumer demands and expectations are at an all-time high. This means your app needs to stand out in an overcrowded market.

It’s important to point out that deploying mobile apps doesn’t follow the same process as a website app.

In this article, we will investigate the challenges of mobile app deployment and release and how feature flags are the key to help you deliver optimized mobile apps that meet your consumers’ needs.

The challenges of mobile app deployment

Mobile development teams are particularly susceptible to bugs and long, tedious release cycles.

In short, there are two main problems when it comes releasing or updating features on mobile apps:

You have to wait until approval from app stores (which could take some time and significantly delay a release).
Afterwards, you have to wait for users to manually download the update from the store (which could also take a lot of time).

For example, let’s take a look at this scenario: you’re working on an update to your mobile app. You finally release it only to find out that there’s a bug you missed that’s causing your app to crash.

By the time you release a new update with a fix to the issue, wait for the release to the app store and watch for users to download the update, you might risk losing a significant number of users.

Mobile developers and engineers are all too familiar with such a scenario.

Therefore, it can be a painstakingly long process to get a release approved. Once approved, any buggy release will need to be fixed and put through the app store approval process all over again, leading to further delays.

Although the review time has improved in recent years, if your app fails to meet the app store review guidelines it may be further delayed. This means that you cannot push real-time updates to production as you would with web apps.

Put simply, the process of deploying mobile apps is not as straightforward as it might be for web apps.

Unlike web apps which are automatically updated once visitors access the site, users have to download an update of the mobile app in their store to get the latest version. As updates pile up after going through the review process, you have no control over whether users download the latest versions.

Therefore, it can take more time to deploy mobile app updates compared to web apps. And in an age where customers demand the best every time, it’s not feasible to have them wait that long for an update, especially when there’s a bug involved, much less having to wait to deploy a new app version once the bug is removed.

In modern software development, when continuous delivery is vital to retain competitiveness and meet fast-changing consumer demands, teams must turn to another solution to achieve a more frequent release cadence.

The value of feature flags in mobile app deployment and release

This is where feature flags come into play.

Unlike client-side testing where experiments are focused on web browsers, feature flags give teams the ability to carry out server-side experiments across multiple channels including mobile apps.

Feature flags allow teams to enable or disable features to users of their choosing to minimize risk and negative impact.

This is because feature flags allow you to decouple deployment from feature releases meaning you can turn functionality on or off remotely without redeploying code to app stores and waiting for its approval or having to wait for all changes to be ready at the same time to release your own changes. This way you can deploy code to whoever whenever you want.

Thus, you can upgrade your app continuously in real time based on feedback from your users without sending an app store update or waiting on its approval. You can also gradually release new features without users having to go through the hassle of always having to update their app version.

With feature flags, mobile developers can safely test in production on a pre-defined audience and disable any features with a kill switch should any issues come up, thereby reducing any negative impact. Developers can then work on pinpointing the issue and fixing it before releasing the feature to all users.

How can you use feature flags in mobile apps?

Feature flags can be used not only by developers but also by product and release managers to optimize mobile experiences in various ways.

Here are some examples of use cases for mobile feature flags:

A/B testing: With feature flags, you can divide your users into subsets with each set of users receiving a different variation of the feature. This allows you to test and determine which is the top-performing variation to roll out to all your users. Put simply, running A/B tests allow you to collect valuable live feedback from your users so you can make informed decisions about how to optimize your features and products.
Targeted rollouts: Teams can use feature flags to test out their ideas by progressively rolling out their feature, giving only a limited number of users ]early access to the app through beta testing for example. This helps generate buzz around your release and lets you monitor its impact on these select users. Targeted rollouts allow teams to make more informed decisions about what to optimize and fine-tune an app based on live user feedback.
Personalization: Feature flags are a great way to personalize experiences for different types of users rather than delivering a unified experience for all your users. By changing the features that certain users receive, you can tailor the user experience in mobile apps to individual users or user segments. For example, you can offer a unique experience based on the country the user is in.
Roll back/kill switch: What’s really unique about feature flags is that they enable teams to roll back any buggy updates quickly. By simply disabling the relevant feature flag, you can solve a bug without going through the lengthy app store review process.

Mobile feature flags: Use cases

We’ve talked mainly about how feature flags can be used in mobile app deployment but they’re also a great way to reduce risk when deploying and testing on mobile sites, especially when it comes to deep level modifications tied to back-end architecture such as testing new payment methods.

This can be easily done using a feature flagging platform, where teams can safely deploy frequent releases with an easy-to-use platform that can be used by all teams.

For example, let’s say you developed two payment features: one for desktop and one for mobile. Before doing a full release, you’re looking to test them on a small group of early adopters to monitor its impact and determine its usage rate.

Using AB Tasty, you can easily create a Feature Toggling use case on your AB Tasty account and choose the KPI you want to follow, in this case that would be the clicks on the “Proceed to checkout” button and then “conversion rate” as a sub KPI.

You can then define the two scenarios: one to enable the feature on desktop and another to enable it on mobile devices. You will then configure the flag that will turn on the new payment method for each scenario as seen in the image below in the “Scenario mobile” on the dashboard.

Next, let’s take a look at real-life examples of how AB Tasty clients use feature flags to carry out mobile testing:

Use case 1

Decathlon, a French sporting goods retailer with over 2,000 stores in 56 countries, wanted to test CTA placement to measure its impact across all devices, including mobile, and product listing pages (PLPs) with the help of feature flags.

In the original version, seen below, Decathlon APAC’s team was looking to test an earlier placement of the “add to cart” button on mobile on the main page below the product image to ensure a positive rollout and measure uplift. In the original version, users had to click on the product to go to the PDP before seeing this button.

With AB Tasty’s robust solution, the team was able to test the impact of this new feature on conversion. Changing the CTA placement proved to be a success, resulting in a 10.37% increase in transaction rate and an $11.27 increase in the average order value.

Use case 2

EDF (Electricité de France) is the largest electricity supplier in France for over 70 years. The team at EDF wanted to increase the number of online subscriptions and calls via their application.

In particular, they wanted to monitor the effect of changing the CTA design in the app. Using feature flags to increase the visibility of the CTAs, the team could then measure the impact on (and boost) clicks for online subscriptions and/or calls with EDF advisors respectively.

The team ran an A/B test with the subscription CTA against an orange background and the call CTA against a green background. They also added text to communicate hours of operation.

The call CTA was the one that yielded more positive results allowing the team to generate more qualified leads with an increase in calls with EDF advisors.

Thus, with a 20% increase in calls, the team could then confidently develop and roll out an adapted variation in the app where the new call CTA was more visible.

Use case 3

Often, A/B tests are a fool-proof way to eliminate potential bias and can save a company from investing in an optimization campaign that would otherwise take up a lot of valuable time and resources.

That was the case with Ornikar, a driving school platform in France with more than 2.5 million customers. The team was looking to revamp the home screen of their application and needed to identify which changes should be kept and which should be discarded.

The team set up an A/B test on AB Tasty to replace the existing carousel of four slides and two CTAs (left image) with a new screen featuring Ornikar benefits, a new CTA order and a more detailed carousel (right image).

The test was conducted for a duration of three weeks. After a week, they found that the new variation was not performing as wellas they expected so the team paused the test and adjusted the CTA and ran the test again for two weeks.

The results were still negative after two weeks and the team decided not to deploy the new home screen into production.

Due to the flexibility of the AB Tasty platform, the team was able to make quick iterations over a short period of time. Above all, Ornikar was able to avoid losing out on conversions and wasting time and resources as well as minimizing negative impact by testing the new home screen first before rolling it out to all its users.

Feature flags: The ultimate tool for better mobile experiences

As we’ve seen, feature flags are a powerful tool allowing teams across an organization to have more control over mobile testing and release while reducing risk.

Beyond giving you full control of new feature releases despite App and Play Store approval processes, feature flags enable teams to optimize their mobile apps and personalize the user experience. They also allow you to ship features more often and obtain quick feedback from your most relevant users.

With increasing mobile usage and millions of mobile apps to compete with, it’s essential to provide the very best user experience on mobile devices. Running experiments and using progressive rollouts with feature flags are key to delivering optimal and great mobile experiences.

Using a third-party feature flagging platform makes it easy to toggle features on and off and remotely configure your flags straight from the interface. By controlling all your feature flags in an easy to use web dashboard, it also ensures you’re keeping up with essential best practices to set you up for success and help you stand out from competitors.

You might also like...

See all

Article

6min read

Which Statistical Model is Best for A/B Testing: Bayesian, Frequentist, CUPED, or Sequential?

Emily Healy

Jul 15, 2025

Article

7min read

Is Your Average Order Value (AOV) Misleading You?

Hubert Wassner

Jul 11, 2025

Article

5min read

Why AB Tasty Delivers 4x Faster

Leo Wiel

Jul 7, 2025

Subscribe to
our Newsletter

Article

Mar 30, 2023

9min read

10 Best Practices for Shopping Cart Page Optimization

Anthony Brebion

Did you know that 70% of online shoppers abandon their carts?

According to the Baymard Institute, this number comes from an in-depth analysis that benchmarked 48 different studies on shopping cart abandonment rates.

This may not come as a surprise for experienced e-marketers, but the truth is that 7 people out of 10 will actually abandon their cart and end their shopping journey without making a purchase.

This is clearly a lot of lost revenue. So, how can you bring that percentage down?

This article aims to answer that question. We will cover best practices to help you diminish the abandonment rate and optimize your e-commerce shopping cart page for conversions.

Adopt clear UX parameters for shopping cart optimization

First of all, let’s start with a firm foundation. Your shopping cart page should adopt a clear, simple, and fast UX. This simple 3-step formula (CSF) is the cornerstone of any successful cart page:

Clear – There should be nothing messy, concerning, or misleading about your cart page. It should ideally display all the important information on a single page without the need to scroll too far or visit any other page.
Simple – Your cart page should display all the information using comprehensible, crystal-clear language and a design that leaves no room for misunderstanding.
Fast – The more time visitors spend on your cart page, the more likely they will leave it. If you apply the first two critical elements (simple+clear) to your page, the resulting cart page experience should also be fast.

As there are many elements on your page that you can optimize and run tests on to find the best solution, it’s important to follow this CSF framework for harmony.

Want to get started on A/B testing for your shopping cart page? AB Tasty is a great example of an A/B testing tool that allows you to quickly set up tests with low code implementation of front-end or UX changes on your web pages, gather insights via an ROI dashboard and determine which route will increase your revenue.

Knowing these 3 crucial elements, it’s high time we dive into our 10 best practices for e-commerce shopping cart pages.

10 best practices for your shopping cart pages

1. Create a detailed product summary

Just moments before your visitors proceed to checkout, they’ll land on your cart page which has one sole mission: lead your visitors to actually pay.

For most e-commerce buyers, the cart is a page used to review their order.

In order to help them do so, your mission is to clearly display all the relevant information regarding the product.

Below is a great example of a check-out page from ASOS that includes all the necessary details for a clear and easy review. Users know exactly which product they will purchase as well as the color, size, and quantity.

Detailed product summary on shopping cart page

When crafting your cart page, be sure to follow this example and include these elements:

Item thumbnail
Exact name
Item specifications (size, power, capacity, memory, features…)
Quantity of items in cart
Item color

Having all these elements shown to your customers allows them to quickly review their order and have confidence in their purchase.

Including all relevant details will decrease the percentage of cart abandonment that is typically caused by the lack of precise information.

2. Choose a clear, user-friendly color code

There have been many studies about the psychology behind colors. However, there’s no single answer on which color will fit all websites and solve all abandonment rate problems.

One thing that we do know for sure is that visitors love harmony and clear designs when it comes to UX.

Let’s look at the luxury shoe brand Jimmy Choo. For their check-out page, they kept a simple design by using a black button that clearly stands out, making it straightforward to click.

Clean color code on shopping cart page

Famous shoe-maker Jimmy Choo uses an elegant yet efficient black and white color code in order to clearly display the information on the cart page.

Notice that the checkout button efficiently stands out as the only black button on the page, making it extremely straightforward to click it.

3. Display explicit and detailed information about shipping and returns

What is the number one reason behind shopping cart abandonment in the US? Hidden shipping costs.
Online buyers despise hidden and last-minute shipping costs. As you can see in the image below from Baymard’s study, it’s by far the most popular reason for cart abandonment compared to others.

cart-abandonment-stats

Displaying transparent shipping and return policies is a key factor in enhancing conversion and gaining customers’ trust.

4. Craft clever information hierarchy and non-competing CTAs

Information hierarchy is the structure used to display and rank information according to its importance.

While designing cart pages, pay attention to the logic behind buttons, columns and titles as they will heavily influence the users’ perception.

You can use various colors in your CTAs (preferably matching your brand) although we recommend a maximum of 3-4 colors at a time.

Colors do help you gain visitors’ attention, so use them wisely:

Highlight important information
Use a distinct color to distinguish the CTA
Use lines or columns to structure your page

In the image below, Zappos, a USA-based shoe marketplace, does a great job of providing an efficient and clean shopping experience.

Shopping cart page with clear information hierarchy

The shopping cart page skillfully guides customers through the buying journey; we appreciate the neat layout emphasized by a simple 3-step color code (orange=very important; blue=important; grey=secondary).

5. Deploy payment options that your users love

Having different payment options is a necessity in today’s ultra-competitive e-commerce environment.

If you run an international e-commerce store, bear in mind that payment methods differ from one country to another: what’s used in North America isn’t necessarily the same as in Europe or in Asia.

To combat this, try to redirect customers based on their IP location to offer them a personalized experience based on the local currency and their preferred payment methods.

Various payment options on shopping cart page

In the image above, Adidas provides 6 different payment methods including Paypal, VISA and Apple Pay. This is an absolute necessity for large and global stores.

6. Show security seals and reassuring elements

In the same report from Baymard, the lack of trust in the payment accounted for 18% of abandoned cart rates. Trust seals, or trust badges, are very important to show your users that your site is legitimate.

In fact, they actually matter more than peer recommendations or trustworthy design. According to Baymard, here are the badges that give visitors the highest perception of a safe and secure site:

trust-seals-online-payments

7. Offer phone, chat or email assistance

Displaying a clear contact number and address details can impact your user’s level of trust. Shoppers want to make sure that your business is legitimate and not an online scam.

Furthermore, your visitors want to feel that there are actual humans behind your website.

Offering a live chat or phone assistance service right on the cart page is a great option to gain customers’ trust, legitimize your business and humanize your brand identity.

Wondering how to design your chatbox? Take a look at Victoria’s Secret’s page in the image below. They have chosen their most vibrant color, pink, to increase their call-to-action visibility.

Live chat integration on shopping cart

8. Add a “continue shopping” option

A “continue shopping” option is a smart way to offer your visitors a way to abandon their cart without leaving your website. They’ll have the freedom to continue browsing after they’ve already added items to their cart.

As some online shoppers actually use carts as “wishlists,” they can store items that they intend on purchasing later on.

9. Display legible thumbnail images

There’s nothing more annoying than a tiny thumbnail that barely helps to identify a product.

When customers review a product, you should give them the opportunity to see it correctly in a convenient size and resolution on both mobile and desktop.

Live chat integration on shopping cart

In the image above, Apple makes it very easy to distinguish the chosen product. The image used is bright and clear. Their customers will be absolutely certain that they’ve got the right item in their bag.

10. Push complementary products

Offering complementary product recommendations, or cross-selling, is nothing new in the e-commerce world.

However, displaying complementary products inside the shopping cart page is something worth testing if it could bring you a higher AOV.

For more testing ideas, check out our e-book: 50 Tests You Should Know For Website Optimization

Complementary products displayed on shopping cart page

Why optimize your shopping cart page?

An optimized shopping cart page is crucial to sales.

The shopping cart page is the last step your visitors take before their purchase. In this step of their digital user journey, it’s important to ease customer concerns in any way possible and promote a seamless checkout flow.

Every online e-commerce should be testing different elements of its cart page to find what works best for converting passive visitors into active customers.

Read more: Go one step further and improve your e-commerce product page performance to push your visitors to the cart page.

You might also like...

See all

Article

6min read

Which Statistical Model is Best for A/B Testing: Bayesian, Frequentist, CUPED, or Sequential?

Emily Healy

Jul 15, 2025

Article

7min read

Is Your Average Order Value (AOV) Misleading You?

Hubert Wassner

Jul 11, 2025

Article

5min read

Why AB Tasty Delivers 4x Faster

Leo Wiel

Jul 7, 2025

Subscribe to
our Newsletter

Article

Mar 20, 2023

8min read

Great Value Proposition Examples

Emily Healy

A value proposition is a clear statement of the fundamentals that define your brand.

To create a value proposition, you need to clearly define your product or service and showcase how you are different from competitors. This is often referred to as a Unique Selling Point (USP).

So, how can you define and present your company’s USP? By crafting a value proposition canvas.

A value proposition canvas is a visual display that maps out your product or service and how it can meet the specific needs of your customers.

It’s essentially an extension of the frequently-used business model canvas and has become a widely popular method of presentation in all corners of the business world.

In this article, we will dive into the specifics of a value proposition, give you tips for creating your own and advice on how to develop it further with a value proposition canvas.

What is a value proposition?

At first glance, a value proposition seems like a very simple and broad concept. However, a particularly effective value proposition is exceptionally complex and loaded with details.

Here are some common questions at the heart of each value proposition:

Why should a customer use your service or buy your product over a competitor’s?
What are you offering that no one else can?
How can you serve the needs of each individual customer?

A value proposition is not a positioning statement designed to appeal to as many customers as possible, rather it’s very specific. To meet the specific needs of a specific customer, you need to be as niche as possible.

For example, a business might be looking for software that comes with customer support packages that are more encompassing than usual due to the fact that they have little experience with a particular product.

Suppose a software company highlights its top-rated customer service team and affordable ongoing support options that their competitors are unable to provide. In that case, it will likely attract businesses that will profit from (and be in need of) this type of support.

Let’s look at another example: Uber – a travel service provider that meets low-cost and on-demand needs. Immediacy, convenience and simplicity are at the heart of their value proposition.

Uber value proposition

Think about the unique benefit that you are providing. If you can define this simply, concisely and accurately, you have a value proposition of worth to your audience.

How to create a value proposition

Now that we’ve defined what a value proposition is, let’s move on to how you can start brainstorming to transform your thoughts and ideas into a clear statement.

Before we get too intense, start by drafting up a few phrases with this template: We help (Customer) to (Goal) by (Unique value proposition).

We should also keep in mind that value propositions don’t need to be subtle. In fact, they can be rather blunt instruments used to draw in new customers or clients.

Value propositions should not be loaded with jargon or complexities. They should be thought of more like a well-researched advertising jingle than a dissertation.

Let’s discuss a few points to keep in mind when developing your value proposition:

Define your target audience
Know your competition
Tell your customers how you meet their needs
Be prepared to combat stereotypes

Define your target audience

As value propositions are highly dependent on those that are being sold to, it’s essential to define your target audience.

This requires a great deal of research, but will undoubtedly pay off in the long run. Age, demographics, marital status, economic status and personal interests can all inform how a value proposition is designed.

Your message needs to be clearly personalized for your audience.

Know the competition

It’s hard to stand out from the crowd if you have little concept of what the crowd is offering.

If a larger competitor offers the same USP (Unique Selling Points), it’s likely that they will get a hold of your potential customers since they will rely on your differences: brand size and reputation.

This requires extensive research to define new and fertile ground. You need to know your competition well – not only to be aware of your similarities but also to let your audience know how you stand out.

Tell your customer how you meet their needs

To start with, it’s best to define exactly what your product or service offers. Then, you can move into what makes it unique:

Customer service?
Affordability?
A simpler design than the competitors?

Once you have defined what you will highlight, it’s time to get more specific with your word choice. Remember: specificity does not mean complexity.

If the customer service is highly rated, why?
Affordability is great, but does the quality of the product match or even improve on your competitor?
Simplicity is admirable, but how does this improve customer satisfaction?

If you can define this in simple, but effective terms, you will have a highly effective value proposition.

Be prepared to combat stereotypes

Every sector has its own negative stereotype that can be harmful even before they get started.

Delivery services are known for not turning up when agreed upon, car salesmen are known for being notoriously pushy and fast food is known for being convenient but unhealthy. These are all examples of initial reactions, or cognitive biases, that most people have before using a product or service.

One of the most powerful kinds of value propositions are those that go against the grain of expectations.

If your particular sector is known for a negative practice, ensuring that your brand is above the stereotype can provide a concise and enticing positioning statement.

Value proposition canvas

A value proposition canvas is a focused way of structuring the main components of the value proposition in order to provide the most concise solutions for potential customers.

Value proposition canvas segments

The value proposition canvas is made up of two segments: the customer segment and the value proposition.

Inside each segment are three sections. The customer segment (shown below in red) has the specific needs of the customer, while the value proposition segment (shown in green) has the corresponding solutions.

value proposition canvas

The three sections of the customer segment (red) in the value proposition canvas are:

Customer jobs – what tasks can your customer accomplish by using your product or service?
Customer pains – what problems, risks or negative experiences can the customer eliminate when using your product or service?
Customer gains – what unique benefit does the customer gain when
using your product or service?

The value proposition map (green) on the other side of the canvas seeks to address these factors while presenting unique solutions and unexpected gains.

Gain creators – this is where you detail the unique improvements and benefits that your product or service offers to your customers
Pain relievers – this is the area to address elements that eliminate a current frustration or pain point (such as undesired costs, accessibility, or negative stereotypes)
Products and services – this is the spot to list all of your products and service

How to create a value proposition canvas

Let’s start with the customer segment.

The first thing to consider when creating a value proposition canvas is the customer. The customer is the central part of the entire premise. Be sure to speak carefully to your specific audience and avoid making generalizations.

Customer pains are generally more straightforward; therefore, pain relievers are often easier to define.

It’s tempting to focus solely on the functional factors, such as how to get from A to B. However, while the function is important, the emotional and social aspects also need deep consideration when developing your value proposition canvas.

Customer gains are a more complex element of the value proposition canvas as they can be slightly more difficult to define. In essence, these cannot be discovered without a substantial understanding of the uniqueness of the business itself. Not only will you list what the customer will gain from your product, but also what the customer will gain by choosing you over the competition.

Now let’s move on to the value proposition portion of the canvas.

Listing the potential jobs, or tasks, that customers want to accomplish with the help of your product or service is a good place to start.

Once completed, you will move on to the pain relievers and gains.

The pain relievers will specifically target the pains listed in the customer segment and should remain consistent. Meanwhile, the gain creators will show where your product or service adds value.

It’s important to remember that gains aren’t the opposite of pains. They are different factors that ultimately encourage consumers to adopt the product or service.

In the gains section, you will address the lesser-known desires of the customers – these are things that the customer themselves may not even be aware of. The lesser-known customer needs can be considered the “bonuses” that your brand offers potential customers.

Let’s go back to Uber as an example. Uber’s customers need quick, accessible and affordable transportation. These are the customer’s first thoughts. In addition, their customers also need safety during their rides. Uber offers insurance during trips to their drivers and passengers to cover them in the case of an accident. Furthermore, they can provide passengers with more comfort by including driver ratings from previous passengers. Customers may not immediately think of these factors, but they address very important human needs.

After development, each part of the right side of the canvas should be balanced with solutions and gains on the left. Ultimately, a value proposition canvas seeks perfect alignment.

Finding your fit in the market

Creating a value proposition canvas is a great tool to help understand your product, where you fit in the market, how you stand against competitors and the best way to market your brand successfully.

Drawing conclusions from this canvas and analyzing your value proposition can help you improve your strategies, messaging and overall product. A unique and profound value proposition is the core of a strong business.