Tag: EN

Article

Feb 5, 2024

5min read

Finding Our Better: AB Tasty’s New Brand Identity

Marylin Montoya

Cheers to Change

Good things happen to those who change. And that’s exactly what we did.

Change is what propels us towards progress.
Change is how we find our better.
Change is how we dare to go further.

Today marks a significant day in our history as a company. Today, we’re thrilled to share our updated brand identity with you. We’re stepping into a new era that better aligns our forever commitment to “test and learn” with our position in the market as a partner that helps brands push ideas even further.

With over 13 years in the industry, we’ve seen dynamic changes in the market. Brands now understand the importance and impact of continual experience optimization. The thriving experimentation sector has launched us into our most successful financial quarters following our strategic technology acquisitions. Beyond our strengthened AI and personalization portfolios, it’s become crystal clear that what makes us unique is our people. And our people are what make our customers happy.

Time to Talk Tasty

You may have noticed a few recent changes to AB Tasty – and we don’t mean just our new brand colors.

“Electric Blue” and “Crash Test Yellow”

Although our vibrant visual identity may catch you by surprise, our rebrand is much more than just a cosmetic makeover. We’ve been very intentional with our decisions at each step of the way.

Over the past 14 months, we’ve embraced some exciting technological advancements within our platform:

In October 2022, we saw a big need in the market for more personalization and acquired a company specializing in recommendations and search solutions.
In June 2023, we extended our personalization offering to help teams better cater to their different audiences and compete on a higher level. We acquired an emotions-based, personalization technology that enriches and expands our portfolio.
Then, we unified those platforms with our own API-based experimentation, personalization engine, and web solution.

Now, we’re happy to say that we are one unified platform offering everything that brands need for complete experience optimization. With our new brand identity, we proudly promote everything we are, everything we can be, and everything we want to be.

Our strategic shift in branding was the logical next step after our tremendous period of growth.

New Look, Same Commitment

One thing hasn’t changed – and that’s our commitment to our clients. They are, and always will be our focus.

Everything we’ve done will better suit the needs of our clients. Unifying our products into one harmonious platform allows for endless optimization opportunities and our our messaging reflects our human touch and leading expertise.

We are the optimization partners pushing brave ideas from the inside out.

Our Brand Story

Our clients need to be different, not just better. And for that, they need an optimization partner in their progress. Our commitment to customer support is consistently recognized on G2 and is something our clients rave about. Our team and the level of support we offer our clients have always been and will always be what makes AB Tasty great. That’s why we embed ourselves at the heart of company culture to push brave ideas from the inside out.

How can we do that? By focusing on our three pillars as our foundation.

Human Touch: Our people are everything – they bring the soul and substance to our technology. We build relationships with our clients to transcend the transactional with our deep partnerships and client understanding.
Leading Expertise: We back brave ideas with data and knowledge. We stay ahead as leaders of the industry and continue to learn with our “test and learn” culture. We make every move by choice, not chance by de-risking brave ideas.
Unifying Product: Our product connects teams, platforms, tools, and collaborators. We transform cultures changing the way our clients work and think. We work as a team with one vision and common goals.

We do all of this so our clients can level up. We make their next step our next challenge. Giving them the courage and push they need to dare to go further.

Conclusion

Every next step looks different for our clients, company, and people. That’s why we provide the courage and conviction to make it happen.

We help our customers DARE TO GO FURTHER.

You might also like...

See all

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Article

3min read

Tag V4: Elevating your Experience Building with an Improved Modification Engine

Julie Dumont

Feb 25, 2025

Article

2min read

New Visual Studio Code Extension: Dev-Friendly Experimentation & Personalization

Julie Dumont

Feb 21, 2025

Subscribe to
our Newsletter

Article

Feb 2, 2024

12min read

A/A Testing: What is it and When Should You Use it?

Hubert Wassner

A/A tests are a legacy from the early days of A/B testing. It’s basically creating an A/B test where two identical versions of a web page or element are tested against each other. Variation B is just a copy of A without any modification.

One of the goals of A/A tests is to check the effectiveness and accuracy of testing tools. The expectation is that, if no winner is declared, the test is a success. Whereas detecting a statistical difference would mean a failure, indicating a problem somewhere in the pipeline.

But it’s not always that simple. We’ll dive into this type of testing and the statistics and tech behind the scenes. We’ll look at why a failed A/A test is not a proof of pipeline failure, and that a successful A/A test isn’t a foolproof sanity check.

What is tested during an A/A test?

Why is there so much buzz around A/A testing? An A/A test can be a way to verify two components of an experimentation platform:

The statistical tool: It may be possible that the formulas chosen don’t fit the real nature of the data, or may contain bugs.
The traffic allocation: The split between variations must be random and respect the proportions it has been given. When a problem occurs, we talk about Sample Ratio Mismatch (SRM); that is, the observed traffic does not match the allocation setting. This means that the split has some bias impacting the analysis quality.
Let’s explore this in more detail.

Statistical tool test

Let’s talk about a “failed” A/A test

The most common idea behind A/A tests is that the statistical tool should yield no significant difference. It is considered a “failed” A/A test if you detect a difference in performance during an A/A test.

However, to understand how weak this conclusion is, you need to understand how statistical tests work. Let’s say that your significance threshold is 95%. This means that there is still a 5% chance that the difference you see is a statistical fluke and no real difference exists between the variations. So even with a perfectly working statistical tool, you still have one chance in twenty (1/20=5%) that you will have a “failed” A/A test and you might start looking for a problem that may not exist.

With that in mind, an acceptable statistical procedure would be to perform 20 A/A tests and expect to have 19 that yield no statistical difference, and one that does detect a significant difference. And even in this case, if 2 tests show significant results, it’s a sign of a real problem. In other words, having 1 successful A/A test is in fact not enough to validate a statistical tool. To validate it fully, you need to show that the tests are successful 95% of the time (=19/20).

Therefore, a meaningful approach would be to perform hundreds of A/A tests and expect ~5% of them to “fail”. It’s worth noting that if it “fails” less than 5% of the time it’s also a problem, maybe indicating that the statistical test simply says “no” too often, leading to a strategy that never detects any winning variation. So one A/A “failed” test doesn’t tell much in reality.

What if it’s a “successful A/A test”?

A “successful” A/A test (yielding no difference) is not proof that everything is working as it should. To understand why, you need to check another important tool in an A/B test: the sample size calculator.

In the following example, we see that from a 5% conversion rate, you need around 30k visitors per variation to reach the 95% significance level if a variation yields a 10% MDE (Minimal Detectable Effect).

But in the context of an A/A test, the Minimal Detectable Effect (MDE) is in fact 0%. Using the same formula, we’ll plug 0% as MDE.

At this point, you will discover that the form does not let you put a 0% here, so let’s try a very small number then. In this case, you get almost 300M visitors, as seen below.

In fact, to be confident that there is exactly no difference between two variations, you need an infinite number of visitors, which is why the form does not let you set 0% as MDE.

Therefore, a successful A/A test only tells you that the difference between the two variations is smaller than a given number but not that the two variations perform exactly the same.

This problem comes from another principle in statistical tests: the power.

The power of a test is the chance that you discover a difference if there is any. In the context of an A/A test, this refers to the chance you discover a statistically significant discrepancy between the two variations’ performance.

The more power, the more chance you will discover a difference. To raise the power of a test you simply raise the number of visitors.

You may have noticed that in the previous screenshots, tests are usually powered at 80%. This means that even if a difference exists between the variations in performance, 20% of the time you will miss it. So one “successful” A/A test (yielding no statistical difference) may just be an occurrence of this 20%. In other words, having just one successful A/A test doesn’t ensure the efficiency of your experimentation tool. You may have a problem and there is a 20% chance that you missed it. Additionally, reaching 100% of power will need an infinite number of visitors, making it impractical.

How do we make sure we can trust the statistical tool then? If you are using a platform that is used by thousands of other customers, chances are that the problem would have already been discovered.

Because statistical software does not change very often and it is not affected by the variation content (whereas the traffic allocation might change, as we will see later), the best option is to trust your provider, or you can double-check the results with an independent provider. You can find a lot of independent calculators on the web. They only need the number of visitors and the number of conversions for each variation to provide the results making it quick to implement.

Traffic allocation test

In this part, we only focus on traffic, not conversions.

The question is: does the splitting operation work as it should? We call this kind of failure a SRM or Sample Ratio Mismatch. You may ask yourself how a simple random choice could fail. In fact, the failure happens either before or after the random choice.

The following demonstrates two examples where that can happen:

The variation contains a bug that may crash some navigators. In this case, the corresponding variation will lose visitors. The bug might depend on the navigator and then you will end up with bias in your data.
If the variation gives a discount coupon (or any other advantage), and some users find a way to force their navigator to run the variation (to get the coupon), then you will have an excess of visitors for that variation that is not due to random chance, which results in biased data.

It’s hard to detect with the naked eye because the allocation is random, so you never get sharp numbers.

For instance, a 50/50 allocation never precisely splits the traffic in groups with the exact same size. As a result, we would need statistical tools to check if the split observed corresponds with the desired allocation.

SRM tests exist. They work more or less the same way as an A/B test except that the SRM formula indicates whether there is a difference between the desired allocation and what really happened. If there is indeed an SRM, then there is a chance that this difference is not due to pure randomness. This means that some data is lost or bias occurred during the experiment entailing trust for future (real) experiments.

On the one hand, detecting an SRM during an A/A test sounds like a good idea. On the other hand, if you think operationally it might not be that useful because the chance of a SRM is low.

Even if some reports say that they are more frequent than you may think, most of the time it happens on complex tests. In that sense, checking SRM within an A/A test will not help you to prevent having one on a more complex experiment later.

If you find a Sample Ration Mismatch on a real experiment or in an A/A test, the following actions remain the same: find the cause, fix it, and restart the experiment. So why waste time and traffic on an A/A test that will give you no information? A real experiment would have given you real information if it worked fine on the first try. If a problem does occur, we would detect it even in a real experiment since we only consider traffic and not conversions.

A/A tests are also unnecessary since most trustworthy A/B testing platforms (like AB Tasty) do SRM checks on an automated basis. So if an SRM occurs, you will be notified anyway.

So where does this “habit” of practicing A/A tests come from?

Over the years, it’s something that engineers building A/B testing platforms have done. It makes sense in this case because they can run a lot of automated experiments, and even simulate users if they don’t have enough at hand, performing a sound statistical approach to A/A tests.

They have reasons to doubt the platform in the works and they have the programming skills to automatically create hundreds of A/A tests to test it properly. Since these people can be seen as pioneers, their voice on the web is loud when they explain what an A/A test is and why it’s important (from an engineering perspective).

However, for a platform user/customer, the context is different as they’ve paid for a ready-to- use and trusted platform and can start a real experiment as soon as possible to get a return on investment. Therefore, it makes little sense to waste time and traffic on an A/A test that won’t provide any valuable information.

Why sometimes it might be better to skip A/A tests

We can conclude that a failed A/A test is not a problem and that a successful one is not proof of sanity.

In order to gain valuable insights from A/A tests, you would need to perform hundreds of them with an infinite number of visitors. Moreover, an efficient platform like AB Tasty does the corresponding checks for you.

That’s why, unless you are developing your own A/B testing platform, running an A/A test may not give you the insights you’re looking for. A/A tests require a considerable amount of time and traffic that could otherwise be used to conduct A/B tests that could give you valuable insights on how to optimize your user experience and increase conversions.

When it makes sense to run an A/A test

It may seem that running A/A tests may not be the right call after all. However, there may be a couple of reasons why it might still be useful to perform A/A tests.

First is when you want to check the data you are collecting and compare it to data already collected with other analytics tools but keep in mind that you will never get the exact same results. The reason is that most of the metric definitions vary on different tools. Nonetheless this comparison is an important onboarding step to ensure that the data is properly collected.

The other reason to perform an A/A test is to know the reference value for your main metrics so you can establish a baseline to analyze your future campaigns more accurately. For example, what is your base conversion rate and/or bounce rate? Which of these metrics need to be improved and are, therefore, a good candidate for your first real A/B test?

This is why AB Tasty has a feature that helps users build A/A tests dedicated to reach these goals and avoids the pitfalls of “old school” methods that are not useful anymore. With our new A/A test feature, A/A test data is collected in one variant (not two); let’s call this an “A test”.

This allows you to have a more accurate estimation of these important metrics as the more data you have, the more accurate the measurements are. Meanwhile, in a classic A/A test, data is collected in two different variants which provides less accurate estimates since you have less data for each variant.

With this approach, AB Tasty enables users to automatically set up A/A tests, which gives better insights than classic “handmade” A/A tests.

You might also like...

See all

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Article

3min read

Tag V4: Elevating your Experience Building with an Improved Modification Engine

Julie Dumont

Feb 25, 2025

Article

2min read

New Visual Studio Code Extension: Dev-Friendly Experimentation & Personalization

Julie Dumont

Feb 21, 2025

Subscribe to
our Newsletter

Article

Jan 9, 2024

8min read

How to Rebrand Your Site Using Experimentation in 5 Easy Steps

AB Tasty

We invited Holly Ingram from our partner REO Digital, an agency dedicated to customer experience, to talk us through the practical ways you can use experimentation when doing a website redesign.

Testing entire site redesigns at once is a huge risk. You can throw away years of incremental gains in UX and site performance if executed incorrectly. Not only do they commonly fail to achieve their goals, but they even fail to achieve parity with the old design. That’s why an incremental approach, where you can isolate changes and accurately measure their impact, is most commonly recommended. That being said, some scenarios warrant an entire redesign, in which case, you need a robust evidence-driven process to reduce this risk.

Step 1 – Generative research to inform your redesign

With the level of collaboration involved in a redesign, changes must be based on evidence over opinion. There’s usually a range of stakeholders who all have their own ideas about how the website should be improved and despite their best intentions, this process often leads to prioritizing what they feel is important, which doesn’t always align with customers goals. The first step in this process is to carry out research to see your site as your customers do and identify areas of struggle.

It’s important here to use a combination of quantitative research (to understand how your users behave) and qualitative research (to understand why). Start off broad using quantitative research to identify areas of the site that are performing the worst, looking for high drop-off rates and poor conversion. Now you have your areas of focus you can look at more granular metrics to gather more context on the points of friction.

Scroll maps: Are users missing key information as it’s placed below the fold?
Click maps: Where are people clicking? Where are they not clicking?
Traffic analysis: What traffic source(s) are driving users to that page? What is the split between new and returning?
Usability testing: What do users that fit your target audience think of these pages? What helps them? What doesn’t help?
Competitor analysis: How do your competitors present themselves? How do they tackle the same issues you face?

Each research method has its pros and cons. Keep in mind the hierarchy of evidence. The hierarchy is visually depicted as a pyramid, with the lowest-quality research methods (having the highest risk of bias) at the bottom of the pyramid and the highest-quality methods (with the lowest risk of bias) at the top. When reviewing your findings place more importance on findings that come from research methods at the top of the pyramid, e.g. previous A/B test findings, than research methods that come at the bottom (e.g. competitor analysis).

Step 2 – Prioritise areas that should be redesigned

Once you have gathered your data and prioritised your findings based on quality of evidence you should be able to see which areas you should focus on first. You should also have an idea of how you might want to improve them. This is where the fun part comes in, and you can start brainstorming ideas. Collaboration is key here to ensure a range of potential solutions are considered. Try and get the perspective of designers, developers, and key stakeholders. Not only will you discover more ideas, but you will also save time as everyone will have context on the changes.

It’s not only about design. A common mistake people make when doing a redesign is purely focussing on design and making the page look ‘prettier’, and not changing the content. Through research, you should have identified content that performs well and content that could do with an update. Make sure you consider this when brainstorming.

Step 3 – Pilot your redesign through a prototype

It can be tempting once you’ve come up with great ideas to go ahead and launch it. Even if you are certain this new page will perform miles better than the original, you’d be surprised how often you’re wrong. Before you go ahead and invest a lot of time and money into building your new page, it’s a good idea to get some outside opinions from your target audience. The quickest way to do this is to build a prototype and get users to feedback on it through user testing. See what their attention is drawn to, if there’s anything on the page they don’t like or think is missing. It’s much quicker to make these changes before launching than after.

Step 4 – A/B test your redesign to know with statistical certainty whether your redesign performs better

Now you have done all this work conducting research, defining problem statements, coming up with hypotheses, ideating solutions and getting feedback, you want to see if your solution actually works better!

However, do not make the mistake of jumping straight into launching on your website. Yes it will be quicker, but you will never be able to quantify the difference all of that work has made to your key metrics. You may see conversion rate increase, but how do you know that is due to the redesign and nothing else (e.g. a marketing campaign or special offer deployed around the same time)? Or worse, you see conversion rate decrease and automatically assume it must be down to the redesign when in fact it’s not.

With an A/B test you can rule out outside noise. For simplicity, imagine the scenario where you have launched your redesign, in reality it made no difference, but due to a successful marketing campaign around the same time you saw an increase in conversion rate. If you had launched your redesign as an A/B test, you would see no difference between the control and the variant, as both would have been equally affected by the marketing campaign.

This is why it is crucial you A/B test your redesign. Not only will you be able to quantify the difference your redesign has made, you will be able to tell whether that change is statistically significant. This means you will know the probability that the change you have seen is due to the test rather than random chance. This can help minimize the risk that redesigns often bring.

Once you have your results you can then choose whether you want to launch the redesign to 100% of users, which you can do through the testing tool whilst you wait for the changes to be hardcoded. As the redesign has already been built for the A/B test, hardcoding it should be a lot quicker!

Step 5 – Evaluative research to validate how your redesign performs

Research shouldn’t stop once the redesign has been launched. We recommend conducting post-launch analysis to evaluate how it performs over time. This especially helps measure metrics that have a longer lead time, such as returns or cancellations.

Redesigns are susceptible to visitor bias, as rolling out a completely different experience can be shocking and uncomfortable for your returning visitors. They are also susceptible to novelty effects, where users can react more positively just because something looks new and shiny. In either case, these effects will wear off with time. That’s why it’s important to monitor performance after it’s deployment.

Things to look out for:

Bounce rate
On-page metrics (scroll rate, click-throughs, heatmap, mouse tracking)
Conversion rate
Funnel progression
Difference in performance for new vs. returning users

Redesigns are all about preparation. It may seem thorough, but it should be with such a big change. If you follow the right process you could dramatically increase sales and conversions, but if done wrong you may have wasted some serious time, effort and money. Don’t skimp on the research and keep a user-centred approach and you could create a website your audience loves.

If you want to find out more about how a redesign worked with a real customer of AB Tasty’s and REO – take a look at this webinar where La Redoute details how they tested the new redesign of their site and sought continuous improvement.

You might also like...

See all

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Article

3min read

Tag V4: Elevating your Experience Building with an Improved Modification Engine

Julie Dumont

Feb 25, 2025

Article

2min read

New Visual Studio Code Extension: Dev-Friendly Experimentation & Personalization

Julie Dumont

Feb 21, 2025

Subscribe to
our Newsletter

Article

Dec 7, 2023

8min read

How to Better Handle Collateral Effects of Experimentation: Dynamic Allocation vs Sequential Testing

Hubert Wassner

When talking about web experimentation, the topics that often come up are learning and earning. However, it’s important to remember that a big part of experimentation is encountering risks and losses. Although losses can be a touchy topic, it’s important to talk about and destigmatize failed tests in experimentation because it encourages problem-solving, thinking outside of your comfort zone and finding ways to mitigate risk.

Therefore, we will take a look at the shortcomings of classic hypothesis testing and look into other options. Basic hypothesis testing follows a rigid protocol:

Creating the variation according to the hypothesis
Waiting a given amount of time
Analyzing the result
Decision-making (implementing the variant, keeping the original, or proposing a new variant)

This rigid protocol and simple approach to testing doesn’t say anything about how to handle losses. This raises the question of what happens if something goes wrong? Additionally, the classic statistical tools used for analysis are not meant to be used before the end of the experiment.

If we consider a very general rule of thumb, let’s say that out of every 10 experiments, 8 will be neutral (show no real difference), one will be positive, and one will be negative. Practicing classic hypothesis testing suggests that you just accept that as a collateral effect of the optimization process hoping to even it out in the long term. It may feel like crossing a street blindfolded.

For many, that may not cut it. Let’s take a look at two approaches that try to better handle this problem:

Dynamic allocation – also known as “Multi Armed Bandit” (MAB). This is where traffic allocation changes for each variation according to their performance, implicitly lowering the losses.
Sequential testing – a method that allows you to stop a test as soon as possible, given a risk aversion threshold.

These approaches are statistically sound but they come with their assumptions. We will go through their pros and cons within the context of web optimization.

First, we’ll look into the classic version of these two techniques and their properties and give tips on how to mitigate some of their problems and risks. Then, we’ll finish this article with some general advice on which techniques to use depending on the context of the experiment.

Dynamic allocation (DA)

Dynamic allocation’s main idea is to use statistical formulas that modify the amount of visitors exposed to a variation depending on the variation’s performance.

This means a poor-performing variation will end up having little traffic which can be seen as a way to save conversions while still searching for the best-performing variation. Formulas ensure the best compromise between avoiding loss and finding the real best-performing variation. However, this implies a lot of assumptions that are not always met and that make DA a risky option.

There are two main concerns, both of which are linked to the time aspect of the experimentation process:

The DA formula does not take time into account

If there is a noticeable delay between the variation exposure and the conversion, the algorithm may go wrong resulting in a visitor being considered a ‘failure’ until they convert. This means that the time between a visit and a conversion will be falsely counted as a failure.

As a result, the DA will use the wrong conversion information in its formula so that any variation gaining traffic will automatically see a (false) performance drop because it will detect a growing number of non-converting visitors. As a result, traffic to that variation will be reduced.

The reverse may also be true: a variation with decreasing traffic will no longer have any new visitors while existing visitors of this variation could eventually convert. In that sense, results would indicate a (false) rise in conversions even when there are no new visitors, which would be highly misleading.

DA gained popularity within the advertising industry where the delay between an ad exposure and its potential conversion (a click) is short. That’s why it works perfectly well in this context. The use of Dynamic Allocation in CRO must be done in a low conversion delay context only.

In other words, DA should only be used in scenarios where visitors convert quickly. It’s not recommended for e-commerce except for short-term campaigns such as flash sales or when there’s not enough traffic for a classic AB test. It can also be used if the conversion goal is clicking on an ad on a media website.

DA and the different days of the week

It’s very common to see different visitor behavior depending on the day of the week. Typically, customers may behave differently on weekends than during weekdays.

With DA, you may be sampling days unevenly, implicitly giving more weight on some days for some variations. However, you should weigh each day the same because, in reality, you have the same amount of weekdays. You should only use Dynamic Allocation if you know that the optimized KPI is not sensitive to fluctuations during the week.

The conclusion is that DA should be considered only when you expect too few total visitors for classic A/B testing. Another requirement is that the KPI under experimentation needs a very short conversion time and no dependence on the day of the week. Taking all this into account: Dynamic Allocation should not be used as a way to secure conversions.

Sequential Testing (ST)

Sequential Testing is when a specific statistical formula is used enabling you to stop an experiment. This will depend on the performance of variations with given guarantees on the risk of false positives.

The Sequential Testing approach is designed to secure conversions by stopping a variation as soon as its underperformance is statistically proven.

However, it still has some limitations. When it comes to effect size estimation, the effect size may be wrong in two senses:

Bad variations will be seen as worse than they really are. It’s not a problem in CRO because the false positive risk is still guaranteed. This means that in the worst-case scenario, you will discard not a strictly losing variation but maybe just an even one, which still makes sense in CRO.
Good variations will be seen as better than they really are. It may be a problem in CRO since not all winning variations are useful for business. The effect size estimation is key to business decision-making. This can easily be mitigated by using sequential testing to stop losing variations only. Winning variations, for their part, should be continued until the planned end of the experiment, ensuring both correct effect size estimation and an even sampling for each day of the week.
It’s important to note that not all CRO software use this hybrid approach. Most of them use ST to stop both winning and losing variations, which is wrong as we’ve just seen.

As we’ve seen, by stopping a losing variation in the middle of the week, there’s a risk you may be discarding a possible winning variation.

However, to actually have a winning variation after ST has shown that it’s underperforming, this variation will need to perform so well that it becomes even with the reference. Then, it would also have to perform so well that it outperforms the reference and all that would need to happen in a few days. This scenario is highly unlikely.

Therefore, it’s safe to stop a losing variation with Sequential Testing, even if all weekdays haven’t been evenly sampled.

The best of both worlds in CRO

Dynamic Allocation is the best approach to experimentation instead of static allocation when you expect a small volume of traffic. It should be used only in the context of ‘short delay KPI’ and with no known weekday effect (for example: flash sales). However, it’s not a way to mitigate risk in a CRO strategy.

To be able to run experiments with all the needed guarantees, you need a hybrid system using Sequential Testing to stop losing variations and a classic method to stop a winning variation. This method will allow you to have the best of both worlds.

You might also like...

See all

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Article

3min read

Tag V4: Elevating your Experience Building with an Improved Modification Engine

Julie Dumont

Feb 25, 2025

Article

2min read

New Visual Studio Code Extension: Dev-Friendly Experimentation & Personalization

Julie Dumont

Feb 21, 2025

Subscribe to
our Newsletter

Article

Nov 22, 2023

9min read

Harmony or Dissonance: Decoding Data Divergence Between AB Tasty and Google Analytics

Julie Dumont

The world of data collection has grown exponentially over the years, providing companies with crucial information to make informed decisions. However, within this complex ecosystem, a major challenge arises: data divergence.

Two analytics tools, even if they seem to be following the same guidelines, can at times produce different results. Why do they differ? How do you leverage both sets of data for your digital strategy?

In this article, we’ll use a concrete example of a user journey to illustrate differences in attribution between AB Tasty and Google Analytics. GA is a powerful tool for gathering and measuring data across the entire user journey. AB Tasty lets you easily make changes to your site and measure the impact on specific goals.

Navigating these differences in attribution strategies will explain why there can be different figures across different types of reports. Both are important to look at and which one you focus on will depend on your objectives:

Specific improvements in cross-session user experiences
Holistic analysis of user behavior

Let’s dive in!

Breaking it down with a simple use case

We’re going to base our analysis on a deliberately very basic use case, based on the user journey of a single visitor.

Campaign A is launched before the first session of the visitor and remains live until the end which occurs after the 3rd session of the visitor.

Here’s an example of the user journey we’ll be looking at in the rest of this article:

Session 1: first visit, Campaign A is not triggered (the visitor didn’t match all of the targeting conditions)
Session 2: second visit, Campaign A is triggered (the visitor matched all of the targeting conditions)
Session 3: third visit, no re-triggering of Campaign A which is still live, and the user carries out a transaction.

NB A visitor triggers a campaign as soon as they meet all the targeting conditions:

They meet the segmentation conditions

During their session, they visit at least one of the targeted pages

They meet the session trigger condition.

In A/B testing, a visitor exposed to a variation of a specific test will continue to see the same variation in future sessions, as long as the test campaign is live. This guarantees reliable measurement of potential changes in behavior across all sessions.

We will now describe how this user journey will be taken into account in the various AB Tasty and GA reports.

Analysis in AB Tasty

In AB Tasty, there is only one report and therefore only one attribution per campaign.

The user journey above will be reported as follows for Campaign A:

Total Users (Unique visitors) = 1, based on a unique user ID contained in a cookie; here there is only one user in our example.
Total Session = 2, s2 and s3, which are the sessions that took place during and after the display of Campaign A, are taken into account even if s3 didn’t re-trigger campaign A
Total Transaction = 1, the s3 transaction will be counted even if s3 has not re-triggered Campaign A.

In short, AB Tasty will collect and display in Campaign A reporting all the visitor’s sessions and events from the moment the visitor first triggered the campaign.

Analysis in Google Analytics

The classic way to analyze A/B test results in GA is to create an analysis segment and apply it to your reports.

However, this segment can be designed using 2 different methods, 2 different scopes, and depending on the scope chosen, the reports will not present the same data.

Method 1: On a user segment/user scope

Here we detail the user scope, which will include all user data corresponding to the segment settings.

In our case, the segment setup might look something like this:

This segment will therefore include all data from all sessions of all users who, at some point during the analysis date range, have received an event with the parameter event action = Campaign A.

We can then see in the GA report for our user journey example:

Total User = 1, based on a user ID contained in a cookie (like AB Tasty); here there is only one user in our example
Total Session = 3, s1, s2 and s3 which are the sessions created by the same user entering the segment and therefore includes all their sessions
Total Transaction = 1, transaction s3 will be counted as it took place in session s3 after the triggering of the campaign.

In short, in this scenario, Google Analytics will count and display all the sessions and events linked to this single visitor (over the selected date range), even those prior to the launch of Campaign A.

Method 2: On a session segment/session scope

The second segment scope detailed below is the session scope. This includes only the sessions that correspond to the settings.

In this second case, the segment setup could look like this:

This segment will include all data from sessions that have, at some point during the analysis date range, received an event with the parameter event action = Campaign A.

As you can see, this setting will include fewer sessions than the previous one.

In the context of our example:

Total User = 1, based on a user ID contained in a cookie (like AB Tasty), here there’s only one user in our example
Total Session = 1, only s2 triggers campaign A and therefore sends the campaign event
Total Transaction = 0, the s3 transaction took place in the s3 session, which does not trigger campaign A and therefore does not send an event, so it is not taken into account.

In short, in this case, Google Analytics will count and display all the sessions – and the events linked to these sessions – that triggered campaign A, and only these.

Attribution model

Tool – scope	Counted in the selected timeframe
AB Tasty	All sessions and events that took place after the visitor first triggered campaign A
Google Analytics – user scope	All sessions and events of a user that triggered campaign A at least once during one their sessions
Google Analytics – session scope	Only sessions that have triggered campaign A

Different attribution for different objectives

Depending on the different attributions of the various reports, we can observe different figures without the type of tracking being different.

The only metric that always remains constant is the sum of Users (Unique visitors in AB Tasty). This is calculated in a similar (but not identical) way between the 2 tools. It’s therefore the benchmark metric, and also the most reliable for detecting malfunctions between A/B testing tools and analytics tools with different calculations.

On the other hand, the attribution of sessions or events (e.g. a transaction) can be very different from one report to another. All the more so as it’s not possible in GA to recreate a report with an attribution model similar to that of AB Tasty.

Ultimately, A/B test performance analysis relies heavily on data attribution, and our exploration of the differences between AB Tasty and Google Analytics highlighted significant distinctions in the way these tools attribute user interactions. These divergences are the result of different designs and distinct objectives.

From campaign performance to holistic analysis: Which is the right solution for you?

AB Tasty, as a solution dedicated to the experimentation and optimization of user experiences, stands out for its more specialized approach to attribution. It offers a clear and specific view of A/B test performance, by grouping attribution data according to campaign objectives.

Making a modification on a platform and testing it aims to measure the impact of this modification on the performance of the platform and its metrics, during the current session and during future sessions of the same user.

On the other hand, Google Analytics focuses on the overall analysis of site activity. It’s a powerful tool for gathering data on the entire user journey, from traffic sources to conversions. However, its approach to attribution is broader, encompassing all session data, which can lead to different data cross-referencing and analysis than AB Tasty, as we have seen in our example.

It’s essential to note that one is not necessarily better than the other, but rather adapted to different needs.

Teams focusing on the targeted improvement of cross-session user experiences will find significant value in the attribution offered by AB Tasty.
On the other hand, Google Analytics remains indispensable for the holistic analysis of user behavior on a site.

The key to effective use of these solutions lies in understanding their differences in attribution, and the ability to exploit them in complementary ways. Ultimately, the choice will depend on the specific objectives of your analysis, and the alignment of these tools with your needs will determine the quality of your insights.

You might also like...

See all

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Article

3min read

Tag V4: Elevating your Experience Building with an Improved Modification Engine

Julie Dumont

Feb 25, 2025

Article

2min read

New Visual Studio Code Extension: Dev-Friendly Experimentation & Personalization

Julie Dumont

Feb 21, 2025

Subscribe to
our Newsletter

Article

Nov 7, 2023

6min read

Taking an Outcome-Driven Approach | Ruben de Boer

Rowan Haddad

Ruben de Boer explains what it takes to create a healthy testing environment that paves the way for better experimentation organization-wide

Ruben de Boer is a lead CRO Manager and consultant with over 14 years of experience in data and optimization. At Online Dialogue, Ruben leads the Conversion Managers team, developing team skills and quality as well as setting the team strategy and goals. He spreads his knowledge far both as a teacher with Udemy with over 12,000 students and as a public speaker on topics such as experimentation, change management, CRO and personal growth.

In 2019, Ruben founded his company, Conversion Ideas, where he helps people kick start their career in Conversion Rate Optimization and Experimentation by providing affordable, high-quality online courses and a number of resources.

AB Tasty’s VP Marketing Marylin Montoya spoke with Ruben about exciting trends and evolutions within the world of experimentation, including the various ways AI can impact the optimization of the experimentation process. Ruben also shares ways to involve cross-functional teams to implement a successful culture of experimentation within the organization and why it’s important to steer these teams towards an outcome- rather than an output-driven mindset.

Here are some key takeaways from their conversation.

The goal should always be outcome-driven

Based on his experience, Ruben believes that one of the biggest pitfalls companies face when trying to kick start their experimentation journey is they focus more on outputs rather than outcomes.

“When a company is still very much in an output mindset, meaning we have to deliver an X amount of sprint points per sprint and we have to release so many new features this year, then of course experimentation can be seen as something that slows it down, right? Let’s say as a rule of thumb, 25% of A/B tests or experiments result in a winner and so 75% of what was built will not be released, which means the manager does not get the output goals.”

In this scenario, experimentation becomes an obstacle that slows down these outputs. Whereas, when a company shifts towards an outcome mindset, it makes more sense to run experiments with the goal to create more value for the customer. With an outcome-mindset, teams embrace experimentation with customers at the heart of the process.

When teams are more outcome-oriented, the product is based more on research and experiments instead of a fixed long-term roadmap. According to Ruben, it’s vital that companies adopt such a way of working as it helps create better products and business outcomes, which ultimately helps them maintain their competitive advantage.

Importance of cross-functional teams

Ruben argues that experimentation is maturing in that it’s becoming more embedded within product teams.

He notes there’s a rising trend of different teams working together, which Ruben believes is essential for knowledge sharing when it comes to learning new things about the customer journey and the product itself. For Ruben, this helps create an ideal, healthy experimentation environment for teams to experiment better and get the results they want.

Ideally, there would be experts in experimentation coming in from different teams sharing knowledge, ideas and insights on a regular basis which helps drive inspiration and innovation when it comes to future test ideas.

The recipe behind the success of these experimentation teams varies and depends on the maturity of the experimentation program and the skills of these teams.

This could start with a look into the culture of the organization by sending questionnaires to various teams to understand their work process and how autonomous they are. This analysis would also help teams to understand what their current state of experimentation is like such as how accepting they are of experimentation. This helps to devise a strategy and roadmap to successfully implement a culture of experimentation throughout the whole organization.

This culture scan also helps determine the maturity of an experimentation program.

“Process, data, team, scope, alignment, and company culture: that’s what I generally look at when I assess the maturity of an organization. Is there a CRO specialist throughout the different product teams? How’s decision-making being done by leadership? Is it based on the HIPPO decisions or fully based on experimentation? Then there’s the outcome versus output mindset, the scope and alignment of experimentation as well as the structure of the team- is it just a single CRO specialist or a multidisciplinary team? What does the process look like? Is it just a single CRO process or is it a process embedded in a project team?” Ruben says.

A world of possibilities with AI

With the advent of AI technology, Ruben believes there’s a lot of possibilities with what can be done with it, particularly in the experimentation process.

While he admits it’s still too early to speculate and that there are also the many privacy concerns that come with such technology, he believes AI can bring a lot of exciting things in the future.

“It would be so nice to have an AI go over experiments on the product detail page with all the results and all the learnings, and just ask the AI, what did I actually learn and what would be good follow up experiments on that? And that would be enormously interesting to have an AI run through all the experiments in the database,” Ruben says.

Therefore, Ruben admits there are a number of possibilities of what teams can do when it comes to designing experiments and saving time and steps in the experimentation process.

“And just think about maybe three or four years from now, everyone will just have an AI app on their phone and say, I need to buy this and I will buy it for you. And maybe a website with only AI apps on it to purchase stuff, who knows? And then optimization becomes very different all of a sudden.”

There’s also significant potential with AI when it comes to changing the way people work as well as provide inspiration and ultimately optimize and bring innovation to the experimentation process.

“Maybe based on all the input we give from chat logs, social media channels, reviews, surveys, we can make the AI behave like a user at some point in the future somewhere, which you then don’t have to run user tests anymore because you just let AI see your website.”

What else can you learn from our conversation with Ruben de Boer?

Evolving trends in experimentation
His take on change management to help organizations adopt experimentation
His own experiences with building cross-functional teams
How to tackle resistance when it comes to experimentation

About Ruben de Boer

With over 14 years of experience as a lead CRO manager and consultant in data and optimization, Ruben is a two-time winner in the Experimentation Elite Awards 2023 and a best-selling instructor on Udemy with over 12,000 students. He is also a public speaker on topics such as experimentation culture, change management, conversion rate optimization, and personal growth. Today, Ruben is the Lead Conversion Manager responsible for leading the Conversion Managers team, developing team skills and quality, setting the team strategy and goals, and business development.

About 1,000 Experiments Club

The 1,000 Experiments Club is an AB Tasty-produced podcast hosted by Marylin Montoya, CMO at AB Tasty. Join Marylin and the Marketing team as they sit down with the most knowledgeable experts in the world of experimentation to uncover their insights on what it takes to build and run successful experimentation programs.

You might also like...

See all

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Article

3min read

Tag V4: Elevating your Experience Building with an Improved Modification Engine

Julie Dumont

Feb 25, 2025

Article

2min read

New Visual Studio Code Extension: Dev-Friendly Experimentation & Personalization

Julie Dumont

Feb 21, 2025

Subscribe to
our Newsletter

Article

Nov 6, 2023

10min read

Overcoming the Challenges of Customer Experience Optimization (EXO): Strategies and Tips

AB Tasty

The combination of intense competition and rapidly evolving technology requires businesses to prioritize customer experience optimization (EXO) to stay ahead.

The fact is, the cost of poor customer experience is high. According to a PWC survey, a third of consumers would stop using a brand they love after just one negative interaction.

In this article, we look at some common EXO challenges businesses face and strategies to overcome them, including practical insights for enhancing the digital customer experience. By implementing these strategies, you can ensure your business takes a customer-centric approach to optimizing the customer experience and building brand loyalty.

What is customer experience optimization?

Customer experience optimization refers to everything your business does to improve the customer’s experience at every touchpoint of their journey. It entails deeply understanding your customer’s needs and preferences and leveraging these insights to develop strategies to improve their interactions with your brand.

In today’s digital landscape, customers are flooded with choices across most categories of products and services. As a result, if you fail to deliver a positive experience, your customers will simply switch to a competing brand. EXO strategies are designed to keep customers satisfied and engaged, build brand loyalty, and reduce churn.

With EXO, it’s essential to deliver an experience that surpasses customers’ expectations and provides them with a seamless experience across all touchpoints and channels, including websites, mobile apps, social media accounts, and email.

Why customer experience optimization is important for business growth

First and foremost, EXO streamlines the customer’s path to purchase. Offering customers a frictionless, positive journey that makes it easy for them to get the information they need to make their purchase decision increases the likelihood of a successful transaction.

Customer EXO is also an ideal way to foster brand loyalty. Customers who have a superior experience with your brand are more likely to become repeat buyers. In fact, Deloitte research shows that a high-quality customer experience makes a customer 2.7 times more likely to keep buying from a business than a low-quality experience. Not only are customers likely to return, but they will also pay up to 16% more for an optimized experience, depending on the product category.

Positive experiences also trigger word-of-mouth recommendations, enhancing your brand’s reputation. Recommendations don’t entail the same acquisition costs as traditional marketing methods, making EXO a comparatively cost-effective way to boost sales and expand your customer base.

Challenges and solutions to customer experience optimization

We recognize there are challenges associated with EXO that may prevent you from delivering the best possible experience to your customers. Here are some strategies for tackling these challenges.

Compiling the right data for accurate measurements

Thanks to the various technologies available, we can now access a wealth of customer data. If interpreted and applied correctly, this data offers invaluable insights into the customer experience and ways of enhancing it. However, the sheer volume of these metrics can lead to information overload. It’s easy to get distracted or focus on the wrong metrics, including pitfall metrics that result in misinformed conclusions when considered in isolation. Some metrics, like cost of sale or cross-sell, don’t offer any meaningful insights into EXO.

The solution is to prioritize the metrics that matter. These include:

Customer satisfaction (CSAT)
Churn rates
Bounce rates
Customer retention rates
Trust ratings, conversion rates
Customer journey analytics
Repeat purchases
Customer segmentation
Buyer personas
Customer lifetime value (CLV)
Net Promoter Score (NPS)

Keep in mind that this data may reside in various departments across your organization, extending beyond sales, marketing and customer service teams. Consolidating this disparate data is essential to gaining a complete and accurate picture of customer experience in your organization.

Developing the right hypothesis

Experimentation is a powerful tool for delivering an optimal customer experience. However, randomly choosing hypotheses to test is a quick route to overlooking optimization opportunities. For example, simply changing the location of the checkout button in response to low conversion rates may not address the underlying issue.

Effective experimentation requires a considered approach to develop the correct hypothesis to test. The first step is identifying the genuine problem that needs addressing. You can then formulate a hypothesis to test to uncover the root cause of the issue and identify a concrete solution.

This second step requires a critical analysis of your current site and potential improvements from the customers’ perspective. Sourcing a range of data, including web analytics, user tests, and customer feedback, can help guide your analysis. You should also consider the psychology of the prospective customer. Getting in their mindset can guide you toward potential solutions.

If we continue with our checkout button example, the core issue may extend beyond conversion rates to a more specific concern: high cart abandonment rates. A hypothesis with a potential solution to this issue may be: “Many customers exit the checkout process at step 5. Reducing the number of steps in our checkout process will reduce cart abandonment rates.” Crafting the right hypothesis is a crucial step in optimizing customer experience.

Resource constraints

Ideally, businesses would have unlimited resources to optimize customer experiences. However, in reality, EXO usually competes with numerous other business priorities, all vying for time, human, and financial resources. Investing in the infrastructure and technology for EXO can be costly. Hiring and retaining people with the necessary skills to implement effective optimization strategies can also be challenging. Data availability is another common resource issue, especially for businesses with lower website traffic who feel they need more information for optimization.

The good news is you can tailor your approach to EXO to align with your business’s circumstances. This includes starting with smaller-scale initiatives and expanding your efforts as your optimization strategies gain traction or more resources become available. Another option is to outsource EXO by engaging the services of a specialist customer optimization agency.

It’s also important to note that high-volume website traffic isn’t a prerequisite for identifying and implementing effective EXO strategies. While a 95% confidence level is often cited as the magic number for drawing meaningful conclusions from your data, you can still optimize websites with less traffic by lowering the threshold. Focusing on optimizing the top of the funnel, where there may be greater opportunities for EXO, is another useful strategy for low-traffic websites.

ROI tunnel vision

When a company works on improving EXO, its main focus is often on immediate ROI in experimentation, sometimes at the expense of other important metrics. While the bottom line is relevant to any business strategy, focusing solely on the financial outcomes of EXO can lead to short-sighted decision-making, jeopardizing longer-term sustainability.

Prioritizing immediate revenue gain above all else can negatively impact the customer experience. It makes it almost impossible for an organization to adopt a customer-centric approach, a fundamental requirement for EXO.

Experimentation isn’t always neatly quantifiable. Experiments are typically run within complex contexts and are influenced by various factors. While measuring ROI may be a criterion when assessing the success of your EXO strategies, it should never be the primary or sole one. Instead, shift your focus to the broader impacts of experimentation, like its contribution to better, more informed decision-making.

Not knowing what your customers want

A customer-centric approach is vital to delivering an optimal customer experience. This requires an in-depth understanding of who your customers are, their needs and preferences, and precisely how they interact with your business. Without these insights, you’re in the dark about what your customers want and when they want it. Meeting—let alone exceeding—customers’ expectations is impossible.

Customer wants and needs are as diverse as your customer base. They may include a desire for higher levels of personalization, seamless online interactions, flexible payment methods, faster customer support, better pricing, transparency or increased mobile responsiveness. What customers want also evolves as their journey progresses. If your EXO strategies fail to align with your customers’ desires at the right time, they are unlikely to succeed.

While there are several ways to uncover customer needs and wants, one of the most effective methods is to go directly to the source. Collecting customer feedback at each stage of their journey—via surveys, feedback management systems, voice of customers, and user interviews—lets you tailor your EXO strategies and deliver the improvements your customers truly want.

Lack of customer experience optimization tools

Successful EXO relies on quality data for insights into your customers’ journeys, needs, and preferences. To achieve this, you need the right tools to capture and analyze accurate data in real-time across multiple channels.

These tools include:

CRM systems to track historical customer behavior and relationships
Customer feedback and survey software to collect individual feedback for deep insights into what your customers want
Behavior analytics tools to interpret your customers’ interactions and identify opportunities to improve their experience
Experience optimization platforms, like AB Tasty, to design and deliver digital omnichannel customer experiences via experimentation

It’s important to review the needs of your EXO strategy and the available tools to choose the ones that best align with your customers’ and business’s needs.

How to improve the digital customer experience

Observe user behavior patterns

A robust data foundation lets you observe and understand customer behavior individually and identify broader trends. This information serves as a compass, guiding your EXO efforts.

Customer insights may reveal common pain points. For example, a frequently searched term may highlight a topic customers want more information on. These insights also help you understand how users interact with your site, how that impacts their journey, and potential improvements. Is there a particular page where customers spend a lot of time? Do they have to navigate back and forth between pages to find the details they need?

Behavior patterns also reveal customer preferences, allowing you to personalize touchpoints within their journey and identify what triggers customers to complete their purchases. These insights serve as a powerful foundation for developing EXO strategies and hypotheses for A/B testing.

Create a journey map to understand the user flow

EXO involves optimizing every customer interaction with your business. A common pitfall to avoid when addressing EXO is approaching it narrowly from a specific touchpoint rather than considering the entire customer journey. A holistic approach delivers more impactful insights that help you manage the root causes of negative or neutral customer experiences.

A great way to understand your user flow and how it affects customer experience is to create a journey map, setting out every touchpoint during the buying process. Navigate your website like a potential customer, systematically stepping through the user journey and noting your findings.

Putting yourself in the customer’s shoes ensures you don’t overlook opportunities to optimize customer experience. This approach can also help you prioritize measures that make the user journey frictionless, improving customer experience and your site’s performance.

Develop a roadmap and set parameters to measure success

The list of available EXO measures is endless. Aligning your strategy with your business objectives requires a considered approach to implementation. To do this, develop a roadmap that outlines your goals, priorities and milestones.

A well-structured roadmap gives your team clear direction and deadlines while guiding decision-making to ensure the greatest impact on customer experience. Everyone understands their role, guaranteeing accountability in the execution of your EXO strategy. It also helps you prioritize initiatives and allocate the necessary resources, including EXO tools.

In your roadmap, you can list the specific metrics and KPIs to measure and track your progress. Doing this allows you to evaluate your EXO measures, readjust those not delivering results, and build on particularly effective ones.

Experiment and re-challenge your past experiments

You’re unlikely to unlock the secret to EXO in your organization on the first try. Instead, you’ll need to run continuous experiments using different hypotheses to find the right combination of strategies that work for your business.

The customer experience is dynamic and your EXO strategies should be equally adaptable. Continue to review your previous experiments to see what more you can learn from them, especially in terms of customer preferences. This process enables you to identify emerging opportunities for improvement and further refine the measures with the most impact to deliver an optimal customer experience.

Customer-centric EXO

Acknowledging that your business must prioritize customer EXO is just the beginning. By understanding the customer experience definition, common EXO challenges, and practical strategies to overcome them, you have the tools to deliver a consistently superior customer experience. By integrating a customer-centric ethos with your EXO strategies, you’ll not only strengthen current customer relationships but also cultivate enduring brand loyalty.

You might also like...

See all

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Article

3min read

Tag V4: Elevating your Experience Building with an Improved Modification Engine

Julie Dumont

Feb 25, 2025

Article

2min read

New Visual Studio Code Extension: Dev-Friendly Experimentation & Personalization

Julie Dumont

Feb 21, 2025

Subscribe to
our Newsletter

Article

Oct 24, 2023

10min read

How to Effectively Use Sequential Testing

Hubert Wassner

In A/B tests where you can see the data coming in a continuous stream, it’s tempting to stop the experiment before the planned end. It’s so tempting that in fact a lot of practitioners don’t even really know why one has to define a testing period beforehand.

Some platforms have even changed their statistical tools to take this into account and have switched to sequential testing which is designed to handle tests this way.

Sequential testing enables you to evaluate data as it’s collected to determine if an early decision can be made, helping you cut down on A/B test duration as you can ‘peak’ at set points.

But, is this an efficient and beneficial type of testing? Spoiler: yes and no, depending on the way you use it.

Why do we need to wait for the predetermined end of the experiment?

Planning and respecting the data collection period of an experiment is crucial. Historical techniques use “fixed horizon testing” that establishes these guidelines for all to follow. If you do not respect this condition, then you don’t have the guarantee provided by the statistical framework. This statistical framework guarantees that you only have a 5% error risk when using the common decision thresholds.

Sequential testing promises that when using the proper statistical formulas, you can stop an experiment as soon as the decision threshold is crossed and still have the 5% error risk guarantee. The test user here is the sequential Z-test, which is based on the classical Z-test with an added correction to take the sequential usage into account.

In the following sections, we will look at two objections that are often raised when it comes to sequential testing that may put it at odds with CRO practices.

Sequential testing objection 1: “Each day has to be sampled the same”

The first objection is that one should sample each day of the week the same way. This is basically to have a sampling that represents reality. This is the case in a classic A/B test. However, this rule may be broken if you use sequential testing since you can stop the test mid-week but this is not always applicable. Since in reality there are seven different days, your sampling unit should be by week and not by day to account for behavioral differences over the course of a week.

As experiments typically last 2-3 weeks, then the promise of sequential testing saving days isn’t necessarily correct unless a winner appears very early in the process. However, it’s more likely that the statistical test yielded significance during the last week. In this case, it’s best to complete the data collection until each day is sampled evenly so that the full period is covered.

Let’s consider the following simulation setting:

One reference with a 5% conversion rate
One variation with a 5.5% conversion rate (a 10% relative improvement)
5,000 visitors as daily traffic
14 days (2 weeks) of data collection
We ran thousands of such experiments to get histograms for different decision index

In the following histogram, the horizontal axis is the day when the sequential testing crosses the significance threshold. The vertical axis is the ratio of experiments which stopped on this day.

In this setting, day 10 is the most likely day for the sequential testing to reach significance. This means that you will need to wait until the planned end of the test to respect the “same sampling each day” rule. And it’s very unlikely that you will get a significant positive result in one week. Thus, in practice, determining the winner sooner with sequential testing doesn’t apply in CRO.

Sequential testing objection 2: “Yes, (effect) size does matter”

In sequential testing, this is often a less obvious problem and may need some further clarification to be properly understood.

In CRO, we consider mainly two statistical indices for decision-making:

The pValue or any other confidence index, which is linked to the fact that there exists (or not) a difference between the original and the variation. This index is used to validate or invalidate the test hypothesis. But a validated hypothesis is not necessarily a good business decision, so we need more information.
The Confidence Interval (CI) around the estimated gain, which indicates the size of the effect. It’s also central to business decisions. For instance, a variation can be a clear winner but with a very little margin that may not cover the implementation or operating costs such as coupon offerings that need to cover the coupon cost.

Confidence intervals can be seen as a best and worst case scenario. For example a CI = [1% ; 12%] means “in the worst case you will only get 1% relative uplift,” which means going from 5% conversion rate to 5.05%.

If the variation has an implementation or operating cost, the results may not be satisfying. In that case, the solution would be to collect more data in order to have a narrower confidence interval, until you get a more satisfying lower bound, or you may find that the upper bound goes very low showing that the effect, even if it exists, is too low to be worth it from a business perspective.

Using the same scenario as above, the lower bound of the confidence interval can be plotted as follows:

Horizontal axis – the percentage value of the lower bound
Vertical axis – the proportion of experiments with this lower bound value
Blue curve – sequential testing CI
Orange curve – classical fixed horizon testing

We can see that sequential testing has a very low confidence interval for the lower bound. Most of the time, this is lower than 2% (in relative gain, which is very small). This means that you will get very poor information for business decisions.

Meanwhile, a classic fixed horizon testing (orange curve) will produce a lower bound >5% in half of the cases, which is a more comfortable margin. Therefore, you can continue the data collection until you have a useful result, which means waiting for more data. Even if by chance the sequential testing found a variant reaching significance in one week, you will still need to collect data for another week to do two things: have a useful estimation of the uplift and sample each day equally.

This makes sense in light of the purpose of sequential testing: quickly detect when a variation produces results that differ from the original, whether for the worse or better.

If done as soon as possible, it makes sense to stop the experiment as soon as the gain confidence interval lays mostly either on the positive or negative side. Then, for the positive side, the CI lower bound is close to 0, which doesn’t allow for efficient business decisions. It’s worth noting that for other applications other than CRO, this behaviour may be optimal and that’s why sequential testing exists.

When does sequential testing in CRO make sense?

As we’ve seen, sequential testing should not be used to quickly determine a winning variation. However, it can be useful in CRO in order to detect losing variations as soon as possible (and prevent loss of conversions, revenue, …).

You may be wondering why it’s acceptable to stop an experiment midway through when your variation is losing rather than when you have a winning variation. This is because of the following reasons:

The most obvious one: To put it simply, you’re losing conversions. This is acceptable in the context of searching for a better variation than the original. However, this makes little sense in cases where there is a notable loss, indicating that the variation has no more chances to be a winner. An alerting system set at a low sensitivity level will help detect such impactful losses.
The less obvious one: Sometimes when an experiment is only slightly “losing” for a good period of time, practitioners tend to let this kind of test run in the hopes that it may turn into a “winner”. Thus, they accept this loss because the variation is only “slightly” losing but they often forget that another valuable component is lost in the process: traffic, which is essential for experimentation. For an optimal CRO strategy, one needs to take these factors into account and consider stopping this kind of useless experiment, doomed to have small effects. In such a scenario, an automated alert system will suggest stopping this kind of test and allocate this traffic to other experiments.

Therefore, sequential testing is, in fact, a valuable tool to alert and stop a losing variation.

However, one more objection could still be raised: by stopping the experiment midway, you are breaking the “sample each day the same” rule.

In this particular case, stopping a losing variation has very little chance to be a bad move. In order for the detected variation to become a winner, it first needs to gain enough conversions tobe comparable to the original version. Then it would need another set of conversions to be a “mild” winner and that still wouldn’t be enough to be considered a business winner (and cover the implementation or exploitation costs of that winner). To be considered a winner for your business, the competing variation will need another high amount of conversions with a sufficient margin. This margin needs to be high enough to cover the cost of implementation, localization, and/or operating costs.

All the aforementioned events should happen in less than a week (ie. the number of days needed to complete the current week). This is very unlikely, which means it’s safe and smart to stop such experiments.

Conclusion

It may be surprising or disappointing to see that there’s no business value in stopping winning experiments early as others may believe. This is because a statistical winner is not a business winner. Stopping a test early is taking away the data you need to reach a significant effect size that would increase your chances of getting a winning variation.

With that in mind, the best way to use this type of testing is as an alert to help spot and stop tests that are either harmful to the business or not worth continuing.

About the Author:

Hubert Wassner has been working as a Senior Data Scientist at AB Tasty since 2014. With a passion for science, data and technology, his work has focused primarily on all the statistical aspects of the platform, which includes building Bayesian statistical tests adapted to the practice of A/B testing for the web and setting up a data science team for machine learning needs.

After getting his degree in Computer Science with a speciality in Signal Processing at ESIEA, Hubert started his career as a research engineer doing research work in the field of voice recognition in Switzerland followed by research in the field of genomic data mining at a biotech company. He was also a professor at ESIEA engineer school where he taught courses in algorithmics and machine learning.

You might also like...

See all

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Article

3min read

Tag V4: Elevating your Experience Building with an Improved Modification Engine

Julie Dumont

Feb 25, 2025

Article

2min read

New Visual Studio Code Extension: Dev-Friendly Experimentation & Personalization

Julie Dumont

Feb 21, 2025

Subscribe to
our Newsletter

Article

Oct 17, 2023

4min read

Doing CRO at Scale? AB Tasty’s Testing Alerts Prevent Revenue Loss

Hubert Wassner

When talking about CRO at scale, a common fear is, what if some experiments go wrong? And we mean seriously wrong. The more experiments you have running, the higher the chance that one could negatively impact the business. It might be a bug in a variant, or it might just be a bad idea – but conversion can drop dramatically, hurting your revenue goals.

That’s why it makes sense to monitor experiments daily even if the experiment protocol says that one shouldn’t make decisions before the planned end of the experiment. This leads to two questions:

How do you make a good decision using statistical indices that are not meant to be used that way? Checking it daily is known as “P-hacking” or “data peeking” and generates a lot of poor decisions.
How do you check tens of experiments daily without diverting the team’s time and efforts?

The solution is to use another statistical framework, adapted to the sequential nature of checking an experiment daily. This is known as sequential testing.

How do I go about checking my experiments daily?

Good news, you don’t have to do this tedious job yourself. AB Tasty has developed an automatic alerting system, based on a specialized statistical test procedure, to help you launch experiments at scale with confidence.

Sequential Testing Alerts offer a sensitivity threshold to detect underperforming variations early. It triggers alerts and can pause experiments, helping you cover your losses. It enables you to make data-driven decisions swiftly and optimize user experiences effectively. This alerting system is based on the percentage of conversions and/or traffic lost.

When creating a test, you simply check a box, select the sensitivity level and you’re done for the rest of the experiment’s life. Then, if no alert has been received, you can go ahead and analyze your experiment as usual; but in case of a problem, you’ll be alerted through the notification center. Then you head on over to the report page to assess what happened and make a final decision about the experiment.

Can I also use this to identify winners faster?

No; this may sound weird but it’s not a good idea, even if some vendors are telling you that you should do it. You may be able to spot a winner faster but the estimation of the effect size will be very poor, preventing you from making wise business decisions.

But what about “dynamic allocation”?

First of all, what is dynamic allocation? As its name suggests, it is an experimentation method where the allocation of traffic is not fixed but depends on the variation’s performance. While this may be seen as a way to limit the effect of bad variations, we don’t recommend using it for this purpose for several reasons:

As you know, the strict testing protocol strongly suggests not to change the allocation during a test, so dynamic allocation is an edge case of A/B testing. We only suggest using it when you have no other choice. Classic use cases are very low traffic, or very short time frames (flash sales, for instance). So if your only concern is avoiding big losses from a bad variation, alerting is a better and safer option that respects the A/B testing protocol.
Since dynamic allocation changes the allocation, in order to avoid the risk of missing out on a winner, it always keeps a minimum traffic amount for the supposed losing variation, even if it’s losing by a lot.

Therefore, dynamic allocation isn’t a way to protect revenue but rather to explore options in the context of limited traffic.

Other benefits

Another benefit of the sequential testing alert feature is better usage of your traffic. Experiments with a lot of traffic with no chance to win are also detected by the alerting system. The system will suggest to the user to stop that experiment and make better use of the traffic for other experiments.

The bottom line is: you can’t do CRO at scale without an automated alert system. Receiving such crucial alerts can protect you from experiments that could otherwise cause revenue loss. To learn more about our sequential testing alert and how it works, refer to our documentation.

You might also like...

See all

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Article

3min read

Tag V4: Elevating your Experience Building with an Improved Modification Engine

Julie Dumont

Feb 25, 2025

Article

2min read

New Visual Studio Code Extension: Dev-Friendly Experimentation & Personalization

Julie Dumont

Feb 21, 2025

Subscribe to
our Newsletter

Article

Oct 13, 2023

9min read

Best Practice for Optimizing Mobile vs. Desktop Experiences

AB Tasty

Best Practice for Optimizing Mobile vs Desktop Experiences

For product and web teams, the prospect of mobile is both wonderful and terrifying. The wonderful: your product or website is available all the time! The terrifying: you now have to create and develop two (or three, or four) user experiences, all at once.

While your customers probably walk around with their phones all the time, they may also spend the majority of their workday on a desktop computer. They might browse your mobile website or app during sporadic free moments throughout the day, then finish their customer journey later, when they’re not limited by small screen sizes and thumb-based typing.

The bottom line? Your entire product experience needs to be seamless, and combined across all platforms and devices. When you’re building an app, chances are you’re building a desktop version for web browsers, and at least two versions for mobile: iOS and Android. Given that people might also use the app on their phone’s web browser—a different experience than on a desktop but not quite the same as a native phone app—it’s easy to wonder where to start with the design process!

Below we’ve outlined some key tips for doing just that.

Plan an Entire Digital Experience

Regardless, your designs across both desktop and mobile have two requirements right out of the gate: they must be intuitive, and they must be accessible for people with disabilities.

You’ll also want to consider the fact that designing apps with a “mobile first” mindset is now a standard, because 1) users will expect to be able to access your digital ecosystem from their phones, and 2) web apps that are optimized for mobile rank higher on Google, which can lead to more conversions.

Before you start designing your app, make sure you plan to accommodate the key differences between desktop and mobile:

Both desktop and mobile experiences are crucial for customer conversion and retention, but often in different ways and in different circumstances. Below, we’ll take a look at best practices for keeping your mobile and desktop experiences updated and healthy for your sales funnel.

Best Practice for the Mobile Digital Experience

Whether your mobile experience exists within a dedicated iOS or Android app or simply as a responsive web-based app, there are some key best practices to be aware of when building your design strategy to maximize conversions and retention.

1. Keep it accessible, always

If your website or app is used by the public, it legally must comply with accessibility standards. These include:

Text size options
Large, easy-to-use buttons and CTAs
Minimum color contrast ratios
Simple touchscreen gestures
Adjustable screen orientation
Compatibility with screen readers
Content translation ability

2. Make sure your mobile content is consistent with desktop

If your users regularly hop between their computer and phones, it’s important that they can easily and consistently find what they need. This can include information, tools, actions, products, and more. If they’re available on desktop, they should also be available on mobile!

For example, if you have a menu on your desktop app that offers users different ways to contact customer service but it’s missing on mobile, it will likely cause frustration when users are trying to get help on the go.

3. Consider your mobile users’ needs

Ask yourself: What data, tools, or functions will your users be accessing? Make everything easy to find, navigate, and use. This might start with an intuitive navigation system that lets users easily:

Make purchases
Find specific tools they love to use
Manage their account or subscription
Get help from your support team

You’ll also want to consider what your different users will be trying to do and from where. Will they be using cellular data (slow) or wi-fi (fast)? Will they be reserving certain actions for mobile vs. desktop? Will tasks that may have started on a processor-heavy desktop app later migrate to mobile (or vice versa)?

For example, a video editor might need to use a desktop app to edit terabytes’ worth of cinema-quality video before switching to mobile to share the final piece on social media. Personal workflows might vary, but it all gets down to having the right tools on hand at all times.

4. Check how your site and app perform across different devices

You’ve built your iOS app. You’ve built your Android app. Your responsive website’s parameters are set so they’ll adjust responsively for users accessing from phones and tablets.

Now comes the testing. App Store or Google Play installations should work seamlessly. Each button, form, menu, and action must work flawlessly. Load times should be fast, and there should be little to no latency between screens (though this can vary with web-based apps, depending on how they’re constructed).

For responsive sites, each element will have CSS parameters that dictate how it will appear on a phone, tablet, or desktop computer. You don’t need separate versions of your website, but each element must be tagged correctly so it renders properly on each device.

5. Make UX choices that encourage conversion and retention

Good UX (user experience) comes from good user research and robust product analytics, but it’s customer conversion and retention that will keep your app alive. If users buy items or services through your app, you’ll want to keep these top UX tips in mind:

Never hide menus and navigation options.
Make search bars easy to find.
Make product browsing simple.
Choose familiar, suitable icons for cart checkout processes.
Label all icons, buttons, and navigation items.
Make checkout funnels easy (including credit card selection/adding), or incorporate simple payment options like Apple Pay or Google Pay.
Consider adding a subscription option so users can set it once and forget it!

Don’t forget that there are product analytics tools available to analyze the particular actions your users take, including Session Replays. These help you identify which parts of your UX design are working and which parts are causing friction.

Best Practice for Desktop Optimization

Desktop apps (many of which are web-based) are also commonly used and have more screen real estate to work with. Here are five tips to optimize your desktop experience to convert (and retain) new customers:

1. Create an engaging homepage

A great UX on your homepage can make or break conversion rates. When you’ve spent time and resources optimizing SEO, you’ll want to give potential customers a good first impression with a strong design. Here’s what we recommend to maximize conversion and retention:

Keep your design simple with a single, clear CTA.
Make sure it meets accessibility standards.
Use a catchy headline and simple buzzwords to make your value proposition clear and concise.
Choose the right colors by employing and understanding color psychology.
Differentiate hyperlinked text with an accent color.
Embrace white space! Space around your text can keep people reading it.
Use well-lit, professional images and videos where they fit organically.
Again, make sure your site is responsive to serve mobile users!

2. Improve website speed

Nothing deflects a website visitor faster than a glacially slow load time. To avoid losing possible customers, you can:

Optimize your images by saving compressed versions at 72dpi.
Limit the number of assets needed to load your page correctly—this can help reduce HTTP requests, which speeds up load times.
Limit the use of external scripts.
Remove unnecessary text, white space, and comments from CSS and Javascript files.
Use browser HTTP caching—this can offer faster load times for users who frequently visit your site.

3. Develop intuitive site navigation

Site navigation—typically in the form of a top menu, side-bar menu, or organized icons—shows users what to do once they reach your site. Use descriptive labels for each item, and simplify drop-down menu structures so users won’t get overwhelmed.

4. Make it easy to search for terms

Adding a search bar to your top menu can be a great way to give users a way to search your site for any term (or action) they want. This gives them a shortcut to find exactly what they need!

5. Ready your desktop site for final steps of conversion and retention

Often, consumers are more likely to say that desktop offers a more “convenient” shopping experience than mobile, and a majority of people complete purchases on a desktop.

As such, desktop websites are still used and still valuable, particularly for final conversation and ongoing retention. Whether it’s because customers are making complex purchases or simply seeing your checkout funnel more clearly on desktop, it’s essential to make sure every step of your experience is seamless.

Once you’ve converted customers, it can also help to keep track of their preferences so you can personalize their experience going forward.

Gain Valuable Insights with Heap

This is a guest blog written by Ben Lempert, Head of Content and Web at Heap.

Heap And AB Tasty provide businesses with the ability to collect and analyze user data efficiently, leading to more informed decisions and higher conversion rates. This dynamic combination offers a comprehensive solution for optimizing web experiences and increasing revenue through data-driven strategies.

Curious how you can find out what your users are doing across both desktop and mobile? Heap offers tools to help you track, measure, and ultimately optimize every touchpoint of users’ journeys across your entire digital experience.

You might also like...

See all

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Article

3min read

Tag V4: Elevating your Experience Building with an Improved Modification Engine

Julie Dumont

Feb 25, 2025

Article

2min read

New Visual Studio Code Extension: Dev-Friendly Experimentation & Personalization

Julie Dumont

Feb 21, 2025

Cheers to Change

Time to Talk Tasty

New Look, Same Commitment

Our Brand Story

Conclusion

You might also like...

What is tested during an A/A test?

Statistical tool test

Let’s talk about a “failed” A/A test

What if it’s a “successful A/A test”?

Traffic allocation test

So where does this “habit” of practicing A/A tests come from?

Why sometimes it might be better to skip A/A tests

When it makes sense to run an A/A test

You might also like...

Step 1 – Generative research to inform your redesign

Step 2 – Prioritise areas that should be redesigned

Step 3 – Pilot your redesign through a prototype

Step 4 – A/B test your redesign to know with statistical certainty whether your redesign performs better

Step 5 – Evaluative research to validate how your redesign performs

You might also like...

Dynamic allocation (DA)

Sequential Testing (ST)

The best of both worlds in CRO

You might also like...

Breaking it down with a simple use case

Analysis in AB Tasty

Analysis in Google Analytics

Method 1: On a user segment/user scope

Method 2: On a session segment/session scope

Attribution model

Different attribution for different objectives

From campaign performance to holistic analysis: Which is the right solution for you?

You might also like...

Ruben de Boer explains what it takes to create a healthy testing environment that paves the way for better experimentation organization-wide

The goal should always be outcome-driven

Importance of cross-functional teams

A world of possibilities with AI

What else can you learn from our conversation with Ruben de Boer?

About Ruben de Boer

About 1,000 Experiments Club

You might also like...

What is customer experience optimization?

Why customer experience optimization is important for business growth

Challenges and solutions to customer experience optimization

Compiling the right data for accurate measurements

Developing the right hypothesis

Resource constraints

ROI tunnel vision

Not knowing what your customers want

Lack of customer experience optimization tools

How to improve the digital customer experience

Observe user behavior patterns

Create a journey map to understand the user flow

Develop a roadmap and set parameters to measure success

Experiment and re-challenge your past experiments

Customer-centric EXO

You might also like...

Why do we need to wait for the predetermined end of the experiment?

Sequential testing objection 1: “Each day has to be sampled the same”

Sequential testing objection 2: “Yes, (effect) size does matter”

When does sequential testing in CRO make sense?

Conclusion

You might also like...

How do I go about checking my experiments daily?

Can I also use this to identify winners faster?

But what about “dynamic allocation”?

Other benefits

You might also like...

Best Practice for Optimizing Mobile vs Desktop Experiences

Plan an Entire Digital Experience

Best Practice for the Mobile Digital Experience

1. Keep it accessible, always

2. Make sure your mobile content is consistent with desktop

3. Consider your mobile users’ needs

4. Check how your site and app perform across different devices

5. Make UX choices that encourage conversion and retention

Best Practice for Desktop Optimization

1. Create an engaging homepage

3. Develop intuitive site navigation