Tag: EN

Article

Nov 22, 2023

9min read

Harmony or Dissonance: Decoding Data Divergence Between AB Tasty and Google Analytics

Julie Dumont

The world of data collection has grown exponentially over the years, providing companies with crucial information to make informed decisions. However, within this complex ecosystem, a major challenge arises: data divergence.

Two analytics tools, even if they seem to be following the same guidelines, can at times produce different results. Why do they differ? How do you leverage both sets of data for your digital strategy?

In this article, we’ll use a concrete example of a user journey to illustrate differences in attribution between AB Tasty and Google Analytics. GA is a powerful tool for gathering and measuring data across the entire user journey. AB Tasty lets you easily make changes to your site and measure the impact on specific goals.

Navigating these differences in attribution strategies will explain why there can be different figures across different types of reports. Both are important to look at and which one you focus on will depend on your objectives:

Specific improvements in cross-session user experiences
Holistic analysis of user behavior

Let’s dive in!

Breaking it down with a simple use case

We’re going to base our analysis on a deliberately very basic use case, based on the user journey of a single visitor.

Campaign A is launched before the first session of the visitor and remains live until the end which occurs after the 3rd session of the visitor.

Here’s an example of the user journey we’ll be looking at in the rest of this article:

Session 1: first visit, Campaign A is not triggered (the visitor didn’t match all of the targeting conditions)
Session 2: second visit, Campaign A is triggered (the visitor matched all of the targeting conditions)
Session 3: third visit, no re-triggering of Campaign A which is still live, and the user carries out a transaction.

NB A visitor triggers a campaign as soon as they meet all the targeting conditions:

They meet the segmentation conditions

During their session, they visit at least one of the targeted pages

They meet the session trigger condition.

In A/B testing, a visitor exposed to a variation of a specific test will continue to see the same variation in future sessions, as long as the test campaign is live. This guarantees reliable measurement of potential changes in behavior across all sessions.

We will now describe how this user journey will be taken into account in the various AB Tasty and GA reports.

Analysis in AB Tasty

In AB Tasty, there is only one report and therefore only one attribution per campaign.

The user journey above will be reported as follows for Campaign A:

Total Users (Unique visitors) = 1, based on a unique user ID contained in a cookie; here there is only one user in our example.
Total Session = 2, s2 and s3, which are the sessions that took place during and after the display of Campaign A, are taken into account even if s3 didn’t re-trigger campaign A
Total Transaction = 1, the s3 transaction will be counted even if s3 has not re-triggered Campaign A.

In short, AB Tasty will collect and display in Campaign A reporting all the visitor’s sessions and events from the moment the visitor first triggered the campaign.

Analysis in Google Analytics

The classic way to analyze A/B test results in GA is to create an analysis segment and apply it to your reports.

However, this segment can be designed using 2 different methods, 2 different scopes, and depending on the scope chosen, the reports will not present the same data.

Method 1: On a user segment/user scope

Here we detail the user scope, which will include all user data corresponding to the segment settings.

In our case, the segment setup might look something like this:

This segment will therefore include all data from all sessions of all users who, at some point during the analysis date range, have received an event with the parameter event action = Campaign A.

We can then see in the GA report for our user journey example:

Total User = 1, based on a user ID contained in a cookie (like AB Tasty); here there is only one user in our example
Total Session = 3, s1, s2 and s3 which are the sessions created by the same user entering the segment and therefore includes all their sessions
Total Transaction = 1, transaction s3 will be counted as it took place in session s3 after the triggering of the campaign.

In short, in this scenario, Google Analytics will count and display all the sessions and events linked to this single visitor (over the selected date range), even those prior to the launch of Campaign A.

Method 2: On a session segment/session scope

The second segment scope detailed below is the session scope. This includes only the sessions that correspond to the settings.

In this second case, the segment setup could look like this:

This segment will include all data from sessions that have, at some point during the analysis date range, received an event with the parameter event action = Campaign A.

As you can see, this setting will include fewer sessions than the previous one.

In the context of our example:

Total User = 1, based on a user ID contained in a cookie (like AB Tasty), here there’s only one user in our example
Total Session = 1, only s2 triggers campaign A and therefore sends the campaign event
Total Transaction = 0, the s3 transaction took place in the s3 session, which does not trigger campaign A and therefore does not send an event, so it is not taken into account.

In short, in this case, Google Analytics will count and display all the sessions – and the events linked to these sessions – that triggered campaign A, and only these.

Attribution model

Tool – scope	Counted in the selected timeframe
AB Tasty	All sessions and events that took place after the visitor first triggered campaign A
Google Analytics – user scope	All sessions and events of a user that triggered campaign A at least once during one their sessions
Google Analytics – session scope	Only sessions that have triggered campaign A

Different attribution for different objectives

Depending on the different attributions of the various reports, we can observe different figures without the type of tracking being different.

The only metric that always remains constant is the sum of Users (Unique visitors in AB Tasty). This is calculated in a similar (but not identical) way between the 2 tools. It’s therefore the benchmark metric, and also the most reliable for detecting malfunctions between A/B testing tools and analytics tools with different calculations.

On the other hand, the attribution of sessions or events (e.g. a transaction) can be very different from one report to another. All the more so as it’s not possible in GA to recreate a report with an attribution model similar to that of AB Tasty.

Ultimately, A/B test performance analysis relies heavily on data attribution, and our exploration of the differences between AB Tasty and Google Analytics highlighted significant distinctions in the way these tools attribute user interactions. These divergences are the result of different designs and distinct objectives.

From campaign performance to holistic analysis: Which is the right solution for you?

AB Tasty, as a solution dedicated to the experimentation and optimization of user experiences, stands out for its more specialized approach to attribution. It offers a clear and specific view of A/B test performance, by grouping attribution data according to campaign objectives.

Making a modification on a platform and testing it aims to measure the impact of this modification on the performance of the platform and its metrics, during the current session and during future sessions of the same user.

On the other hand, Google Analytics focuses on the overall analysis of site activity. It’s a powerful tool for gathering data on the entire user journey, from traffic sources to conversions. However, its approach to attribution is broader, encompassing all session data, which can lead to different data cross-referencing and analysis than AB Tasty, as we have seen in our example.

It’s essential to note that one is not necessarily better than the other, but rather adapted to different needs.

Teams focusing on the targeted improvement of cross-session user experiences will find significant value in the attribution offered by AB Tasty.
On the other hand, Google Analytics remains indispensable for the holistic analysis of user behavior on a site.

The key to effective use of these solutions lies in understanding their differences in attribution, and the ability to exploit them in complementary ways. Ultimately, the choice will depend on the specific objectives of your analysis, and the alignment of these tools with your needs will determine the quality of your insights.

You might also like...

See all

Article

8min read

A Practical Guide to the European Accessibility Act

Thomas Hein

May 23, 2025

Article

2min read

Your Domain, Your Rules: Domain Delegation by AB Tasty

AB Tasty

May 21, 2025

Article

3min read

Feedback Copilot: Turn The Customer Voice Into Action in Seconds with AI

AB Tasty

May 20, 2025

Subscribe to
our Newsletter

Article

Nov 7, 2023

6min read

Taking an Outcome-Driven Approach | Ruben de Boer

Rowan Haddad

Ruben de Boer explains what it takes to create a healthy testing environment that paves the way for better experimentation organization-wide

Ruben de Boer is a lead CRO Manager and consultant with over 14 years of experience in data and optimization. At Online Dialogue, Ruben leads the Conversion Managers team, developing team skills and quality as well as setting the team strategy and goals. He spreads his knowledge far both as a teacher with Udemy with over 12,000 students and as a public speaker on topics such as experimentation, change management, CRO and personal growth.

In 2019, Ruben founded his company, Conversion Ideas, where he helps people kick start their career in Conversion Rate Optimization and Experimentation by providing affordable, high-quality online courses and a number of resources.

AB Tasty’s VP Marketing Marylin Montoya spoke with Ruben about exciting trends and evolutions within the world of experimentation, including the various ways AI can impact the optimization of the experimentation process. Ruben also shares ways to involve cross-functional teams to implement a successful culture of experimentation within the organization and why it’s important to steer these teams towards an outcome- rather than an output-driven mindset.

Here are some key takeaways from their conversation.

The goal should always be outcome-driven

Based on his experience, Ruben believes that one of the biggest pitfalls companies face when trying to kick start their experimentation journey is they focus more on outputs rather than outcomes.

“When a company is still very much in an output mindset, meaning we have to deliver an X amount of sprint points per sprint and we have to release so many new features this year, then of course experimentation can be seen as something that slows it down, right? Let’s say as a rule of thumb, 25% of A/B tests or experiments result in a winner and so 75% of what was built will not be released, which means the manager does not get the output goals.”

In this scenario, experimentation becomes an obstacle that slows down these outputs. Whereas, when a company shifts towards an outcome mindset, it makes more sense to run experiments with the goal to create more value for the customer. With an outcome-mindset, teams embrace experimentation with customers at the heart of the process.

When teams are more outcome-oriented, the product is based more on research and experiments instead of a fixed long-term roadmap. According to Ruben, it’s vital that companies adopt such a way of working as it helps create better products and business outcomes, which ultimately helps them maintain their competitive advantage.

Importance of cross-functional teams

Ruben argues that experimentation is maturing in that it’s becoming more embedded within product teams.

He notes there’s a rising trend of different teams working together, which Ruben believes is essential for knowledge sharing when it comes to learning new things about the customer journey and the product itself. For Ruben, this helps create an ideal, healthy experimentation environment for teams to experiment better and get the results they want.

Ideally, there would be experts in experimentation coming in from different teams sharing knowledge, ideas and insights on a regular basis which helps drive inspiration and innovation when it comes to future test ideas.

The recipe behind the success of these experimentation teams varies and depends on the maturity of the experimentation program and the skills of these teams.

This could start with a look into the culture of the organization by sending questionnaires to various teams to understand their work process and how autonomous they are. This analysis would also help teams to understand what their current state of experimentation is like such as how accepting they are of experimentation. This helps to devise a strategy and roadmap to successfully implement a culture of experimentation throughout the whole organization.

This culture scan also helps determine the maturity of an experimentation program.

“Process, data, team, scope, alignment, and company culture: that’s what I generally look at when I assess the maturity of an organization. Is there a CRO specialist throughout the different product teams? How’s decision-making being done by leadership? Is it based on the HIPPO decisions or fully based on experimentation? Then there’s the outcome versus output mindset, the scope and alignment of experimentation as well as the structure of the team- is it just a single CRO specialist or a multidisciplinary team? What does the process look like? Is it just a single CRO process or is it a process embedded in a project team?” Ruben says.

A world of possibilities with AI

With the advent of AI technology, Ruben believes there’s a lot of possibilities with what can be done with it, particularly in the experimentation process.

While he admits it’s still too early to speculate and that there are also the many privacy concerns that come with such technology, he believes AI can bring a lot of exciting things in the future.

“It would be so nice to have an AI go over experiments on the product detail page with all the results and all the learnings, and just ask the AI, what did I actually learn and what would be good follow up experiments on that? And that would be enormously interesting to have an AI run through all the experiments in the database,” Ruben says.

Therefore, Ruben admits there are a number of possibilities of what teams can do when it comes to designing experiments and saving time and steps in the experimentation process.

“And just think about maybe three or four years from now, everyone will just have an AI app on their phone and say, I need to buy this and I will buy it for you. And maybe a website with only AI apps on it to purchase stuff, who knows? And then optimization becomes very different all of a sudden.”

There’s also significant potential with AI when it comes to changing the way people work as well as provide inspiration and ultimately optimize and bring innovation to the experimentation process.

“Maybe based on all the input we give from chat logs, social media channels, reviews, surveys, we can make the AI behave like a user at some point in the future somewhere, which you then don’t have to run user tests anymore because you just let AI see your website.”

What else can you learn from our conversation with Ruben de Boer?

Evolving trends in experimentation
His take on change management to help organizations adopt experimentation
His own experiences with building cross-functional teams
How to tackle resistance when it comes to experimentation

About Ruben de Boer

With over 14 years of experience as a lead CRO manager and consultant in data and optimization, Ruben is a two-time winner in the Experimentation Elite Awards 2023 and a best-selling instructor on Udemy with over 12,000 students. He is also a public speaker on topics such as experimentation culture, change management, conversion rate optimization, and personal growth. Today, Ruben is the Lead Conversion Manager responsible for leading the Conversion Managers team, developing team skills and quality, setting the team strategy and goals, and business development.

About 1,000 Experiments Club

The 1,000 Experiments Club is an AB Tasty-produced podcast hosted by Marylin Montoya, CMO at AB Tasty. Join Marylin and the Marketing team as they sit down with the most knowledgeable experts in the world of experimentation to uncover their insights on what it takes to build and run successful experimentation programs.

You might also like...

See all

Article

8min read

A Practical Guide to the European Accessibility Act

Thomas Hein

May 23, 2025

Article

2min read

Your Domain, Your Rules: Domain Delegation by AB Tasty

AB Tasty

May 21, 2025

Article

3min read

Feedback Copilot: Turn The Customer Voice Into Action in Seconds with AI

AB Tasty

May 20, 2025

Subscribe to
our Newsletter

Article

Nov 6, 2023

10min read

Overcoming the Challenges of Customer Experience Optimization (EXO): Strategies and Tips

AB Tasty

The combination of intense competition and rapidly evolving technology requires businesses to prioritize customer experience optimization (EXO) to stay ahead.

The fact is, the cost of poor customer experience is high. According to a PWC survey, a third of consumers would stop using a brand they love after just one negative interaction.

In this article, we look at some common EXO challenges businesses face and strategies to overcome them, including practical insights for enhancing the digital customer experience. By implementing these strategies, you can ensure your business takes a customer-centric approach to optimizing the customer experience and building brand loyalty.

What is customer experience optimization?

Customer experience optimization refers to everything your business does to improve the customer’s experience at every touchpoint of their journey. It entails deeply understanding your customer’s needs and preferences and leveraging these insights to develop strategies to improve their interactions with your brand.

In today’s digital landscape, customers are flooded with choices across most categories of products and services. As a result, if you fail to deliver a positive experience, your customers will simply switch to a competing brand. EXO strategies are designed to keep customers satisfied and engaged, build brand loyalty, and reduce churn.

With EXO, it’s essential to deliver an experience that surpasses customers’ expectations and provides them with a seamless experience across all touchpoints and channels, including websites, mobile apps, social media accounts, and email.

Why customer experience optimization is important for business growth

First and foremost, EXO streamlines the customer’s path to purchase. Offering customers a frictionless, positive journey that makes it easy for them to get the information they need to make their purchase decision increases the likelihood of a successful transaction.

Customer EXO is also an ideal way to foster brand loyalty. Customers who have a superior experience with your brand are more likely to become repeat buyers. In fact, Deloitte research shows that a high-quality customer experience makes a customer 2.7 times more likely to keep buying from a business than a low-quality experience. Not only are customers likely to return, but they will also pay up to 16% more for an optimized experience, depending on the product category.

Positive experiences also trigger word-of-mouth recommendations, enhancing your brand’s reputation. Recommendations don’t entail the same acquisition costs as traditional marketing methods, making EXO a comparatively cost-effective way to boost sales and expand your customer base.

Challenges and solutions to customer experience optimization

We recognize there are challenges associated with EXO that may prevent you from delivering the best possible experience to your customers. Here are some strategies for tackling these challenges.

Compiling the right data for accurate measurements

Thanks to the various technologies available, we can now access a wealth of customer data. If interpreted and applied correctly, this data offers invaluable insights into the customer experience and ways of enhancing it. However, the sheer volume of these metrics can lead to information overload. It’s easy to get distracted or focus on the wrong metrics, including pitfall metrics that result in misinformed conclusions when considered in isolation. Some metrics, like cost of sale or cross-sell, don’t offer any meaningful insights into EXO.

The solution is to prioritize the metrics that matter. These include:

Customer satisfaction (CSAT)
Churn rates
Bounce rates
Customer retention rates
Trust ratings, conversion rates
Customer journey analytics
Repeat purchases
Customer segmentation
Buyer personas
Customer lifetime value (CLV)
Net Promoter Score (NPS)

Keep in mind that this data may reside in various departments across your organization, extending beyond sales, marketing and customer service teams. Consolidating this disparate data is essential to gaining a complete and accurate picture of customer experience in your organization.

Developing the right hypothesis

Experimentation is a powerful tool for delivering an optimal customer experience. However, randomly choosing hypotheses to test is a quick route to overlooking optimization opportunities. For example, simply changing the location of the checkout button in response to low conversion rates may not address the underlying issue.

Effective experimentation requires a considered approach to develop the correct hypothesis to test. The first step is identifying the genuine problem that needs addressing. You can then formulate a hypothesis to test to uncover the root cause of the issue and identify a concrete solution.

This second step requires a critical analysis of your current site and potential improvements from the customers’ perspective. Sourcing a range of data, including web analytics, user tests, and customer feedback, can help guide your analysis. You should also consider the psychology of the prospective customer. Getting in their mindset can guide you toward potential solutions.

If we continue with our checkout button example, the core issue may extend beyond conversion rates to a more specific concern: high cart abandonment rates. A hypothesis with a potential solution to this issue may be: “Many customers exit the checkout process at step 5. Reducing the number of steps in our checkout process will reduce cart abandonment rates.” Crafting the right hypothesis is a crucial step in optimizing customer experience.

Resource constraints

Ideally, businesses would have unlimited resources to optimize customer experiences. However, in reality, EXO usually competes with numerous other business priorities, all vying for time, human, and financial resources. Investing in the infrastructure and technology for EXO can be costly. Hiring and retaining people with the necessary skills to implement effective optimization strategies can also be challenging. Data availability is another common resource issue, especially for businesses with lower website traffic who feel they need more information for optimization.

The good news is you can tailor your approach to EXO to align with your business’s circumstances. This includes starting with smaller-scale initiatives and expanding your efforts as your optimization strategies gain traction or more resources become available. Another option is to outsource EXO by engaging the services of a specialist customer optimization agency.

It’s also important to note that high-volume website traffic isn’t a prerequisite for identifying and implementing effective EXO strategies. While a 95% confidence level is often cited as the magic number for drawing meaningful conclusions from your data, you can still optimize websites with less traffic by lowering the threshold. Focusing on optimizing the top of the funnel, where there may be greater opportunities for EXO, is another useful strategy for low-traffic websites.

ROI tunnel vision

When a company works on improving EXO, its main focus is often on immediate ROI in experimentation, sometimes at the expense of other important metrics. While the bottom line is relevant to any business strategy, focusing solely on the financial outcomes of EXO can lead to short-sighted decision-making, jeopardizing longer-term sustainability.

Prioritizing immediate revenue gain above all else can negatively impact the customer experience. It makes it almost impossible for an organization to adopt a customer-centric approach, a fundamental requirement for EXO.

Experimentation isn’t always neatly quantifiable. Experiments are typically run within complex contexts and are influenced by various factors. While measuring ROI may be a criterion when assessing the success of your EXO strategies, it should never be the primary or sole one. Instead, shift your focus to the broader impacts of experimentation, like its contribution to better, more informed decision-making.

Not knowing what your customers want

A customer-centric approach is vital to delivering an optimal customer experience. This requires an in-depth understanding of who your customers are, their needs and preferences, and precisely how they interact with your business. Without these insights, you’re in the dark about what your customers want and when they want it. Meeting—let alone exceeding—customers’ expectations is impossible.

Customer wants and needs are as diverse as your customer base. They may include a desire for higher levels of personalization, seamless online interactions, flexible payment methods, faster customer support, better pricing, transparency or increased mobile responsiveness. What customers want also evolves as their journey progresses. If your EXO strategies fail to align with your customers’ desires at the right time, they are unlikely to succeed.

While there are several ways to uncover customer needs and wants, one of the most effective methods is to go directly to the source. Collecting customer feedback at each stage of their journey—via surveys, feedback management systems, voice of customers, and user interviews—lets you tailor your EXO strategies and deliver the improvements your customers truly want.

Lack of customer experience optimization tools

Successful EXO relies on quality data for insights into your customers’ journeys, needs, and preferences. To achieve this, you need the right tools to capture and analyze accurate data in real-time across multiple channels.

These tools include:

CRM systems to track historical customer behavior and relationships
Customer feedback and survey software to collect individual feedback for deep insights into what your customers want
Behavior analytics tools to interpret your customers’ interactions and identify opportunities to improve their experience
Experience optimization platforms, like AB Tasty, to design and deliver digital omnichannel customer experiences via experimentation

It’s important to review the needs of your EXO strategy and the available tools to choose the ones that best align with your customers’ and business’s needs.

How to improve the digital customer experience

Observe user behavior patterns

A robust data foundation lets you observe and understand customer behavior individually and identify broader trends. This information serves as a compass, guiding your EXO efforts.

Customer insights may reveal common pain points. For example, a frequently searched term may highlight a topic customers want more information on. These insights also help you understand how users interact with your site, how that impacts their journey, and potential improvements. Is there a particular page where customers spend a lot of time? Do they have to navigate back and forth between pages to find the details they need?

Behavior patterns also reveal customer preferences, allowing you to personalize touchpoints within their journey and identify what triggers customers to complete their purchases. These insights serve as a powerful foundation for developing EXO strategies and hypotheses for A/B testing.

Create a journey map to understand the user flow

EXO involves optimizing every customer interaction with your business. A common pitfall to avoid when addressing EXO is approaching it narrowly from a specific touchpoint rather than considering the entire customer journey. A holistic approach delivers more impactful insights that help you manage the root causes of negative or neutral customer experiences.

A great way to understand your user flow and how it affects customer experience is to create a journey map, setting out every touchpoint during the buying process. Navigate your website like a potential customer, systematically stepping through the user journey and noting your findings.

Putting yourself in the customer’s shoes ensures you don’t overlook opportunities to optimize customer experience. This approach can also help you prioritize measures that make the user journey frictionless, improving customer experience and your site’s performance.

Develop a roadmap and set parameters to measure success

The list of available EXO measures is endless. Aligning your strategy with your business objectives requires a considered approach to implementation. To do this, develop a roadmap that outlines your goals, priorities and milestones.

A well-structured roadmap gives your team clear direction and deadlines while guiding decision-making to ensure the greatest impact on customer experience. Everyone understands their role, guaranteeing accountability in the execution of your EXO strategy. It also helps you prioritize initiatives and allocate the necessary resources, including EXO tools.

In your roadmap, you can list the specific metrics and KPIs to measure and track your progress. Doing this allows you to evaluate your EXO measures, readjust those not delivering results, and build on particularly effective ones.

Experiment and re-challenge your past experiments

You’re unlikely to unlock the secret to EXO in your organization on the first try. Instead, you’ll need to run continuous experiments using different hypotheses to find the right combination of strategies that work for your business.

The customer experience is dynamic and your EXO strategies should be equally adaptable. Continue to review your previous experiments to see what more you can learn from them, especially in terms of customer preferences. This process enables you to identify emerging opportunities for improvement and further refine the measures with the most impact to deliver an optimal customer experience.

Customer-centric EXO

Acknowledging that your business must prioritize customer EXO is just the beginning. By understanding the customer experience definition, common EXO challenges, and practical strategies to overcome them, you have the tools to deliver a consistently superior customer experience. By integrating a customer-centric ethos with your EXO strategies, you’ll not only strengthen current customer relationships but also cultivate enduring brand loyalty.

You might also like...

See all

Article

8min read

A Practical Guide to the European Accessibility Act

Thomas Hein

May 23, 2025

Article

2min read

Your Domain, Your Rules: Domain Delegation by AB Tasty

AB Tasty

May 21, 2025

Article

3min read

Feedback Copilot: Turn The Customer Voice Into Action in Seconds with AI

AB Tasty

May 20, 2025

Subscribe to
our Newsletter

Article

Oct 24, 2023

10min read

How to Effectively Use Sequential Testing

Hubert Wassner

In A/B tests where you can see the data coming in a continuous stream, it’s tempting to stop the experiment before the planned end. It’s so tempting that in fact a lot of practitioners don’t even really know why one has to define a testing period beforehand.

Some platforms have even changed their statistical tools to take this into account and have switched to sequential testing which is designed to handle tests this way.

Sequential testing enables you to evaluate data as it’s collected to determine if an early decision can be made, helping you cut down on A/B test duration as you can ‘peak’ at set points.

But, is this an efficient and beneficial type of testing? Spoiler: yes and no, depending on the way you use it.

Why do we need to wait for the predetermined end of the experiment?

Planning and respecting the data collection period of an experiment is crucial. Historical techniques use “fixed horizon testing” that establishes these guidelines for all to follow. If you do not respect this condition, then you don’t have the guarantee provided by the statistical framework. This statistical framework guarantees that you only have a 5% error risk when using the common decision thresholds.

Sequential testing promises that when using the proper statistical formulas, you can stop an experiment as soon as the decision threshold is crossed and still have the 5% error risk guarantee. The test user here is the sequential Z-test, which is based on the classical Z-test with an added correction to take the sequential usage into account.

In the following sections, we will look at two objections that are often raised when it comes to sequential testing that may put it at odds with CRO practices.

Sequential testing objection 1: “Each day has to be sampled the same”

The first objection is that one should sample each day of the week the same way. This is basically to have a sampling that represents reality. This is the case in a classic A/B test. However, this rule may be broken if you use sequential testing since you can stop the test mid-week but this is not always applicable. Since in reality there are seven different days, your sampling unit should be by week and not by day to account for behavioral differences over the course of a week.

As experiments typically last 2-3 weeks, then the promise of sequential testing saving days isn’t necessarily correct unless a winner appears very early in the process. However, it’s more likely that the statistical test yielded significance during the last week. In this case, it’s best to complete the data collection until each day is sampled evenly so that the full period is covered.

Let’s consider the following simulation setting:

One reference with a 5% conversion rate
One variation with a 5.5% conversion rate (a 10% relative improvement)
5,000 visitors as daily traffic
14 days (2 weeks) of data collection
We ran thousands of such experiments to get histograms for different decision index

In the following histogram, the horizontal axis is the day when the sequential testing crosses the significance threshold. The vertical axis is the ratio of experiments which stopped on this day.

In this setting, day 10 is the most likely day for the sequential testing to reach significance. This means that you will need to wait until the planned end of the test to respect the “same sampling each day” rule. And it’s very unlikely that you will get a significant positive result in one week. Thus, in practice, determining the winner sooner with sequential testing doesn’t apply in CRO.

Sequential testing objection 2: “Yes, (effect) size does matter”

In sequential testing, this is often a less obvious problem and may need some further clarification to be properly understood.

In CRO, we consider mainly two statistical indices for decision-making:

The pValue or any other confidence index, which is linked to the fact that there exists (or not) a difference between the original and the variation. This index is used to validate or invalidate the test hypothesis. But a validated hypothesis is not necessarily a good business decision, so we need more information.
The Confidence Interval (CI) around the estimated gain, which indicates the size of the effect. It’s also central to business decisions. For instance, a variation can be a clear winner but with a very little margin that may not cover the implementation or operating costs such as coupon offerings that need to cover the coupon cost.

Confidence intervals can be seen as a best and worst case scenario. For example a CI = [1% ; 12%] means “in the worst case you will only get 1% relative uplift,” which means going from 5% conversion rate to 5.05%.

If the variation has an implementation or operating cost, the results may not be satisfying. In that case, the solution would be to collect more data in order to have a narrower confidence interval, until you get a more satisfying lower bound, or you may find that the upper bound goes very low showing that the effect, even if it exists, is too low to be worth it from a business perspective.

Using the same scenario as above, the lower bound of the confidence interval can be plotted as follows:

Horizontal axis – the percentage value of the lower bound
Vertical axis – the proportion of experiments with this lower bound value
Blue curve – sequential testing CI
Orange curve – classical fixed horizon testing

We can see that sequential testing has a very low confidence interval for the lower bound. Most of the time, this is lower than 2% (in relative gain, which is very small). This means that you will get very poor information for business decisions.

Meanwhile, a classic fixed horizon testing (orange curve) will produce a lower bound >5% in half of the cases, which is a more comfortable margin. Therefore, you can continue the data collection until you have a useful result, which means waiting for more data. Even if by chance the sequential testing found a variant reaching significance in one week, you will still need to collect data for another week to do two things: have a useful estimation of the uplift and sample each day equally.

This makes sense in light of the purpose of sequential testing: quickly detect when a variation produces results that differ from the original, whether for the worse or better.

If done as soon as possible, it makes sense to stop the experiment as soon as the gain confidence interval lays mostly either on the positive or negative side. Then, for the positive side, the CI lower bound is close to 0, which doesn’t allow for efficient business decisions. It’s worth noting that for other applications other than CRO, this behaviour may be optimal and that’s why sequential testing exists.

When does sequential testing in CRO make sense?

As we’ve seen, sequential testing should not be used to quickly determine a winning variation. However, it can be useful in CRO in order to detect losing variations as soon as possible (and prevent loss of conversions, revenue, …).

You may be wondering why it’s acceptable to stop an experiment midway through when your variation is losing rather than when you have a winning variation. This is because of the following reasons:

The most obvious one: To put it simply, you’re losing conversions. This is acceptable in the context of searching for a better variation than the original. However, this makes little sense in cases where there is a notable loss, indicating that the variation has no more chances to be a winner. An alerting system set at a low sensitivity level will help detect such impactful losses.
The less obvious one: Sometimes when an experiment is only slightly “losing” for a good period of time, practitioners tend to let this kind of test run in the hopes that it may turn into a “winner”. Thus, they accept this loss because the variation is only “slightly” losing but they often forget that another valuable component is lost in the process: traffic, which is essential for experimentation. For an optimal CRO strategy, one needs to take these factors into account and consider stopping this kind of useless experiment, doomed to have small effects. In such a scenario, an automated alert system will suggest stopping this kind of test and allocate this traffic to other experiments.

Therefore, sequential testing is, in fact, a valuable tool to alert and stop a losing variation.

However, one more objection could still be raised: by stopping the experiment midway, you are breaking the “sample each day the same” rule.

In this particular case, stopping a losing variation has very little chance to be a bad move. In order for the detected variation to become a winner, it first needs to gain enough conversions tobe comparable to the original version. Then it would need another set of conversions to be a “mild” winner and that still wouldn’t be enough to be considered a business winner (and cover the implementation or exploitation costs of that winner). To be considered a winner for your business, the competing variation will need another high amount of conversions with a sufficient margin. This margin needs to be high enough to cover the cost of implementation, localization, and/or operating costs.

All the aforementioned events should happen in less than a week (ie. the number of days needed to complete the current week). This is very unlikely, which means it’s safe and smart to stop such experiments.

Conclusion

It may be surprising or disappointing to see that there’s no business value in stopping winning experiments early as others may believe. This is because a statistical winner is not a business winner. Stopping a test early is taking away the data you need to reach a significant effect size that would increase your chances of getting a winning variation.

With that in mind, the best way to use this type of testing is as an alert to help spot and stop tests that are either harmful to the business or not worth continuing.

About the Author:

Hubert Wassner has been working as a Senior Data Scientist at AB Tasty since 2014. With a passion for science, data and technology, his work has focused primarily on all the statistical aspects of the platform, which includes building Bayesian statistical tests adapted to the practice of A/B testing for the web and setting up a data science team for machine learning needs.

After getting his degree in Computer Science with a speciality in Signal Processing at ESIEA, Hubert started his career as a research engineer doing research work in the field of voice recognition in Switzerland followed by research in the field of genomic data mining at a biotech company. He was also a professor at ESIEA engineer school where he taught courses in algorithmics and machine learning.

You might also like...

See all

Article

8min read

A Practical Guide to the European Accessibility Act

Thomas Hein

May 23, 2025

Article

2min read

Your Domain, Your Rules: Domain Delegation by AB Tasty

AB Tasty

May 21, 2025

Article

3min read

Feedback Copilot: Turn The Customer Voice Into Action in Seconds with AI

AB Tasty

May 20, 2025

Subscribe to
our Newsletter

Article

Oct 17, 2023

4min read

Doing CRO at Scale? AB Tasty’s Testing Alerts Prevent Revenue Loss

Hubert Wassner

When talking about CRO at scale, a common fear is, what if some experiments go wrong? And we mean seriously wrong. The more experiments you have running, the higher the chance that one could negatively impact the business. It might be a bug in a variant, or it might just be a bad idea – but conversion can drop dramatically, hurting your revenue goals.

That’s why it makes sense to monitor experiments daily even if the experiment protocol says that one shouldn’t make decisions before the planned end of the experiment. This leads to two questions:

How do you make a good decision using statistical indices that are not meant to be used that way? Checking it daily is known as “P-hacking” or “data peeking” and generates a lot of poor decisions.
How do you check tens of experiments daily without diverting the team’s time and efforts?

The solution is to use another statistical framework, adapted to the sequential nature of checking an experiment daily. This is known as sequential testing.

How do I go about checking my experiments daily?

Good news, you don’t have to do this tedious job yourself. AB Tasty has developed an automatic alerting system, based on a specialized statistical test procedure, to help you launch experiments at scale with confidence.

Sequential Testing Alerts offer a sensitivity threshold to detect underperforming variations early. It triggers alerts and can pause experiments, helping you cover your losses. It enables you to make data-driven decisions swiftly and optimize user experiences effectively. This alerting system is based on the percentage of conversions and/or traffic lost.

When creating a test, you simply check a box, select the sensitivity level and you’re done for the rest of the experiment’s life. Then, if no alert has been received, you can go ahead and analyze your experiment as usual; but in case of a problem, you’ll be alerted through the notification center. Then you head on over to the report page to assess what happened and make a final decision about the experiment.

Can I also use this to identify winners faster?

No; this may sound weird but it’s not a good idea, even if some vendors are telling you that you should do it. You may be able to spot a winner faster but the estimation of the effect size will be very poor, preventing you from making wise business decisions.

But what about “dynamic allocation”?

First of all, what is dynamic allocation? As its name suggests, it is an experimentation method where the allocation of traffic is not fixed but depends on the variation’s performance. While this may be seen as a way to limit the effect of bad variations, we don’t recommend using it for this purpose for several reasons:

As you know, the strict testing protocol strongly suggests not to change the allocation during a test, so dynamic allocation is an edge case of A/B testing. We only suggest using it when you have no other choice. Classic use cases are very low traffic, or very short time frames (flash sales, for instance). So if your only concern is avoiding big losses from a bad variation, alerting is a better and safer option that respects the A/B testing protocol.
Since dynamic allocation changes the allocation, in order to avoid the risk of missing out on a winner, it always keeps a minimum traffic amount for the supposed losing variation, even if it’s losing by a lot.

Therefore, dynamic allocation isn’t a way to protect revenue but rather to explore options in the context of limited traffic.

Other benefits

Another benefit of the sequential testing alert feature is better usage of your traffic. Experiments with a lot of traffic with no chance to win are also detected by the alerting system. The system will suggest to the user to stop that experiment and make better use of the traffic for other experiments.

The bottom line is: you can’t do CRO at scale without an automated alert system. Receiving such crucial alerts can protect you from experiments that could otherwise cause revenue loss. To learn more about our sequential testing alert and how it works, refer to our documentation.

You might also like...

See all

Article

8min read

A Practical Guide to the European Accessibility Act

Thomas Hein

May 23, 2025

Article

2min read

Your Domain, Your Rules: Domain Delegation by AB Tasty

AB Tasty

May 21, 2025

Article

3min read

Feedback Copilot: Turn The Customer Voice Into Action in Seconds with AI

AB Tasty

May 20, 2025

Subscribe to
our Newsletter

Article

Oct 13, 2023

9min read

Best Practice for Optimizing Mobile vs. Desktop Experiences

AB Tasty

Best Practice for Optimizing Mobile vs Desktop Experiences

For product and web teams, the prospect of mobile is both wonderful and terrifying. The wonderful: your product or website is available all the time! The terrifying: you now have to create and develop two (or three, or four) user experiences, all at once.

While your customers probably walk around with their phones all the time, they may also spend the majority of their workday on a desktop computer. They might browse your mobile website or app during sporadic free moments throughout the day, then finish their customer journey later, when they’re not limited by small screen sizes and thumb-based typing.

The bottom line? Your entire product experience needs to be seamless, and combined across all platforms and devices. When you’re building an app, chances are you’re building a desktop version for web browsers, and at least two versions for mobile: iOS and Android. Given that people might also use the app on their phone’s web browser—a different experience than on a desktop but not quite the same as a native phone app—it’s easy to wonder where to start with the design process!

Below we’ve outlined some key tips for doing just that.

Plan an Entire Digital Experience

Regardless, your designs across both desktop and mobile have two requirements right out of the gate: they must be intuitive, and they must be accessible for people with disabilities.

You’ll also want to consider the fact that designing apps with a “mobile first” mindset is now a standard, because 1) users will expect to be able to access your digital ecosystem from their phones, and 2) web apps that are optimized for mobile rank higher on Google, which can lead to more conversions.

Before you start designing your app, make sure you plan to accommodate the key differences between desktop and mobile:

Both desktop and mobile experiences are crucial for customer conversion and retention, but often in different ways and in different circumstances. Below, we’ll take a look at best practices for keeping your mobile and desktop experiences updated and healthy for your sales funnel.

Best Practice for the Mobile Digital Experience

Whether your mobile experience exists within a dedicated iOS or Android app or simply as a responsive web-based app, there are some key best practices to be aware of when building your design strategy to maximize conversions and retention.

1. Keep it accessible, always

If your website or app is used by the public, it legally must comply with accessibility standards. These include:

Text size options
Large, easy-to-use buttons and CTAs
Minimum color contrast ratios
Simple touchscreen gestures
Adjustable screen orientation
Compatibility with screen readers
Content translation ability

2. Make sure your mobile content is consistent with desktop

If your users regularly hop between their computer and phones, it’s important that they can easily and consistently find what they need. This can include information, tools, actions, products, and more. If they’re available on desktop, they should also be available on mobile!

For example, if you have a menu on your desktop app that offers users different ways to contact customer service but it’s missing on mobile, it will likely cause frustration when users are trying to get help on the go.

3. Consider your mobile users’ needs

Ask yourself: What data, tools, or functions will your users be accessing? Make everything easy to find, navigate, and use. This might start with an intuitive navigation system that lets users easily:

Make purchases
Find specific tools they love to use
Manage their account or subscription
Get help from your support team

You’ll also want to consider what your different users will be trying to do and from where. Will they be using cellular data (slow) or wi-fi (fast)? Will they be reserving certain actions for mobile vs. desktop? Will tasks that may have started on a processor-heavy desktop app later migrate to mobile (or vice versa)?

For example, a video editor might need to use a desktop app to edit terabytes’ worth of cinema-quality video before switching to mobile to share the final piece on social media. Personal workflows might vary, but it all gets down to having the right tools on hand at all times.

4. Check how your site and app perform across different devices

You’ve built your iOS app. You’ve built your Android app. Your responsive website’s parameters are set so they’ll adjust responsively for users accessing from phones and tablets.

Now comes the testing. App Store or Google Play installations should work seamlessly. Each button, form, menu, and action must work flawlessly. Load times should be fast, and there should be little to no latency between screens (though this can vary with web-based apps, depending on how they’re constructed).

For responsive sites, each element will have CSS parameters that dictate how it will appear on a phone, tablet, or desktop computer. You don’t need separate versions of your website, but each element must be tagged correctly so it renders properly on each device.

5. Make UX choices that encourage conversion and retention

Good UX (user experience) comes from good user research and robust product analytics, but it’s customer conversion and retention that will keep your app alive. If users buy items or services through your app, you’ll want to keep these top UX tips in mind:

Never hide menus and navigation options.
Make search bars easy to find.
Make product browsing simple.
Choose familiar, suitable icons for cart checkout processes.
Label all icons, buttons, and navigation items.
Make checkout funnels easy (including credit card selection/adding), or incorporate simple payment options like Apple Pay or Google Pay.
Consider adding a subscription option so users can set it once and forget it!

Don’t forget that there are product analytics tools available to analyze the particular actions your users take, including Session Replays. These help you identify which parts of your UX design are working and which parts are causing friction.

Best Practice for Desktop Optimization

Desktop apps (many of which are web-based) are also commonly used and have more screen real estate to work with. Here are five tips to optimize your desktop experience to convert (and retain) new customers:

1. Create an engaging homepage

A great UX on your homepage can make or break conversion rates. When you’ve spent time and resources optimizing SEO, you’ll want to give potential customers a good first impression with a strong design. Here’s what we recommend to maximize conversion and retention:

Keep your design simple with a single, clear CTA.
Make sure it meets accessibility standards.
Use a catchy headline and simple buzzwords to make your value proposition clear and concise.
Choose the right colors by employing and understanding color psychology.
Differentiate hyperlinked text with an accent color.
Embrace white space! Space around your text can keep people reading it.
Use well-lit, professional images and videos where they fit organically.
Again, make sure your site is responsive to serve mobile users!

2. Improve website speed

Nothing deflects a website visitor faster than a glacially slow load time. To avoid losing possible customers, you can:

Optimize your images by saving compressed versions at 72dpi.
Limit the number of assets needed to load your page correctly—this can help reduce HTTP requests, which speeds up load times.
Limit the use of external scripts.
Remove unnecessary text, white space, and comments from CSS and Javascript files.
Use browser HTTP caching—this can offer faster load times for users who frequently visit your site.

3. Develop intuitive site navigation

Site navigation—typically in the form of a top menu, side-bar menu, or organized icons—shows users what to do once they reach your site. Use descriptive labels for each item, and simplify drop-down menu structures so users won’t get overwhelmed.

4. Make it easy to search for terms

Adding a search bar to your top menu can be a great way to give users a way to search your site for any term (or action) they want. This gives them a shortcut to find exactly what they need!

5. Ready your desktop site for final steps of conversion and retention

Often, consumers are more likely to say that desktop offers a more “convenient” shopping experience than mobile, and a majority of people complete purchases on a desktop.

As such, desktop websites are still used and still valuable, particularly for final conversation and ongoing retention. Whether it’s because customers are making complex purchases or simply seeing your checkout funnel more clearly on desktop, it’s essential to make sure every step of your experience is seamless.

Once you’ve converted customers, it can also help to keep track of their preferences so you can personalize their experience going forward.

Gain Valuable Insights with Heap

This is a guest blog written by Ben Lempert, Head of Content and Web at Heap.

Heap And AB Tasty provide businesses with the ability to collect and analyze user data efficiently, leading to more informed decisions and higher conversion rates. This dynamic combination offers a comprehensive solution for optimizing web experiences and increasing revenue through data-driven strategies.

Curious how you can find out what your users are doing across both desktop and mobile? Heap offers tools to help you track, measure, and ultimately optimize every touchpoint of users’ journeys across your entire digital experience.

You might also like...

See all

Article

8min read

A Practical Guide to the European Accessibility Act

Thomas Hein

May 23, 2025

Article

2min read

Your Domain, Your Rules: Domain Delegation by AB Tasty

AB Tasty

May 21, 2025

Article

3min read

Feedback Copilot: Turn The Customer Voice Into Action in Seconds with AI

AB Tasty

May 20, 2025

Subscribe to
our Newsletter

Article

Oct 5, 2023

9min read

Inconclusive A/B Test Results – What’s Next?

Hubert Wassner

Have you ever had an experiment leave you with an unexpected result and were unsure of what to do next? This is the case for many when receiving neutral, flat, or inconclusive A/B test results and this is a question we aim to answer.

In this article, we are going to discuss what an inconclusive experimentation result is, what you can learn from it, and what the next step is when you receive this type of result.

What is an inconclusive experiment result?

We have two definitions for an inconclusive experiment: a practitioner’s answer and a more broken-down answer. A basic practitioner’s answer is a numerical answer that shows statistical information depending on the platform you’re using:

The probability of a winner is less than 90-95%
The pValue is bigger than 0.05
The lift confidence interval includes 0

In other words, an inconclusive result happens when the results of an experiment are non-statistically significant or an uplift is too small to be measured.

However, let’s take note of the true meaning of “significance” in this case: the significance is the threshold one has previously set as a metric or a statistic for measurement. If this previously set threshold is crossed, then an action will be made, usually implementing the winning variation.

Setting thresholds for experimentation

It’s important to note that the user sets the threshold and there are no magic formulas for calculating a threshold value. The only mandatory thing that must be done is that the threshold must be set before the beginning of an experiment. In doing so, this statistical hypothesis protocol provides caution and mitigates the risks of making a poor decision or missing an opportunity during experimentation.

To set a proper threshold, you will need a mix of statistical and business knowledge considering the context.

There is no golden rule, but there is a widespread consensus for using a “95% significance threshold.” However, it’s best to use this generalization cautiously as using the 95% threshold may be a bad choice in some contexts.

To make things simple, let’s consider that you’ve set a significance threshold that fits your experiment context. Then, having a “flat” result may have different meanings – we will dive into this more in the following sections.

The best tool: the confidence interval (CI)

The first thing to do after the planned end of an experiment is to check the confidence interval (CI) that can tell useful information without any notion of significance. The usage is a 95% confidence level to build these intervals. This means that there is a 95% chance that the real value lies between its boundaries. You can consider the boundaries to be an estimate of the best and worst-case scenarios.

Let’s say that your experiment is collaborating with a brand ambassador (or influencer) to attract more attention and sales. You want to see the impact the brand ambassador has on the conversion rate. There are several possible scenarios depending on the CI values:

Scenario 1:

The confidence interval of the lift is [-1% : +1%]. This means that in the best-case scenario, this ambassador effect is a 1% gain and in the worst-case scenario, the effect is -1%. If this 1% relative gain is less than the cost of the ambassador, then you know that it’s okay to stop this collaboration.

A basic estimation can be done by taking this 1% of your global revenue from an appropriate period. If this is smaller than the cost of the ambassador, then there is no need for “significance“ to validate the decision – you are losing money.

Sometimes neutrality is a piece of actionable information.

Scenario 2:

The confidence interval of the lift is [-1% : +10%]. Although this sounds promising, it’s important not to make quick assumptions. Since the 0 is still in the confidence interval, you’re still unsure if this collaboration has a real impact on conversion. In this case, it would make sense to extend the experiment period because there are more chances that the gain will be positive than negative.

It’s best to extend the experimentation period until the left bound gets to a “comfortable” margin.

Let’s say that the cost of the collaboration is covered if the gain is as small as 3%, then any CI [3%, XXX%] will be okay. With a CI like this, you are ensuring that the worst-case scenario is even. And with more data, you will also have a better estimate of the best-case scenario, which will certainly be lower than the initial 10%.

Important notice: do not repeat this too often, otherwise you may be waiting until your variant beats the original just by chance.

When extending a testing period, it’s safer to do it by looking at the CI rather than the “chances to win” or P-value, because the CI provides you with an estimate of the effect size. When the variant wins only by chance (which you increase when extending the testing period), it will yield a very small effect size.

You will notice the size of the gain by looking at the CI, whereas a p-value (or any statistical index) will not inform you about the size. This is a known statistical mistake called p-hacking. P-hacking is basically running an experiment until you get what you expect.

The dangers of P-hacking in experimentation

It’s important to be cautious of p-hacking. Statistical tests are meant to be used once. Splitting the analysis into segments, to some extent, can be seen as portraying different experiences. Therefore, if making a unique decision at a 95% significance level means accepting a 5% risk of having a false positive, then checking for 2 segments implicitly leads to doubling this risk to 10% (roughly).

We recommend the following advice may help to mitigate this risk:

Limit the number of segments you are studying to only segments that could have a reason to interact differently with the variation. For example: if it’s a user interface modification (such as the screen size or the navigator used), it may have an impact on how the modification is displayed, but not the geolocation.
Use segments that convey strong information regarding the experiment. For example: Changing the wording of anything may have no link to the navigator used. It may only have an effect on the emotional needs of the visitors, which is something you can capture with new AI technology when using AB Tasty.
Don’t check the smallest segments. The smallest segments will not greatly impact your business overall and are often the least statistically significant. Raising the significance threshold may also be useful to mitigate the risk of having a false positive

Should you extend the experiment period often?

If you notice that you often need to extend the experiment period, you might be skipping an important step in the test protocol: estimating the sample size you need for your experiment.

Unfortunately, many people are skipping this part of the experiment thinking that they can fix it later by extending the period. However, this is bad practice for several reasons:

This brings you close to P-hacking
You may lose time and traffic on tests that will never be significant

Asking a question you can’t know the answer to can be very difficult: what will be the size of the lift? It’s impossible to know. This is one reason why experimenters don’t often use sample size calculators. The reason you test and experiment is because you do not know the outcome.

A far more intuitive approach is to use a Minimal Detectable Effect (MDE) calculator. Based on the base conversion rate and the number of visitors you send to a given experiment, an MDE calculator can help you come up with the answer to the question: what is the smallest effect you may be able to detect? (if it exists).

For example, if the total traffic on a given page is 15k for 2 weeks, and the conversion rate is 3% – the calculator will tell you that the MDE is about 25% (relative). This means that what you are about to test must have a quite big impact: going from 3% to 3.75% (25% relative growth).

If your variant is only changing some colors to a small button, developing an entire experiment may not be worth the time. Even if the new colors are better and give you a small uplift, it will not be significant in the classic statistical way (having a “chance to win” >95% or a p-value < 0.05).

On the other hand, if your variation tests a big change such as offering a coupon or a brand new product page format, then this test has a chance to give usable results in the given period.

Digging deeper into ‘flatness’

Some experiments may appear to be flat or inconclusive when in reality, they need a closer look.

For example, frequent visitors may be puzzled by your changes because they expect your website to remain the same, whereas new visitors may instantly prefer your variation. This combined effect of the two groups may cancel each other out when looking at the overall results instead of further investigating the data. This is why it’s very important to take the time to dig into your visitor segments as it can provide useful insights.

This can lead to very useful personalization where only a given segment will be exposed to the variation with benefits.

What is the next step after receiving an inconclusive experimentation result?

Let’s consider that your variant has no effect at all, or at least not enough to have a business impact. This still means something. If you reach this point, it means that all previous ideas fell short; You discovered no behavioral difference despite the changes you made in your variation.

What is the next step in this case? The next step is actually to go back to the previous step – the hypothesis. If you are correctly applying the testing protocol, you should have stated a clear hypothesis. It’s time to use it now.

There might be several meta-hypotheses about why your hypothesis has not been validated by your experiment:

The signal is too weak. You might have made a change, but perhaps it’s barely noticeable. If you offered free shipping, your visitors might not have seen the message if it’s too low on the page.
The change itself is too weak. In this case, try to make the change more significant. If you have increased the product picture on the page by 5% – it’s time to try 10% or 15%.
The hypothesis might need revision. Maybe the trend is reversed. For instance, if the confidence interval of the gain is more on the negative side, why not try the opposite idea to implement?
Think of your audience. Another consideration is that even if you have a strong belief about your hypothesis, it’s just time to change your mind about what is important for your visitors and try something different.

It’s important to notice that this change is something that you’ve learned thanks to your experiment. This is not a waste of time – it’s another step forward to better knowing your audience.

Yielding an inconclusive experiment

An experiment not yielding a clear winner (or loser), is often called neutral, inconclusive, or flat. This still produces valuable information if you know how and where to search. It’s not an end, it’s just another step further in your understanding of who you’re targeting.

In other words, an inconclusive experiment result is always a valuable result.

You might also like...

See all

Article

8min read

A Practical Guide to the European Accessibility Act

Thomas Hein

May 23, 2025

Article

2min read

Your Domain, Your Rules: Domain Delegation by AB Tasty

AB Tasty

May 21, 2025

Article

3min read

Feedback Copilot: Turn The Customer Voice Into Action in Seconds with AI

AB Tasty

May 20, 2025

Subscribe to
our Newsletter

Article

Sep 21, 2023

10min read

Software Development Team Best Practices

Rowan Haddad

Software development isn’t just developers writing code. It also includes less technical processes that precede the actual development process such as the planning and the testing stages as well as post-development when the software is released and feedback is gathered from end-users.

What this means is that many teams beyond development are involved in the software development life cycle, such as product, design, testing, sales and marketing teams. These teams are all involved in achieving common objectives and ensuring a high-quality product.

Software development teams

To understand why it’s so important for teams to work cross-functionally, it helps to take a look at the different types of teams involved in software development. This will uncover the best practices to get these teams on the road for enhanced collaboration.

When we visualize a software development team, it’s easy to imagine a group of developers writing and releasing code but the reality is that it’s much more than that.

A software development team brings together a wide range of expertise from different teams within an organization to ensure the success of a project. This means that most teams are not solely made up of developers because while they’re responsible for creating the product, there also needs to be people dedicated to building the vision of the product, managing its life cycle, testing the product and marketing it and so on.

A software development team typically consists of the following roles:

Business analyst
Product owner
Project manager
UI/UX designer
Developers (frontend and backend)
QA engineer
Testers
DevOps engineer

With various teams from different departments coming together, there’s a great advantage in having expertise across multiple disciplines, which can bring innovative solutions to problems and insights which otherwise would be overlooked if these teams were to work in silos. Therefore, each of these roles is key to the effective development of your product.

Many factors will influence the structure of your development team such as project complexity, budget and size as well as the needs and expectations of stakeholders.

Ultimately, the team you put together will determine your project’s likelihood of success or failure and their collaboration will be key to achieving desired outcomes.

Why is cross-functional collaboration important?

It’s inevitable for different teams to clash during projects. For example, developers and product teams tend to approach projects from vastly different perspectives and their metrics for success will also differ.

Product managers are often focused on achieving outcomes and overall business objectives. They’re looking to quickly validate their ideas and just as quickly release features that will bring in more revenue for the business. Developers, for their part, aim to build the best possible product by focusing on the deliverables that come with building this product.

The same applies to other teams within the organization with the different types of mindset, skills and goals coming into play during a project.

It’s vital for cross-functional teams to communicate and collaborate effectively around a shared goal in order to successfully achieve it.

Effective collaboration results in better products. Less conflict between the different teams translates to more time dedicated to building and releasing high-quality products. In other words, when teams are on the same page, it allows them to focus on what really matters, which is creating value for the customer through quality software.

Furthermore, cross-functional collaboration is a key driver for creativity and innovation. Collaboration often results in new ideas that can help companies gain competitive advantages as different people come together to work on a project as they encourage each other to consider things from different angles.

Software development team best practices

To enhance productivity and to continue to deliver value to customers, software development teams should stick to some best practices to put them on the path to improved collaboration and success.

Define clear goals

While each team has their own set of internal goals, they still need to make sure that those goals align with the overall business objectives of the company (and product). It’s essential that all teams have a shared understanding of business goals and how to achieve them.

This means keeping all teams coordinated and aligned around the product vision throughout the software development life cycle. It also involves determining the project scope and requirements that will best achieve these objectives.

Once shared goals are established, teams can work with each other instead of against each other even as they perform their own distinct tasks while pushing ahead in the same direction.

The responsibilities of everyone involved in the software development process need to be clearly defined to avoid clashes and create a sense of accountability. The clearer the roles, the less chance of confusion as teams go deeper into the project.

This is a time when having the right leadership can make all the difference. Leaders must clearly define roles and responsibilities and ensure that all teams understand how their work creates value for the organization and its customers.

Choose the appropriate project management methodology

Depending on the size and complexity of your project and teams, it’s important to choose a methodology that works well with the organization’s culture and values.

From Waterfall to Agile methodologies, there are many approaches you can choose. For large projects with a clearly defined start and end-point, the Waterfall approach would work best. However, for projects that are adaptable and divided to smaller sprints, an Agile approach is better suited.

This will serve as the foundation on how you begin to structure your teams and the type of tools they should implement in their daily workflows.

Invest in DevOps

Many software development companies organize themselves in a way that often leads to functional silos so that all the different teams tend to work in isolation while focused on their own goals.

As we’ve discussed, a project also consists of many moving parts and effective communication throughout the different stages of the project becomes complicated as there’s less visibility during these stages.

The DevOps methodology mainly grew out of frustration of the silos between teams, primarily development and operations teams. However, the term has evolved in modern software development to encompass a set of practices and tools that increase an organization’s ability to quickly deliver new software by promoting enhanced collaboration and communication between different teams.

DevOps goes beyond adopting the right tools to achieve high performance and better quality. It also involves undergoing a cultural transformation and teams adopting a shared culture and mindset that allows them to focus on quality of software and speed of delivery.

Choose the right tools

There are a number of collaborative tools your teams can adopt to establish efficient communication practices. The tools you choose will serve as the foundation to help teams work together towards common goals.

Among those are ones that can help you plan your projects from the bigger picture to the little details, which is especially useful for large organizations with multiple teams and team members collaborating. This also enables teams to have greater visibility over each member’s progress in the project and empowers teams to share input and encourage fast feedback loops in order to build a culture of DevOps and open communication.

Different teams will use their own tools to track work and progress throughout a project. For example, product teams will typically use roadmapping tools to plan, prioritize and track features while development teams will use development tools such as version control systems.

Therefore, there are many tools to choose from depending on the needs of the organization and teams from collaboration to project management tools as well as automation tools to streamline processes.

For example, teams adopting DevOps practices into their workflows will also need to choose the right stack of tools according to their unique business needs in order to implement DevOps successfully.

Communicate constantly and efficiently

From the onset of a project, all teams and other stakeholders will need to be involved in the decision-making process to keep issues down to a minimum and avoid miscommunication further down the road.

For example, many product teams often take the lead on setting the product vision and requirements without reviewing them with the development and engineering teams. In this scenario, it’s essential to understand that developers are the ones who will be writing for the product in question and so they must be consulted early on to give them the necessary context to build and prioritize the right features.

In this scenario, product managers and owners will need to effectively communicate the strategic direction and vision of a product with the help of a dedicated product roadmap. This will enable developers to understand why they’re building the product, its value and how it’s linked to overall business objectives.

A project often consists of many moving parts which means it might not be possible for teams to have full visibility over them. However, Many important decisions are collaborative in nature and require input from multiple sides. It’s imperative that everyone on the team is encouraged to actively provide feedback and share information about the progress of development. This also helps teams identify any potential roadblocks and take quick action to address them.

Set KPIs and metrics to track performance

Depending on the business objectives set at the beginning, teams will need to establish metrics to track and measure performance throughout the development process and beyond.

These metrics will be essential in the short and long term to make data-driven decisions to optimize products and improve team performance.

Teams can set any kind of metrics that will allow them to assess their efficiency. These could include productivity metrics such as velocity and cycle time, customer metrics such as customer satisfaction and net promoter scores and and more DevOps specific metrics such as DORA metrics.

Software development is a team effort

To produce high-quality software that meets your customers’ expectations and needs, different teams have to come together within a collaborative environment to solve complex problems.

If organizations work on fostering collaboration across the entire software development life cycle, software development teams can overcome challenges, maximize productivity and software quality as well as deliver better value to customers. It’s a win-win situation.

At the end of the day, all teams within an organization are looking to accomplish the same end-goals and outcomes, mainly to keep the business running smoothly and bring in profit as well as create top-notch products that customers will love.

You might also like...

See all

Article

8min read

A Practical Guide to the European Accessibility Act

Thomas Hein

May 23, 2025

Article

2min read

Your Domain, Your Rules: Domain Delegation by AB Tasty

AB Tasty

May 21, 2025

Article

3min read

Feedback Copilot: Turn The Customer Voice Into Action in Seconds with AI

AB Tasty

May 20, 2025

Subscribe to
our Newsletter

Article

Sep 15, 2023

3min read

Supercharge Your Coding Experience With AB Tasty’s VS Code Extension

AB Tasty

Throughout the coding and testing process, developers find themselves having to switch between a lot of different tools. At AB Tasty, we’ve understood that this can be a hassle which is why we’ve worked to make it easier to work between your coding environment in VS Code and the AB Tasty feature flagging platform.

AB Tasty’s new open beta for VS Code extension means that teams can work faster and take their coding experience to the next level.

This new extension will allow you to use AB Tasty Feature Experimentation and Rollouts, formerly Flagship, directly in the VS Code environment.

This means that you no longer have to switch between your Visual Studio code environment and your flags in the AB Tasty platform.

Why should you use AB Tasty’s VS Code extension?

With the VS Code extension, implementing feature flags in your codebase has never been simpler.

The extension enables you to seamlessly connect your Visual Studio Code environment with AB Tasty for full visibility over feature flags in your files and then retrieve the flag and its details directly in your code, which helps save time and eliminate complexity.

This significantly boosts productivity as you no longer need to switch between your coding environment and platform making the management and implementation of feature flags much easier.

Getting started

To be able to use this extension, you will need to have an account in AB Tasty. Then all you need to do is follow the steps below:

1)Create an access name on our Remote Control API

By creating an access to our Remote Control API, you’ll be able to manage the right scopes and get access to the extension’s features.

2)Save your client ID and client secret

You will then receive your “client ID” and “client secret”. These credentials will allow you to log in to the extension.

Ready to get started?

You’ll be able to find the extension on the Microsoft marketplace or in the extension marketplace directly in VS Code in order to download it.

Once the download process is complete, you can then follow the steps in our documentation to create a configuration and start using the extension.

Don’t forget to rate and review Flagship Code on the VS Code Marketplace to help us continue improving your coding experience.

AB Tasty code supports various programming languages and frameworks, making it adaptable to your tech stack – and if that’s not the case, feel free to get in touch.

You might also like...

See all

Article

8min read

A Practical Guide to the European Accessibility Act

Thomas Hein

May 23, 2025

Article

2min read

Your Domain, Your Rules: Domain Delegation by AB Tasty

AB Tasty

May 21, 2025

Article

3min read

Feedback Copilot: Turn The Customer Voice Into Action in Seconds with AI

AB Tasty

May 20, 2025

Subscribe to
our Newsletter

Article

Sep 8, 2023

8min read

The Ideal CRO Structure for Sustainable Growth

Emily Healy

With experimentation, the goal is simple: find out what resonates best with your digital audience to create a relationship and drive business growth. But, how do you reach the point of success?

Experimentation opens the door to fresh insights that are only found through testing, compelling you to continuously refine different facets of your website for an improved digital experience across the board. Once you take your first steps down your experimentation roadmap, your path toward optimization evolves to the point where you can become a more prominent digital player.

However, experimentation success will introduce growing pains – especially if you’re a company starting its CRO journey. Allocating your CRO resources early and efficiently is important to set your business up for continued success, prosperity, and evolution.

A firm foundation and building good habits from the start is the best way to ensure that your growth won’t stop.

How to build out your CRO team following the centralized model

A successful CRO team needs to be well-equipped with the necessary resources to carry out their missions which include time, tools, people and technology.

The first step in creating your team is to focus on leadership. The leader of your team needs to set an example by prioritizing experimentation and making it a part of your organization’s values. Your leader needs to value and encourage experimentation by creating a safe environment for testing where failures are seen as learning opportunities. CRO organizations need to create a culture of collaboration and communication where everyone works together to achieve a common goal.

It’s important to keep in mind that experimentation requires a lot of collaboration. By having a vast team equipped with different skills, you’ll need to facilitate communication between different teams, such as designers, developers, marketers, and data analysts.

This means that everyone needs to be aware of the goals and deliverables of each experiment, the roles of each stakeholder, the project timeline, and certainly if there are changes to the roadmap. This requires constant and open communication to keep everyone prepared. Each team member needs to be able to trust their teammates to perform certain tasks and have confidence in their own individual role.

With open communication and frequent regroups to check progress and share ideas, you can ensure that everyone is aligned and working towards the same objectives. Sharing results builds trust between team members and gives everyone an opportunity to celebrate wins, support each other through the learning opportunities and create a positive environment where feedback is welcome.

What is the ideal CRO team structure?

When picking the ideal structure for your CRO team, you have to keep in mind that this will vary depending on your organization’s size, goals, and resources at hand.

A small CRO team following the centralized model will need to have individuals responsible for covering all core responsibilities – from ideation to implementation to examination. Ideally, this would include:

CRO Manager
UX/UI Designer
Data Analyst
Web Developer
Content Specialist

To continue CRO team expansion, a medium-sized or large team should adopt the positions above and some or all positions listed below:

Product Manager
Product Designer
Data Scientist
Content Designer
Content Writer
Conversion Rate Optimizer/Tester
Technical Web Analyst
Website Animation Specialist

The skills needed to perform CRO are vast. A person equipped to be a great addition to your CRO team will most likely have a background in one of the following areas:

Chief Data Officer
Full Stack Developer
Functional Designer
Digital Marketing Specialists
Data Scientist (Specializing in CRO)
Web Analyst

Keep in mind that a CRO team is typically a cross-functional team and team members may be involved in other projects simultaneously. As each organization is completely unique, there are no hard and fast rules for the “perfect” team. Your ideal structure may shift as you go, reminding you of the importance of flexibility.

Rapid CRO growth

To put the rapid growth of CRO teams into context, let’s take a quick look at one global leader in the premium cosmetics industry: Shiseido.

Even though Shiseido already had a CRO team in place, they wanted to grow and turn their constricted experimentation strategy into an intuitive and scalable optimization program. They went from running four tests per year to over 10 tests per month using AB Tasty and expanded their team accordingly to cover more ground and expand their experimentation goals. Growth can happen quickly when setting new priorities and adopting a new mindset. See how Shiseido revitalized its experience optimization strategy with AB Tasty.

Steps for successful CRO implementation

Mindset shift

Building a culture of experimentation is crucial for a successful CRO organization. There needs to be a mindset shift towards data-driven decision-making, embracing bold decisions and viewing failure as an opportunity to learn and improve.

One of the most significant obstacles in establishing this culture is the fear and apprehension linked to failure. CRO teams need to recognize that failure is a natural part of the experimentation process and that every failed experiment provides valuable insights and learnings. By embracing what doesn’t work, CRO teams can create a culture that encourages experimentation and embraces risks.

All data derived from tests is valuable for building out future steps. The sooner an organization can adapt to this mentality, the more stable its CRO foundation will be.

LOOKING FOR MORE about the culture of experimentation?

Listen to the 1000 Experiments Club Podcast. The only podcast that interviews industry experts who have run over 1,000 experiments.

Set goals for your CRO team

CRO teams need to define exactly what they want to achieve through experimentation and how they will measure success. With this being said, data should be at the heart of all experimentation. Decisions should be made based on data collected and not only a gut feeling. By setting goals and assigning metrics to track progress, CRO teams can stay focused on their vision to achieve their objectives and track their progress.

Define the challenges of CRO implementation

There will be challenges to any success story. It’s important to address the potential challenges that may arise early on to keep your team prepared for any tough moments.

Barriers to continual success could include time restrictions, lack of adequate resources, employees with sub-par attitudes, pressure from HIPPOS, technology or anything that could potentially interfere with your roadmap.

After setting your goals and defining the next steps on your roadmap, it’s easier to outline the barriers that may prevent you from achieving those objectives, such as technical limitations or budget constraints.

Outline the team’s roles and responsibilities

Next, define the team’s roles and responsibilities. All team members should be aware of their personal objectives and how their work contributes to the overall success of the project (and their impact on the organization).

This includes identifying who will be responsible for testing, analyzing data, creating content, and making technical improvements to the website or app. Especially if team members have cross-functional roles where their time is divided, their responsibilities during each project should be clearly defined.

Standardize the A/B test process with your CRO Team

To standardize the A/B test process in your organization, there needs to be coordination of all digital teams around A/B tests and your overall CRO strategy. Your testing roadmap should outline the experiments your team will conduct, the hypotheses they will test, and the metrics they will use to measure success. By developing a testing plan, CRO teams can ensure that their experiments are aligned with their goals and that they are testing the right elements of the website or landing page.

With your new CRO team, it’s important to always start with identifying the most valuable tests at the right time. By brainstorming with your team to identify multiple elements, you will have various high-value optimization paths available to you when your team has the bandwidth.

When implementing a test, you must have a team ready to create the design and content for the test and another team available to put it all into production.

As a post-launch follow-up plan, you will need to develop an optimization plan to cater to the results.

Implement the winning variation – If your variation shows better results when compared to the original, plan for adequate time in your roadmap to incorporate any permanent changes.
Develop a new variation – Let’s say your variation wasn’t more influential than the original version. You’ve learned more about your audience that you can use in the future. If you’ve found what doesn’t work, leave room in your plans to go back to the drawing board to find a variation that resonates better with your audience.
Accept the original version – If you and your team are happy with the performance of the original version of your webpage, it’s time to move on to the next priority on your optimization list.
Re-challenge the winning variation – Consumer preferences are constantly changing. What worked 6 months ago might not resonate with your audience in the same way down the road. Plan time in your roadmap for more challenges to see continued success.

To promote communication, your experimentation roadmap and the results of each experiment should be accessible to everyone and promote transparency. This keeps your team aligned to standardize your process.

In CRO, you need to be adaptable. You won’t know the outcome of a test until it’s over (you don’t want to develop a bias by trying to guess the results either!). Based on the results, you and your team need to be ready to react quickly to follow the next steps of whichever path you choose.

A centralized CRO team built for sustainable growth

Developing a CRO team that’s built to grow and build a sustainable culture of experimentation is not the easiest task. There is always room for trial and error when figuring out what works best for your organization.

With a mindset shift, a well-equipped team, and a clear understanding of goals, barriers, and team roles, your organization will be prepped to carry out your winning strategy. With these elements in place, your organization can continuously test and optimize all digital e-commerce channels, leading to increased conversions, higher customer satisfaction, and ultimately, better business results.

You might also like...

See all

Article

8min read

A Practical Guide to the European Accessibility Act

Thomas Hein

May 23, 2025

Article

2min read

Your Domain, Your Rules: Domain Delegation by AB Tasty

AB Tasty

May 21, 2025

Article

3min read

Feedback Copilot: Turn The Customer Voice Into Action in Seconds with AI

AB Tasty

May 20, 2025

Breaking it down with a simple use case

Analysis in AB Tasty

Analysis in Google Analytics

Method 1: On a user segment/user scope

Method 2: On a session segment/session scope

Attribution model

Different attribution for different objectives

From campaign performance to holistic analysis: Which is the right solution for you?

You might also like...

Ruben de Boer explains what it takes to create a healthy testing environment that paves the way for better experimentation organization-wide

The goal should always be outcome-driven

Importance of cross-functional teams

A world of possibilities with AI

What else can you learn from our conversation with Ruben de Boer?

About Ruben de Boer

About 1,000 Experiments Club

You might also like...

What is customer experience optimization?

Why customer experience optimization is important for business growth

Challenges and solutions to customer experience optimization

Compiling the right data for accurate measurements

Developing the right hypothesis

Resource constraints

ROI tunnel vision

Not knowing what your customers want

Lack of customer experience optimization tools

How to improve the digital customer experience

Observe user behavior patterns

Create a journey map to understand the user flow

Develop a roadmap and set parameters to measure success

Experiment and re-challenge your past experiments

Customer-centric EXO

You might also like...

Why do we need to wait for the predetermined end of the experiment?

Sequential testing objection 1: “Each day has to be sampled the same”

Sequential testing objection 2: “Yes, (effect) size does matter”

When does sequential testing in CRO make sense?

Conclusion

You might also like...

How do I go about checking my experiments daily?

Can I also use this to identify winners faster?

But what about “dynamic allocation”?

Other benefits

You might also like...

Best Practice for Optimizing Mobile vs Desktop Experiences

Plan an Entire Digital Experience

Best Practice for the Mobile Digital Experience

1. Keep it accessible, always

2. Make sure your mobile content is consistent with desktop

3. Consider your mobile users’ needs

4. Check how your site and app perform across different devices

5. Make UX choices that encourage conversion and retention

Best Practice for Desktop Optimization

1. Create an engaging homepage

3. Develop intuitive site navigation

4. Make it easy to search for terms

5. Ready your desktop site for final steps of conversion and retention

Gain Valuable Insights with Heap

You might also like...

What is an inconclusive experiment result?

Setting thresholds for experimentation

The best tool: the confidence interval (CI)

The dangers of P-hacking in experimentation

Should you extend the experiment period often?

Digging deeper into ‘flatness’

What is the next step after receiving an inconclusive experimentation result?

Yielding an inconclusive experiment

You might also like...

Software development teams

Why is cross-functional collaboration important?

Software development team best practices

Define clear goals

Choose the appropriate project management methodology

Invest in DevOps

Choose the right tools

Communicate constantly and efficiently

Set KPIs and metrics to track performance

Software development is a team effort

You might also like...

Why should you use AB Tasty’s VS Code extension?