Category: Uncategorized

Article

Oct 18, 2024

11min read

Understanding the Digital Customer Journey

Emily Healy

In a highly competitive digital marketplace, optimizing your website for a unique and seamless digital customer journey is no longer just a competitive advantage — it’s a necessity.

It’s important to remember that the digital customer journey does not begin and end with a purchase – it’s a web of all customer interactions and touchpoints with your brand.

AB Tasty has mapped out seven customer phases that we consider crucial in the journey. To craft unique experiences, you’ll need to differentiate these seven phases customers pass through and understand how to animate their digital journey.

Once you have a better understanding of these phases, you will be better equipped to set your business goals and properly optimize your website for growth and impact.

Click to view the full-sized infographic in a new tab

How exactly can you optimize each phase of the digital customer journey? Let’s dive right in and take a look at some examples.

Phase 1: Awareness

When visitors land on your website for the first time, a great first impression is crucial. Your page needs to be both visually appealing and intuitive. A dynamic above the fold design is a great place to start.

In this first phase, it’s important to let your best ideas shine to capture and keep your visitors’ attention. You can accomplish this by creating personalized welcome messages for first-time visitors, displaying your value proposition and organizing high-impact elements for better visibility.

Let’s take a look at Just Over The Top’s experiment to modify the layout of their homepage. They used AB Tasty’s experience optimization platform to test if their users responded better seeing a layout with product categories rather than individual products.

Original:

Variation:

After creating a test variation to run against the original layout, they saw a 17.5% click increase on the three blocks below the hero image. This brought many more users into the second phase of their customer journey.

Phase 2: Discovery

When consumers reach the second phase, they’ve already discovered your brand and they’re getting curious.

To accommodate visitors during this phase, your website should be optimized for an excellent browsing experience. Whether this means making your search bar more visible, creating dynamic filters while searching, or using a virtual assistant to get to know your visitors’ interests with a series of questions, an easy browsing experience with intelligent search is key.

In this example, Claudie Pierlot focused on optimizing the customer browsing experience by testing the search bar visibility. In their variation, the small search icon was made more visible by adding the word “recherche” (or search in English) in the top right-hand corner.

Original:

Variation:

This clear above the fold design made it easier for visitors to identify the search bar to begin their browsing experience. With this simple A/B test, they saw a 47% increase in search bar clicks and a 7% increase in conversion rates coming directly from the search bar.

In another example, Villeroy & Boch, a ceramic manufacturing company, wanted to leverage intelligent search on their website. With the help of AB Tasty, they implemented an AI search algorithm to navigate online shoppers.

With our solution, they designed a new and intuitive navigation complete with filters and a comprehensive autosuggestion feature.

By changing their search functions, Villeroy & Boch saw a 33% increase in search results clicks and a 20% increase in sales through the search function.

Phase 3: Consideration

Now is the time when your visitors are considering your brand and which products they are interested in. Showcasing your product pages in their best light during the consideration phase might be exactly what your visitor needs to continue moving down the funnel.

Let’s look at how Hanna Anderson optimized their product pages during this phase.

The clothing retail company wanted to experiment with the images on their product listing pages. Previously, their toddler line had only images of clothing sizes for an older child. They were convinced there was room for improvement and decided to run a test by changing their images to include toddler sizes.

Original:

Variation:

After implementing age-appropriate clothing images, the results were clear. During this test, the clicks on PLPs increased by almost 8% and the purchase rate on those items skyrocketed by 22%.

Phase 4: Intent

During the intent phase, your visitors are on the verge of becoming customers but need to be convinced to make a purchase.

Social proof, urgency messaging, and bundling algorithms are a few ideas to lightly nudge visitors to add to cart or add more to cart.

Let’s take a look at the impact that urgency messaging can have: IZIPIZI, an eyewear retailer, decided to add a special message flag next to their product description to show viewers how many people have purchased this product. The idea of this message is to show viewers that this product is popular and to encourage them to take action.

With this simple sentence of social proof to validate a product’s desirability, they saw a 36% increase in add-to-basket rate.

In another scenario, you can see that adding a progress bar is a simple way to upsell. With a progress par, you are showing your customer how close they are to earning free shipping, which entices them to add more to their cart.

Vanessa Bruno experimented with this additive with the help of AB Tasty and saw a 3.15% increase in transactions and a €6 AOV uplift.

Phase 5: Purchase

Purchase frustration is real. If customers experience friction during checkout, you risk losing money.

Friction refers to any issues the visitors may encounter such as unclear messaging during the payment (did the payment actually go through?), confusing or expensive shipping options, discounts not working, double authentication check-in delays, difficult sign-in, and more.

Optimizing your checkout sequence for your audience with rollouts and KPI-triggered rollbacks can help you find a seamless fit for your website.

Let’s look at an example for this phase: Galeries Lafayette, the French luxury department store, saw an opportunity to optimize their checkout by displaying default payment methods that do not require double authentication.

During this test, they saw a €113,661 increase in profit, a €5 uplift in average order value, and a 38% increase in the conversion rate by adding the CB (bank card) option for a quicker checkout.

Phase 6: Experience

Optimizing the buyer experience doesn’t end after the purchase. Now is the time to grow your customer base and stop churn in its tracks. So, how do you keep your customers interested? By maintaining the same level of quality in your messages and personalization.

Let’s look at how Envie de Fraise, a French boutique, leveraged their user information to transform a normal post-purchase encounter into a personalized experience.

One of their customers had just purchased a maternity dress and had been browsing multiple maternity dresses prior to their purchase. By knowing this information, they experimented with using the “you will love these products” algorithm to gently nudge their customer to continue shopping.

With a customized recommendation like this, Envie de Fraise saw a €127K increase in their potential profit. As your customer spends more time with your brand, you will learn more about their habits and interests. The more time they spend with you, the more personalized you can make their experience.

Phase 7: Loyalty

In the final step of your customer’s journey, they move into the loyalty phase. To turn customers into champions of your brand, it’s important to remind them that you value their loyalty.

This can be done by sending emails with individual offers, social proof, product suggestions, or incentives for joining a loyalty program to earn rewards or complete product reviews.

Another example of this is sending a personalized email displaying items that are frequently bought together that align with their purchase. This will remind the customer about your brand and give them recommendations for future purchases.

Why Optimizing the Digital Customer Journey is Essential to Boost Conversions

The fierce competition in the e-commerce marketplace is undeniable. In order to attract and retain customers, you have to focus on crafting personalized user experiences to turn passive visitors into active buyers.

Understanding their needs in each phase and optimizing your digital space is your best solution to nudge visitors down the purchasing funnel.

By personalizing the experience of your customers during each phase of the digital customer journey, you can ensure an optimal shopping experience, boost purchases, increase customer satisfaction, and see more repeat customers.

AB Tasty is the best-in-class experience optimization platform that empowers you to create a richer digital experience – fast. From experimentation to personalization, this solution can help you activate and engage your audience to boost your conversions.

Using personalization to enhance your customer journey

With the vast array of products and brands to choose from, customer loyalty has become more important than ever. By focusing on personalizing the digital customer journey, you can reduce the chances of your customer abandoning their purchase or opting for another brand.

An individualized customer journey is beneficial for the following reasons:

Option overload: As online brands fight for the attention of consumers, it’s important to set your brand apart from the rest, with a customer journey that is tailored to their needs.
Analysis paralysis: With a plethora of information now readily available to customers who are researching and comparing potential purchases, your digital customer journey can help to deliver the information they need with ease and offer them the best shopping experience, thereby tipping the scale in your favor.
Lack of loyalty: The ease with which a customer can change service providers has increased the pressure and importance of meeting their needs during the entirety of the digital customer journey. Every interaction matters.

Offering a personalized experience elevates the customer journey and helps to ensure customer satisfaction. By leveraging the power of personalization, you can adapt the individual phases of the customer journey to each customer’s needs for an optimal e-commerce experience.

Personalization is the key to customer satisfaction

There is no way to deny the intense competition in the e-commerce space. Attracting and retaining customers is more difficult now than it has ever been. To advance ahead of the competition, you must understand customer needs and personalize each user journey with the help of a customer journey model.

By personalizing each experience your customers have with your brand, you can be sure to give your customers an optimal shopping experience, guarantee customer satisfaction, and encourage customer loyalty.

You might also like...

See all

Article

6min read

Unify GA4 with BigQuery to Strengthen Experiments

Katherine Weber

Nov 7, 2024

Article

1min read

5 Ecommerce Merchandising Strategies with Examples

Emily Healy

Nov 5, 2024

Article

5min read

Why AB Tasty Delivers 4x Faster

Leo Wiel

Nov 4, 2024

Subscribe to
our Newsletter

Article

Dec 7, 2023

8min read

How to Better Handle Collateral Effects of Experimentation: Dynamic Allocation vs Sequential Testing

Hubert Wassner

When talking about web experimentation, the topics that often come up are learning and earning. However, it’s important to remember that a big part of experimentation is encountering risks and losses. Although losses can be a touchy topic, it’s important to talk about and destigmatize failed tests in experimentation because it encourages problem-solving, thinking outside of your comfort zone and finding ways to mitigate risk.

Therefore, we will take a look at the shortcomings of classic hypothesis testing and look into other options. Basic hypothesis testing follows a rigid protocol:

Creating the variation according to the hypothesis
Waiting a given amount of time
Analyzing the result
Decision-making (implementing the variant, keeping the original, or proposing a new variant)

This rigid protocol and simple approach to testing doesn’t say anything about how to handle losses. This raises the question of what happens if something goes wrong? Additionally, the classic statistical tools used for analysis are not meant to be used before the end of the experiment.

If we consider a very general rule of thumb, let’s say that out of every 10 experiments, 8 will be neutral (show no real difference), one will be positive, and one will be negative. Practicing classic hypothesis testing suggests that you just accept that as a collateral effect of the optimization process hoping to even it out in the long term. It may feel like crossing a street blindfolded.

For many, that may not cut it. Let’s take a look at two approaches that try to better handle this problem:

Dynamic allocation – also known as “Multi Armed Bandit” (MAB). This is where traffic allocation changes for each variation according to their performance, implicitly lowering the losses.
Sequential testing – a method that allows you to stop a test as soon as possible, given a risk aversion threshold.

These approaches are statistically sound but they come with their assumptions. We will go through their pros and cons within the context of web optimization.

First, we’ll look into the classic version of these two techniques and their properties and give tips on how to mitigate some of their problems and risks. Then, we’ll finish this article with some general advice on which techniques to use depending on the context of the experiment.

Dynamic allocation (DA)

Dynamic allocation’s main idea is to use statistical formulas that modify the amount of visitors exposed to a variation depending on the variation’s performance.

This means a poor-performing variation will end up having little traffic which can be seen as a way to save conversions while still searching for the best-performing variation. Formulas ensure the best compromise between avoiding loss and finding the real best-performing variation. However, this implies a lot of assumptions that are not always met and that make DA a risky option.

There are two main concerns, both of which are linked to the time aspect of the experimentation process:

The DA formula does not take time into account

If there is a noticeable delay between the variation exposure and the conversion, the algorithm may go wrong resulting in a visitor being considered a ‘failure’ until they convert. This means that the time between a visit and a conversion will be falsely counted as a failure.

As a result, the DA will use the wrong conversion information in its formula so that any variation gaining traffic will automatically see a (false) performance drop because it will detect a growing number of non-converting visitors. As a result, traffic to that variation will be reduced.

The reverse may also be true: a variation with decreasing traffic will no longer have any new visitors while existing visitors of this variation could eventually convert. In that sense, results would indicate a (false) rise in conversions even when there are no new visitors, which would be highly misleading.

DA gained popularity within the advertising industry where the delay between an ad exposure and its potential conversion (a click) is short. That’s why it works perfectly well in this context. The use of Dynamic Allocation in CRO must be done in a low conversion delay context only.

In other words, DA should only be used in scenarios where visitors convert quickly. It’s not recommended for e-commerce except for short-term campaigns such as flash sales or when there’s not enough traffic for a classic AB test. It can also be used if the conversion goal is clicking on an ad on a media website.

DA and the different days of the week

It’s very common to see different visitor behavior depending on the day of the week. Typically, customers may behave differently on weekends than during weekdays.

With DA, you may be sampling days unevenly, implicitly giving more weight on some days for some variations. However, you should weigh each day the same because, in reality, you have the same amount of weekdays. You should only use Dynamic Allocation if you know that the optimized KPI is not sensitive to fluctuations during the week.

The conclusion is that DA should be considered only when you expect too few total visitors for classic A/B testing. Another requirement is that the KPI under experimentation needs a very short conversion time and no dependence on the day of the week. Taking all this into account: Dynamic Allocation should not be used as a way to secure conversions.

Sequential Testing (ST)

Sequential Testing is when a specific statistical formula is used enabling you to stop an experiment. This will depend on the performance of variations with given guarantees on the risk of false positives.

The Sequential Testing approach is designed to secure conversions by stopping a variation as soon as its underperformance is statistically proven.

However, it still has some limitations. When it comes to effect size estimation, the effect size may be wrong in two senses:

Bad variations will be seen as worse than they really are. It’s not a problem in CRO because the false positive risk is still guaranteed. This means that in the worst-case scenario, you will discard not a strictly losing variation but maybe just an even one, which still makes sense in CRO.
Good variations will be seen as better than they really are. It may be a problem in CRO since not all winning variations are useful for business. The effect size estimation is key to business decision-making. This can easily be mitigated by using sequential testing to stop losing variations only. Winning variations, for their part, should be continued until the planned end of the experiment, ensuring both correct effect size estimation and an even sampling for each day of the week.
It’s important to note that not all CRO software use this hybrid approach. Most of them use ST to stop both winning and losing variations, which is wrong as we’ve just seen.

As we’ve seen, by stopping a losing variation in the middle of the week, there’s a risk you may be discarding a possible winning variation.

However, to actually have a winning variation after ST has shown that it’s underperforming, this variation will need to perform so well that it becomes even with the reference. Then, it would also have to perform so well that it outperforms the reference and all that would need to happen in a few days. This scenario is highly unlikely.

Therefore, it’s safe to stop a losing variation with Sequential Testing, even if all weekdays haven’t been evenly sampled.

The best of both worlds in CRO

Dynamic Allocation is the best approach to experimentation instead of static allocation when you expect a small volume of traffic. It should be used only in the context of ‘short delay KPI’ and with no known weekday effect (for example: flash sales). However, it’s not a way to mitigate risk in a CRO strategy.

To be able to run experiments with all the needed guarantees, you need a hybrid system using Sequential Testing to stop losing variations and a classic method to stop a winning variation. This method will allow you to have the best of both worlds.

You might also like...

See all

Article

6min read

Unify GA4 with BigQuery to Strengthen Experiments

Katherine Weber

Nov 7, 2024

Article

1min read

5 Ecommerce Merchandising Strategies with Examples

Emily Healy

Nov 5, 2024

Article

5min read

Why AB Tasty Delivers 4x Faster

Leo Wiel

Nov 4, 2024

Subscribe to
our Newsletter

Article

Nov 22, 2023

9min read

Harmony or Dissonance: Decoding Data Divergence Between AB Tasty and Google Analytics

Julie Dumont

The world of data collection has grown exponentially over the years, providing companies with crucial information to make informed decisions. However, within this complex ecosystem, a major challenge arises: data divergence.

Two analytics tools, even if they seem to be following the same guidelines, can at times produce different results. Why do they differ? How do you leverage both sets of data for your digital strategy?

In this article, we’ll use a concrete example of a user journey to illustrate differences in attribution between AB Tasty and Google Analytics. GA is a powerful tool for gathering and measuring data across the entire user journey. AB Tasty lets you easily make changes to your site and measure the impact on specific goals.

Navigating these differences in attribution strategies will explain why there can be different figures across different types of reports. Both are important to look at and which one you focus on will depend on your objectives:

Specific improvements in cross-session user experiences
Holistic analysis of user behavior

Let’s dive in!

Breaking it down with a simple use case

We’re going to base our analysis on a deliberately very basic use case, based on the user journey of a single visitor.

Campaign A is launched before the first session of the visitor and remains live until the end which occurs after the 3rd session of the visitor.

Here’s an example of the user journey we’ll be looking at in the rest of this article:

Session 1: first visit, Campaign A is not triggered (the visitor didn’t match all of the targeting conditions)
Session 2: second visit, Campaign A is triggered (the visitor matched all of the targeting conditions)
Session 3: third visit, no re-triggering of Campaign A which is still live, and the user carries out a transaction.

NB A visitor triggers a campaign as soon as they meet all the targeting conditions:

They meet the segmentation conditions

During their session, they visit at least one of the targeted pages

They meet the session trigger condition.

In A/B testing, a visitor exposed to a variation of a specific test will continue to see the same variation in future sessions, as long as the test campaign is live. This guarantees reliable measurement of potential changes in behavior across all sessions.

We will now describe how this user journey will be taken into account in the various AB Tasty and GA reports.

Analysis in AB Tasty

In AB Tasty, there is only one report and therefore only one attribution per campaign.

The user journey above will be reported as follows for Campaign A:

Total Users (Unique visitors) = 1, based on a unique user ID contained in a cookie; here there is only one user in our example.
Total Session = 2, s2 and s3, which are the sessions that took place during and after the display of Campaign A, are taken into account even if s3 didn’t re-trigger campaign A
Total Transaction = 1, the s3 transaction will be counted even if s3 has not re-triggered Campaign A.

In short, AB Tasty will collect and display in Campaign A reporting all the visitor’s sessions and events from the moment the visitor first triggered the campaign.

Analysis in Google Analytics

The classic way to analyze A/B test results in GA is to create an analysis segment and apply it to your reports.

However, this segment can be designed using 2 different methods, 2 different scopes, and depending on the scope chosen, the reports will not present the same data.

Method 1: On a user segment/user scope

Here we detail the user scope, which will include all user data corresponding to the segment settings.

In our case, the segment setup might look something like this:

This segment will therefore include all data from all sessions of all users who, at some point during the analysis date range, have received an event with the parameter event action = Campaign A.

We can then see in the GA report for our user journey example:

Total User = 1, based on a user ID contained in a cookie (like AB Tasty); here there is only one user in our example
Total Session = 3, s1, s2 and s3 which are the sessions created by the same user entering the segment and therefore includes all their sessions
Total Transaction = 1, transaction s3 will be counted as it took place in session s3 after the triggering of the campaign.

In short, in this scenario, Google Analytics will count and display all the sessions and events linked to this single visitor (over the selected date range), even those prior to the launch of Campaign A.

Method 2: On a session segment/session scope

The second segment scope detailed below is the session scope. This includes only the sessions that correspond to the settings.

In this second case, the segment setup could look like this:

This segment will include all data from sessions that have, at some point during the analysis date range, received an event with the parameter event action = Campaign A.

As you can see, this setting will include fewer sessions than the previous one.

In the context of our example:

Total User = 1, based on a user ID contained in a cookie (like AB Tasty), here there’s only one user in our example
Total Session = 1, only s2 triggers campaign A and therefore sends the campaign event
Total Transaction = 0, the s3 transaction took place in the s3 session, which does not trigger campaign A and therefore does not send an event, so it is not taken into account.

In short, in this case, Google Analytics will count and display all the sessions – and the events linked to these sessions – that triggered campaign A, and only these.

Attribution model

Tool – scope	Counted in the selected timeframe
AB Tasty	All sessions and events that took place after the visitor first triggered campaign A
Google Analytics – user scope	All sessions and events of a user that triggered campaign A at least once during one their sessions
Google Analytics – session scope	Only sessions that have triggered campaign A

Different attribution for different objectives

Depending on the different attributions of the various reports, we can observe different figures without the type of tracking being different.

The only metric that always remains constant is the sum of Users (Unique visitors in AB Tasty). This is calculated in a similar (but not identical) way between the 2 tools. It’s therefore the benchmark metric, and also the most reliable for detecting malfunctions between A/B testing tools and analytics tools with different calculations.

On the other hand, the attribution of sessions or events (e.g. a transaction) can be very different from one report to another. All the more so as it’s not possible in GA to recreate a report with an attribution model similar to that of AB Tasty.

Ultimately, A/B test performance analysis relies heavily on data attribution, and our exploration of the differences between AB Tasty and Google Analytics highlighted significant distinctions in the way these tools attribute user interactions. These divergences are the result of different designs and distinct objectives.

From campaign performance to holistic analysis: Which is the right solution for you?

AB Tasty, as a solution dedicated to the experimentation and optimization of user experiences, stands out for its more specialized approach to attribution. It offers a clear and specific view of A/B test performance, by grouping attribution data according to campaign objectives.

Making a modification on a platform and testing it aims to measure the impact of this modification on the performance of the platform and its metrics, during the current session and during future sessions of the same user.

On the other hand, Google Analytics focuses on the overall analysis of site activity. It’s a powerful tool for gathering data on the entire user journey, from traffic sources to conversions. However, its approach to attribution is broader, encompassing all session data, which can lead to different data cross-referencing and analysis than AB Tasty, as we have seen in our example.

It’s essential to note that one is not necessarily better than the other, but rather adapted to different needs.

Teams focusing on the targeted improvement of cross-session user experiences will find significant value in the attribution offered by AB Tasty.
On the other hand, Google Analytics remains indispensable for the holistic analysis of user behavior on a site.

The key to effective use of these solutions lies in understanding their differences in attribution, and the ability to exploit them in complementary ways. Ultimately, the choice will depend on the specific objectives of your analysis, and the alignment of these tools with your needs will determine the quality of your insights.

You might also like...

See all

Article

6min read

Unify GA4 with BigQuery to Strengthen Experiments

Katherine Weber

Nov 7, 2024

Article

1min read

5 Ecommerce Merchandising Strategies with Examples

Emily Healy

Nov 5, 2024

Article

5min read

Why AB Tasty Delivers 4x Faster

Leo Wiel

Nov 4, 2024

Subscribe to
our Newsletter

Article

Nov 7, 2023

6min read

Taking an Outcome-Driven Approach | Ruben de Boer

Rowan Haddad

Ruben de Boer explains what it takes to create a healthy testing environment that paves the way for better experimentation organization-wide

Ruben de Boer is a lead CRO Manager and consultant with over 14 years of experience in data and optimization. At Online Dialogue, Ruben leads the Conversion Managers team, developing team skills and quality as well as setting the team strategy and goals. He spreads his knowledge far both as a teacher with Udemy with over 12,000 students and as a public speaker on topics such as experimentation, change management, CRO and personal growth.

In 2019, Ruben founded his company, Conversion Ideas, where he helps people kick start their career in Conversion Rate Optimization and Experimentation by providing affordable, high-quality online courses and a number of resources.

AB Tasty’s VP Marketing Marylin Montoya spoke with Ruben about exciting trends and evolutions within the world of experimentation, including the various ways AI can impact the optimization of the experimentation process. Ruben also shares ways to involve cross-functional teams to implement a successful culture of experimentation within the organization and why it’s important to steer these teams towards an outcome- rather than an output-driven mindset.

Here are some key takeaways from their conversation.

The goal should always be outcome-driven

Based on his experience, Ruben believes that one of the biggest pitfalls companies face when trying to kick start their experimentation journey is they focus more on outputs rather than outcomes.

“When a company is still very much in an output mindset, meaning we have to deliver an X amount of sprint points per sprint and we have to release so many new features this year, then of course experimentation can be seen as something that slows it down, right? Let’s say as a rule of thumb, 25% of A/B tests or experiments result in a winner and so 75% of what was built will not be released, which means the manager does not get the output goals.”

In this scenario, experimentation becomes an obstacle that slows down these outputs. Whereas, when a company shifts towards an outcome mindset, it makes more sense to run experiments with the goal to create more value for the customer. With an outcome-mindset, teams embrace experimentation with customers at the heart of the process.

When teams are more outcome-oriented, the product is based more on research and experiments instead of a fixed long-term roadmap. According to Ruben, it’s vital that companies adopt such a way of working as it helps create better products and business outcomes, which ultimately helps them maintain their competitive advantage.

Importance of cross-functional teams

Ruben argues that experimentation is maturing in that it’s becoming more embedded within product teams.

He notes there’s a rising trend of different teams working together, which Ruben believes is essential for knowledge sharing when it comes to learning new things about the customer journey and the product itself. For Ruben, this helps create an ideal, healthy experimentation environment for teams to experiment better and get the results they want.

Ideally, there would be experts in experimentation coming in from different teams sharing knowledge, ideas and insights on a regular basis which helps drive inspiration and innovation when it comes to future test ideas.

The recipe behind the success of these experimentation teams varies and depends on the maturity of the experimentation program and the skills of these teams.

This could start with a look into the culture of the organization by sending questionnaires to various teams to understand their work process and how autonomous they are. This analysis would also help teams to understand what their current state of experimentation is like such as how accepting they are of experimentation. This helps to devise a strategy and roadmap to successfully implement a culture of experimentation throughout the whole organization.

This culture scan also helps determine the maturity of an experimentation program.

“Process, data, team, scope, alignment, and company culture: that’s what I generally look at when I assess the maturity of an organization. Is there a CRO specialist throughout the different product teams? How’s decision-making being done by leadership? Is it based on the HIPPO decisions or fully based on experimentation? Then there’s the outcome versus output mindset, the scope and alignment of experimentation as well as the structure of the team- is it just a single CRO specialist or a multidisciplinary team? What does the process look like? Is it just a single CRO process or is it a process embedded in a project team?” Ruben says.

A world of possibilities with AI

With the advent of AI technology, Ruben believes there’s a lot of possibilities with what can be done with it, particularly in the experimentation process.

While he admits it’s still too early to speculate and that there are also the many privacy concerns that come with such technology, he believes AI can bring a lot of exciting things in the future.

“It would be so nice to have an AI go over experiments on the product detail page with all the results and all the learnings, and just ask the AI, what did I actually learn and what would be good follow up experiments on that? And that would be enormously interesting to have an AI run through all the experiments in the database,” Ruben says.

Therefore, Ruben admits there are a number of possibilities of what teams can do when it comes to designing experiments and saving time and steps in the experimentation process.

“And just think about maybe three or four years from now, everyone will just have an AI app on their phone and say, I need to buy this and I will buy it for you. And maybe a website with only AI apps on it to purchase stuff, who knows? And then optimization becomes very different all of a sudden.”

There’s also significant potential with AI when it comes to changing the way people work as well as provide inspiration and ultimately optimize and bring innovation to the experimentation process.

“Maybe based on all the input we give from chat logs, social media channels, reviews, surveys, we can make the AI behave like a user at some point in the future somewhere, which you then don’t have to run user tests anymore because you just let AI see your website.”

What else can you learn from our conversation with Ruben de Boer?

Evolving trends in experimentation
His take on change management to help organizations adopt experimentation
His own experiences with building cross-functional teams
How to tackle resistance when it comes to experimentation

About Ruben de Boer

With over 14 years of experience as a lead CRO manager and consultant in data and optimization, Ruben is a two-time winner in the Experimentation Elite Awards 2023 and a best-selling instructor on Udemy with over 12,000 students. He is also a public speaker on topics such as experimentation culture, change management, conversion rate optimization, and personal growth. Today, Ruben is the Lead Conversion Manager responsible for leading the Conversion Managers team, developing team skills and quality, setting the team strategy and goals, and business development.

About 1,000 Experiments Club

The 1,000 Experiments Club is an AB Tasty-produced podcast hosted by Marylin Montoya, CMO at AB Tasty. Join Marylin and the Marketing team as they sit down with the most knowledgeable experts in the world of experimentation to uncover their insights on what it takes to build and run successful experimentation programs.

You might also like...

See all

Article

6min read

Unify GA4 with BigQuery to Strengthen Experiments

Katherine Weber

Nov 7, 2024

Article

1min read

5 Ecommerce Merchandising Strategies with Examples

Emily Healy

Nov 5, 2024

Article

5min read

Why AB Tasty Delivers 4x Faster

Leo Wiel

Nov 4, 2024

Subscribe to
our Newsletter

Article

Oct 24, 2023

10min read

How to Effectively Use Sequential Testing

Hubert Wassner

In A/B tests where you can see the data coming in a continuous stream, it’s tempting to stop the experiment before the planned end. It’s so tempting that in fact a lot of practitioners don’t even really know why one has to define a testing period beforehand.

Some platforms have even changed their statistical tools to take this into account and have switched to sequential testing which is designed to handle tests this way.

Sequential testing enables you to evaluate data as it’s collected to determine if an early decision can be made, helping you cut down on A/B test duration as you can ‘peak’ at set points.

But, is this an efficient and beneficial type of testing? Spoiler: yes and no, depending on the way you use it.

Why do we need to wait for the predetermined end of the experiment?

Planning and respecting the data collection period of an experiment is crucial. Historical techniques use “fixed horizon testing” that establishes these guidelines for all to follow. If you do not respect this condition, then you don’t have the guarantee provided by the statistical framework. This statistical framework guarantees that you only have a 5% error risk when using the common decision thresholds.

Sequential testing promises that when using the proper statistical formulas, you can stop an experiment as soon as the decision threshold is crossed and still have the 5% error risk guarantee. The test user here is the sequential Z-test, which is based on the classical Z-test with an added correction to take the sequential usage into account.

In the following sections, we will look at two objections that are often raised when it comes to sequential testing that may put it at odds with CRO practices.

Sequential testing objection 1: “Each day has to be sampled the same”

The first objection is that one should sample each day of the week the same way. This is basically to have a sampling that represents reality. This is the case in a classic A/B test. However, this rule may be broken if you use sequential testing since you can stop the test mid-week but this is not always applicable. Since in reality there are seven different days, your sampling unit should be by week and not by day to account for behavioral differences over the course of a week.

As experiments typically last 2-3 weeks, then the promise of sequential testing saving days isn’t necessarily correct unless a winner appears very early in the process. However, it’s more likely that the statistical test yielded significance during the last week. In this case, it’s best to complete the data collection until each day is sampled evenly so that the full period is covered.

Let’s consider the following simulation setting:

One reference with a 5% conversion rate
One variation with a 5.5% conversion rate (a 10% relative improvement)
5,000 visitors as daily traffic
14 days (2 weeks) of data collection
We ran thousands of such experiments to get histograms for different decision index

In the following histogram, the horizontal axis is the day when the sequential testing crosses the significance threshold. The vertical axis is the ratio of experiments which stopped on this day.

In this setting, day 10 is the most likely day for the sequential testing to reach significance. This means that you will need to wait until the planned end of the test to respect the “same sampling each day” rule. And it’s very unlikely that you will get a significant positive result in one week. Thus, in practice, determining the winner sooner with sequential testing doesn’t apply in CRO.

Sequential testing objection 2: “Yes, (effect) size does matter”

In sequential testing, this is often a less obvious problem and may need some further clarification to be properly understood.

In CRO, we consider mainly two statistical indices for decision-making:

The pValue or any other confidence index, which is linked to the fact that there exists (or not) a difference between the original and the variation. This index is used to validate or invalidate the test hypothesis. But a validated hypothesis is not necessarily a good business decision, so we need more information.
The Confidence Interval (CI) around the estimated gain, which indicates the size of the effect. It’s also central to business decisions. For instance, a variation can be a clear winner but with a very little margin that may not cover the implementation or operating costs such as coupon offerings that need to cover the coupon cost.

Confidence intervals can be seen as a best and worst case scenario. For example a CI = [1% ; 12%] means “in the worst case you will only get 1% relative uplift,” which means going from 5% conversion rate to 5.05%.

If the variation has an implementation or operating cost, the results may not be satisfying. In that case, the solution would be to collect more data in order to have a narrower confidence interval, until you get a more satisfying lower bound, or you may find that the upper bound goes very low showing that the effect, even if it exists, is too low to be worth it from a business perspective.

Using the same scenario as above, the lower bound of the confidence interval can be plotted as follows:

Horizontal axis – the percentage value of the lower bound
Vertical axis – the proportion of experiments with this lower bound value
Blue curve – sequential testing CI
Orange curve – classical fixed horizon testing

We can see that sequential testing has a very low confidence interval for the lower bound. Most of the time, this is lower than 2% (in relative gain, which is very small). This means that you will get very poor information for business decisions.

Meanwhile, a classic fixed horizon testing (orange curve) will produce a lower bound >5% in half of the cases, which is a more comfortable margin. Therefore, you can continue the data collection until you have a useful result, which means waiting for more data. Even if by chance the sequential testing found a variant reaching significance in one week, you will still need to collect data for another week to do two things: have a useful estimation of the uplift and sample each day equally.

This makes sense in light of the purpose of sequential testing: quickly detect when a variation produces results that differ from the original, whether for the worse or better.

If done as soon as possible, it makes sense to stop the experiment as soon as the gain confidence interval lays mostly either on the positive or negative side. Then, for the positive side, the CI lower bound is close to 0, which doesn’t allow for efficient business decisions. It’s worth noting that for other applications other than CRO, this behaviour may be optimal and that’s why sequential testing exists.

When does sequential testing in CRO make sense?

As we’ve seen, sequential testing should not be used to quickly determine a winning variation. However, it can be useful in CRO in order to detect losing variations as soon as possible (and prevent loss of conversions, revenue, …).

You may be wondering why it’s acceptable to stop an experiment midway through when your variation is losing rather than when you have a winning variation. This is because of the following reasons:

The most obvious one: To put it simply, you’re losing conversions. This is acceptable in the context of searching for a better variation than the original. However, this makes little sense in cases where there is a notable loss, indicating that the variation has no more chances to be a winner. An alerting system set at a low sensitivity level will help detect such impactful losses.
The less obvious one: Sometimes when an experiment is only slightly “losing” for a good period of time, practitioners tend to let this kind of test run in the hopes that it may turn into a “winner”. Thus, they accept this loss because the variation is only “slightly” losing but they often forget that another valuable component is lost in the process: traffic, which is essential for experimentation. For an optimal CRO strategy, one needs to take these factors into account and consider stopping this kind of useless experiment, doomed to have small effects. In such a scenario, an automated alert system will suggest stopping this kind of test and allocate this traffic to other experiments.

Therefore, sequential testing is, in fact, a valuable tool to alert and stop a losing variation.

However, one more objection could still be raised: by stopping the experiment midway, you are breaking the “sample each day the same” rule.

In this particular case, stopping a losing variation has very little chance to be a bad move. In order for the detected variation to become a winner, it first needs to gain enough conversions tobe comparable to the original version. Then it would need another set of conversions to be a “mild” winner and that still wouldn’t be enough to be considered a business winner (and cover the implementation or exploitation costs of that winner). To be considered a winner for your business, the competing variation will need another high amount of conversions with a sufficient margin. This margin needs to be high enough to cover the cost of implementation, localization, and/or operating costs.

All the aforementioned events should happen in less than a week (ie. the number of days needed to complete the current week). This is very unlikely, which means it’s safe and smart to stop such experiments.

Conclusion

It may be surprising or disappointing to see that there’s no business value in stopping winning experiments early as others may believe. This is because a statistical winner is not a business winner. Stopping a test early is taking away the data you need to reach a significant effect size that would increase your chances of getting a winning variation.

With that in mind, the best way to use this type of testing is as an alert to help spot and stop tests that are either harmful to the business or not worth continuing.

About the Author:

Hubert Wassner has been working as a Senior Data Scientist at AB Tasty since 2014. With a passion for science, data and technology, his work has focused primarily on all the statistical aspects of the platform, which includes building Bayesian statistical tests adapted to the practice of A/B testing for the web and setting up a data science team for machine learning needs.

After getting his degree in Computer Science with a speciality in Signal Processing at ESIEA, Hubert started his career as a research engineer doing research work in the field of voice recognition in Switzerland followed by research in the field of genomic data mining at a biotech company. He was also a professor at ESIEA engineer school where he taught courses in algorithmics and machine learning.

You might also like...

See all

Article

6min read

Unify GA4 with BigQuery to Strengthen Experiments

Katherine Weber

Nov 7, 2024

Article

1min read

5 Ecommerce Merchandising Strategies with Examples

Emily Healy

Nov 5, 2024

Article

5min read

Why AB Tasty Delivers 4x Faster

Leo Wiel

Nov 4, 2024

Subscribe to
our Newsletter

Article

Oct 17, 2023

4min read

Doing CRO at Scale? AB Tasty’s Testing Alerts Prevent Revenue Loss

Hubert Wassner

When talking about CRO at scale, a common fear is, what if some experiments go wrong? And we mean seriously wrong. The more experiments you have running, the higher the chance that one could negatively impact the business. It might be a bug in a variant, or it might just be a bad idea – but conversion can drop dramatically, hurting your revenue goals.

That’s why it makes sense to monitor experiments daily even if the experiment protocol says that one shouldn’t make decisions before the planned end of the experiment. This leads to two questions:

How do you make a good decision using statistical indices that are not meant to be used that way? Checking it daily is known as “P-hacking” or “data peeking” and generates a lot of poor decisions.
How do you check tens of experiments daily without diverting the team’s time and efforts?

The solution is to use another statistical framework, adapted to the sequential nature of checking an experiment daily. This is known as sequential testing.

How do I go about checking my experiments daily?

Good news, you don’t have to do this tedious job yourself. AB Tasty has developed an automatic alerting system, based on a specialized statistical test procedure, to help you launch experiments at scale with confidence.

Sequential Testing Alerts offer a sensitivity threshold to detect underperforming variations early. It triggers alerts and can pause experiments, helping you cover your losses. It enables you to make data-driven decisions swiftly and optimize user experiences effectively. This alerting system is based on the percentage of conversions and/or traffic lost.

When creating a test, you simply check a box, select the sensitivity level and you’re done for the rest of the experiment’s life. Then, if no alert has been received, you can go ahead and analyze your experiment as usual; but in case of a problem, you’ll be alerted through the notification center. Then you head on over to the report page to assess what happened and make a final decision about the experiment.

Can I also use this to identify winners faster?

No; this may sound weird but it’s not a good idea, even if some vendors are telling you that you should do it. You may be able to spot a winner faster but the estimation of the effect size will be very poor, preventing you from making wise business decisions.

But what about “dynamic allocation”?

First of all, what is dynamic allocation? As its name suggests, it is an experimentation method where the allocation of traffic is not fixed but depends on the variation’s performance. While this may be seen as a way to limit the effect of bad variations, we don’t recommend using it for this purpose for several reasons:

As you know, the strict testing protocol strongly suggests not to change the allocation during a test, so dynamic allocation is an edge case of A/B testing. We only suggest using it when you have no other choice. Classic use cases are very low traffic, or very short time frames (flash sales, for instance). So if your only concern is avoiding big losses from a bad variation, alerting is a better and safer option that respects the A/B testing protocol.
Since dynamic allocation changes the allocation, in order to avoid the risk of missing out on a winner, it always keeps a minimum traffic amount for the supposed losing variation, even if it’s losing by a lot.

Therefore, dynamic allocation isn’t a way to protect revenue but rather to explore options in the context of limited traffic.

Other benefits

Another benefit of the sequential testing alert feature is better usage of your traffic. Experiments with a lot of traffic with no chance to win are also detected by the alerting system. The system will suggest to the user to stop that experiment and make better use of the traffic for other experiments.

The bottom line is: you can’t do CRO at scale without an automated alert system. Receiving such crucial alerts can protect you from experiments that could otherwise cause revenue loss. To learn more about our sequential testing alert and how it works, refer to our documentation.

You might also like...

See all

Article

6min read

Unify GA4 with BigQuery to Strengthen Experiments

Katherine Weber

Nov 7, 2024

Article

1min read

5 Ecommerce Merchandising Strategies with Examples

Emily Healy

Nov 5, 2024

Article

5min read

Why AB Tasty Delivers 4x Faster

Leo Wiel

Nov 4, 2024

Subscribe to
our Newsletter

Article

Oct 5, 2023

9min read

Inconclusive A/B Test Results – What’s Next?

Hubert Wassner

Have you ever had an experiment leave you with an unexpected result and were unsure of what to do next? This is the case for many when receiving neutral, flat, or inconclusive A/B test results and this is a question we aim to answer.

In this article, we are going to discuss what an inconclusive experimentation result is, what you can learn from it, and what the next step is when you receive this type of result.

What is an inconclusive experiment result?

We have two definitions for an inconclusive experiment: a practitioner’s answer and a more broken-down answer. A basic practitioner’s answer is a numerical answer that shows statistical information depending on the platform you’re using:

The probability of a winner is less than 90-95%
The pValue is bigger than 0.05
The lift confidence interval includes 0

In other words, an inconclusive result happens when the results of an experiment are non-statistically significant or an uplift is too small to be measured.

However, let’s take note of the true meaning of “significance” in this case: the significance is the threshold one has previously set as a metric or a statistic for measurement. If this previously set threshold is crossed, then an action will be made, usually implementing the winning variation.

Setting thresholds for experimentation

It’s important to note that the user sets the threshold and there are no magic formulas for calculating a threshold value. The only mandatory thing that must be done is that the threshold must be set before the beginning of an experiment. In doing so, this statistical hypothesis protocol provides caution and mitigates the risks of making a poor decision or missing an opportunity during experimentation.

To set a proper threshold, you will need a mix of statistical and business knowledge considering the context.

There is no golden rule, but there is a widespread consensus for using a “95% significance threshold.” However, it’s best to use this generalization cautiously as using the 95% threshold may be a bad choice in some contexts.

To make things simple, let’s consider that you’ve set a significance threshold that fits your experiment context. Then, having a “flat” result may have different meanings – we will dive into this more in the following sections.

The best tool: the confidence interval (CI)

The first thing to do after the planned end of an experiment is to check the confidence interval (CI) that can tell useful information without any notion of significance. The usage is a 95% confidence level to build these intervals. This means that there is a 95% chance that the real value lies between its boundaries. You can consider the boundaries to be an estimate of the best and worst-case scenarios.

Let’s say that your experiment is collaborating with a brand ambassador (or influencer) to attract more attention and sales. You want to see the impact the brand ambassador has on the conversion rate. There are several possible scenarios depending on the CI values:

Scenario 1:

The confidence interval of the lift is [-1% : +1%]. This means that in the best-case scenario, this ambassador effect is a 1% gain and in the worst-case scenario, the effect is -1%. If this 1% relative gain is less than the cost of the ambassador, then you know that it’s okay to stop this collaboration.

A basic estimation can be done by taking this 1% of your global revenue from an appropriate period. If this is smaller than the cost of the ambassador, then there is no need for “significance“ to validate the decision – you are losing money.

Sometimes neutrality is a piece of actionable information.

Scenario 2:

The confidence interval of the lift is [-1% : +10%]. Although this sounds promising, it’s important not to make quick assumptions. Since the 0 is still in the confidence interval, you’re still unsure if this collaboration has a real impact on conversion. In this case, it would make sense to extend the experiment period because there are more chances that the gain will be positive than negative.

It’s best to extend the experimentation period until the left bound gets to a “comfortable” margin.

Let’s say that the cost of the collaboration is covered if the gain is as small as 3%, then any CI [3%, XXX%] will be okay. With a CI like this, you are ensuring that the worst-case scenario is even. And with more data, you will also have a better estimate of the best-case scenario, which will certainly be lower than the initial 10%.

Important notice: do not repeat this too often, otherwise you may be waiting until your variant beats the original just by chance.

When extending a testing period, it’s safer to do it by looking at the CI rather than the “chances to win” or P-value, because the CI provides you with an estimate of the effect size. When the variant wins only by chance (which you increase when extending the testing period), it will yield a very small effect size.

You will notice the size of the gain by looking at the CI, whereas a p-value (or any statistical index) will not inform you about the size. This is a known statistical mistake called p-hacking. P-hacking is basically running an experiment until you get what you expect.

The dangers of P-hacking in experimentation

It’s important to be cautious of p-hacking. Statistical tests are meant to be used once. Splitting the analysis into segments, to some extent, can be seen as portraying different experiences. Therefore, if making a unique decision at a 95% significance level means accepting a 5% risk of having a false positive, then checking for 2 segments implicitly leads to doubling this risk to 10% (roughly).

We recommend the following advice may help to mitigate this risk:

Limit the number of segments you are studying to only segments that could have a reason to interact differently with the variation. For example: if it’s a user interface modification (such as the screen size or the navigator used), it may have an impact on how the modification is displayed, but not the geolocation.
Use segments that convey strong information regarding the experiment. For example: Changing the wording of anything may have no link to the navigator used. It may only have an effect on the emotional needs of the visitors, which is something you can capture with new AI technology when using AB Tasty.
Don’t check the smallest segments. The smallest segments will not greatly impact your business overall and are often the least statistically significant. Raising the significance threshold may also be useful to mitigate the risk of having a false positive

Should you extend the experiment period often?

If you notice that you often need to extend the experiment period, you might be skipping an important step in the test protocol: estimating the sample size you need for your experiment.

Unfortunately, many people are skipping this part of the experiment thinking that they can fix it later by extending the period. However, this is bad practice for several reasons:

This brings you close to P-hacking
You may lose time and traffic on tests that will never be significant

Asking a question you can’t know the answer to can be very difficult: what will be the size of the lift? It’s impossible to know. This is one reason why experimenters don’t often use sample size calculators. The reason you test and experiment is because you do not know the outcome.

A far more intuitive approach is to use a Minimal Detectable Effect (MDE) calculator. Based on the base conversion rate and the number of visitors you send to a given experiment, an MDE calculator can help you come up with the answer to the question: what is the smallest effect you may be able to detect? (if it exists).

For example, if the total traffic on a given page is 15k for 2 weeks, and the conversion rate is 3% – the calculator will tell you that the MDE is about 25% (relative). This means that what you are about to test must have a quite big impact: going from 3% to 3.75% (25% relative growth).

If your variant is only changing some colors to a small button, developing an entire experiment may not be worth the time. Even if the new colors are better and give you a small uplift, it will not be significant in the classic statistical way (having a “chance to win” >95% or a p-value < 0.05).

On the other hand, if your variation tests a big change such as offering a coupon or a brand new product page format, then this test has a chance to give usable results in the given period.

Digging deeper into ‘flatness’

Some experiments may appear to be flat or inconclusive when in reality, they need a closer look.

For example, frequent visitors may be puzzled by your changes because they expect your website to remain the same, whereas new visitors may instantly prefer your variation. This combined effect of the two groups may cancel each other out when looking at the overall results instead of further investigating the data. This is why it’s very important to take the time to dig into your visitor segments as it can provide useful insights.

This can lead to very useful personalization where only a given segment will be exposed to the variation with benefits.

What is the next step after receiving an inconclusive experimentation result?

Let’s consider that your variant has no effect at all, or at least not enough to have a business impact. This still means something. If you reach this point, it means that all previous ideas fell short; You discovered no behavioral difference despite the changes you made in your variation.

What is the next step in this case? The next step is actually to go back to the previous step – the hypothesis. If you are correctly applying the testing protocol, you should have stated a clear hypothesis. It’s time to use it now.

There might be several meta-hypotheses about why your hypothesis has not been validated by your experiment:

The signal is too weak. You might have made a change, but perhaps it’s barely noticeable. If you offered free shipping, your visitors might not have seen the message if it’s too low on the page.
The change itself is too weak. In this case, try to make the change more significant. If you have increased the product picture on the page by 5% – it’s time to try 10% or 15%.
The hypothesis might need revision. Maybe the trend is reversed. For instance, if the confidence interval of the gain is more on the negative side, why not try the opposite idea to implement?
Think of your audience. Another consideration is that even if you have a strong belief about your hypothesis, it’s just time to change your mind about what is important for your visitors and try something different.

It’s important to notice that this change is something that you’ve learned thanks to your experiment. This is not a waste of time – it’s another step forward to better knowing your audience.

Yielding an inconclusive experiment

An experiment not yielding a clear winner (or loser), is often called neutral, inconclusive, or flat. This still produces valuable information if you know how and where to search. It’s not an end, it’s just another step further in your understanding of who you’re targeting.

In other words, an inconclusive experiment result is always a valuable result.

You might also like...

See all

Article

6min read

Unify GA4 with BigQuery to Strengthen Experiments

Katherine Weber

Nov 7, 2024

Article

1min read

5 Ecommerce Merchandising Strategies with Examples

Emily Healy

Nov 5, 2024

Article

5min read

Why AB Tasty Delivers 4x Faster

Leo Wiel

Nov 4, 2024

Subscribe to
our Newsletter

Article

Sep 21, 2023

10min read

Software Development Team Best Practices

Rowan Haddad

Software development isn’t just developers writing code. It also includes less technical processes that precede the actual development process such as the planning and the testing stages as well as post-development when the software is released and feedback is gathered from end-users.

What this means is that many teams beyond development are involved in the software development life cycle, such as product, design, testing, sales and marketing teams. These teams are all involved in achieving common objectives and ensuring a high-quality product.

Software development teams

To understand why it’s so important for teams to work cross-functionally, it helps to take a look at the different types of teams involved in software development. This will uncover the best practices to get these teams on the road for enhanced collaboration.

When we visualize a software development team, it’s easy to imagine a group of developers writing and releasing code but the reality is that it’s much more than that.

A software development team brings together a wide range of expertise from different teams within an organization to ensure the success of a project. This means that most teams are not solely made up of developers because while they’re responsible for creating the product, there also needs to be people dedicated to building the vision of the product, managing its life cycle, testing the product and marketing it and so on.

A software development team typically consists of the following roles:

Business analyst
Product owner
Project manager
UI/UX designer
Developers (frontend and backend)
QA engineer
Testers
DevOps engineer

With various teams from different departments coming together, there’s a great advantage in having expertise across multiple disciplines, which can bring innovative solutions to problems and insights which otherwise would be overlooked if these teams were to work in silos. Therefore, each of these roles is key to the effective development of your product.

Many factors will influence the structure of your development team such as project complexity, budget and size as well as the needs and expectations of stakeholders.

Ultimately, the team you put together will determine your project’s likelihood of success or failure and their collaboration will be key to achieving desired outcomes.

Why is cross-functional collaboration important?

It’s inevitable for different teams to clash during projects. For example, developers and product teams tend to approach projects from vastly different perspectives and their metrics for success will also differ.

Product managers are often focused on achieving outcomes and overall business objectives. They’re looking to quickly validate their ideas and just as quickly release features that will bring in more revenue for the business. Developers, for their part, aim to build the best possible product by focusing on the deliverables that come with building this product.

The same applies to other teams within the organization with the different types of mindset, skills and goals coming into play during a project.

It’s vital for cross-functional teams to communicate and collaborate effectively around a shared goal in order to successfully achieve it.

Effective collaboration results in better products. Less conflict between the different teams translates to more time dedicated to building and releasing high-quality products. In other words, when teams are on the same page, it allows them to focus on what really matters, which is creating value for the customer through quality software.

Furthermore, cross-functional collaboration is a key driver for creativity and innovation. Collaboration often results in new ideas that can help companies gain competitive advantages as different people come together to work on a project as they encourage each other to consider things from different angles.

Software development team best practices

To enhance productivity and to continue to deliver value to customers, software development teams should stick to some best practices to put them on the path to improved collaboration and success.

Define clear goals

While each team has their own set of internal goals, they still need to make sure that those goals align with the overall business objectives of the company (and product). It’s essential that all teams have a shared understanding of business goals and how to achieve them.

This means keeping all teams coordinated and aligned around the product vision throughout the software development life cycle. It also involves determining the project scope and requirements that will best achieve these objectives.

Once shared goals are established, teams can work with each other instead of against each other even as they perform their own distinct tasks while pushing ahead in the same direction.

The responsibilities of everyone involved in the software development process need to be clearly defined to avoid clashes and create a sense of accountability. The clearer the roles, the less chance of confusion as teams go deeper into the project.

This is a time when having the right leadership can make all the difference. Leaders must clearly define roles and responsibilities and ensure that all teams understand how their work creates value for the organization and its customers.

Choose the appropriate project management methodology

Depending on the size and complexity of your project and teams, it’s important to choose a methodology that works well with the organization’s culture and values.

From Waterfall to Agile methodologies, there are many approaches you can choose. For large projects with a clearly defined start and end-point, the Waterfall approach would work best. However, for projects that are adaptable and divided to smaller sprints, an Agile approach is better suited.

This will serve as the foundation on how you begin to structure your teams and the type of tools they should implement in their daily workflows.

Invest in DevOps

Many software development companies organize themselves in a way that often leads to functional silos so that all the different teams tend to work in isolation while focused on their own goals.

As we’ve discussed, a project also consists of many moving parts and effective communication throughout the different stages of the project becomes complicated as there’s less visibility during these stages.

The DevOps methodology mainly grew out of frustration of the silos between teams, primarily development and operations teams. However, the term has evolved in modern software development to encompass a set of practices and tools that increase an organization’s ability to quickly deliver new software by promoting enhanced collaboration and communication between different teams.

DevOps goes beyond adopting the right tools to achieve high performance and better quality. It also involves undergoing a cultural transformation and teams adopting a shared culture and mindset that allows them to focus on quality of software and speed of delivery.

Choose the right tools

There are a number of collaborative tools your teams can adopt to establish efficient communication practices. The tools you choose will serve as the foundation to help teams work together towards common goals.

Among those are ones that can help you plan your projects from the bigger picture to the little details, which is especially useful for large organizations with multiple teams and team members collaborating. This also enables teams to have greater visibility over each member’s progress in the project and empowers teams to share input and encourage fast feedback loops in order to build a culture of DevOps and open communication.

Different teams will use their own tools to track work and progress throughout a project. For example, product teams will typically use roadmapping tools to plan, prioritize and track features while development teams will use development tools such as version control systems.

Therefore, there are many tools to choose from depending on the needs of the organization and teams from collaboration to project management tools as well as automation tools to streamline processes.

For example, teams adopting DevOps practices into their workflows will also need to choose the right stack of tools according to their unique business needs in order to implement DevOps successfully.

Communicate constantly and efficiently

From the onset of a project, all teams and other stakeholders will need to be involved in the decision-making process to keep issues down to a minimum and avoid miscommunication further down the road.

For example, many product teams often take the lead on setting the product vision and requirements without reviewing them with the development and engineering teams. In this scenario, it’s essential to understand that developers are the ones who will be writing for the product in question and so they must be consulted early on to give them the necessary context to build and prioritize the right features.

In this scenario, product managers and owners will need to effectively communicate the strategic direction and vision of a product with the help of a dedicated product roadmap. This will enable developers to understand why they’re building the product, its value and how it’s linked to overall business objectives.

A project often consists of many moving parts which means it might not be possible for teams to have full visibility over them. However, Many important decisions are collaborative in nature and require input from multiple sides. It’s imperative that everyone on the team is encouraged to actively provide feedback and share information about the progress of development. This also helps teams identify any potential roadblocks and take quick action to address them.

Set KPIs and metrics to track performance

Depending on the business objectives set at the beginning, teams will need to establish metrics to track and measure performance throughout the development process and beyond.

These metrics will be essential in the short and long term to make data-driven decisions to optimize products and improve team performance.

Teams can set any kind of metrics that will allow them to assess their efficiency. These could include productivity metrics such as velocity and cycle time, customer metrics such as customer satisfaction and net promoter scores and and more DevOps specific metrics such as DORA metrics.

Software development is a team effort

To produce high-quality software that meets your customers’ expectations and needs, different teams have to come together within a collaborative environment to solve complex problems.

If organizations work on fostering collaboration across the entire software development life cycle, software development teams can overcome challenges, maximize productivity and software quality as well as deliver better value to customers. It’s a win-win situation.

At the end of the day, all teams within an organization are looking to accomplish the same end-goals and outcomes, mainly to keep the business running smoothly and bring in profit as well as create top-notch products that customers will love.

You might also like...

See all

Article

6min read

Unify GA4 with BigQuery to Strengthen Experiments

Katherine Weber

Nov 7, 2024

Article

1min read

5 Ecommerce Merchandising Strategies with Examples

Emily Healy

Nov 5, 2024

Article

5min read

Why AB Tasty Delivers 4x Faster

Leo Wiel

Nov 4, 2024

Subscribe to
our Newsletter

Article

Sep 15, 2023

3min read

Supercharge Your Coding Experience With AB Tasty’s VS Code Extension

AB Tasty

Throughout the coding and testing process, developers find themselves having to switch between a lot of different tools. At AB Tasty, we’ve understood that this can be a hassle which is why we’ve worked to make it easier to work between your coding environment in VS Code and the AB Tasty feature flagging platform.

AB Tasty’s new open beta for VS Code extension means that teams can work faster and take their coding experience to the next level.

This new extension will allow you to use AB Tasty Feature Experimentation and Rollouts, formerly Flagship, directly in the VS Code environment.

This means that you no longer have to switch between your Visual Studio code environment and your flags in the AB Tasty platform.

Why should you use AB Tasty’s VS Code extension?

With the VS Code extension, implementing feature flags in your codebase has never been simpler.

The extension enables you to seamlessly connect your Visual Studio Code environment with AB Tasty for full visibility over feature flags in your files and then retrieve the flag and its details directly in your code, which helps save time and eliminate complexity.

This significantly boosts productivity as you no longer need to switch between your coding environment and platform making the management and implementation of feature flags much easier.

Getting started

To be able to use this extension, you will need to have an account in AB Tasty. Then all you need to do is follow the steps below:

1)Create an access name on our Remote Control API

By creating an access to our Remote Control API, you’ll be able to manage the right scopes and get access to the extension’s features.

2)Save your client ID and client secret

You will then receive your “client ID” and “client secret”. These credentials will allow you to log in to the extension.

Ready to get started?

You’ll be able to find the extension on the Microsoft marketplace or in the extension marketplace directly in VS Code in order to download it.

Once the download process is complete, you can then follow the steps in our documentation to create a configuration and start using the extension.

Don’t forget to rate and review Flagship Code on the VS Code Marketplace to help us continue improving your coding experience.

AB Tasty code supports various programming languages and frameworks, making it adaptable to your tech stack – and if that’s not the case, feel free to get in touch.

You might also like...

See all

Article

6min read

Unify GA4 with BigQuery to Strengthen Experiments

Katherine Weber

Nov 7, 2024

Article

1min read

5 Ecommerce Merchandising Strategies with Examples

Emily Healy

Nov 5, 2024

Article

5min read

Why AB Tasty Delivers 4x Faster

Leo Wiel

Nov 4, 2024

Subscribe to
our Newsletter

Article

Sep 8, 2023

8min read

The Ideal CRO Structure for Sustainable Growth

Emily Healy

With experimentation, the goal is simple: find out what resonates best with your digital audience to create a relationship and drive business growth. But, how do you reach the point of success?

Experimentation opens the door to fresh insights that are only found through testing, compelling you to continuously refine different facets of your website for an improved digital experience across the board. Once you take your first steps down your experimentation roadmap, your path toward optimization evolves to the point where you can become a more prominent digital player.

However, experimentation success will introduce growing pains – especially if you’re a company starting its CRO journey. Allocating your CRO resources early and efficiently is important to set your business up for continued success, prosperity, and evolution.

A firm foundation and building good habits from the start is the best way to ensure that your growth won’t stop.

How to build out your CRO team following the centralized model

A successful CRO team needs to be well-equipped with the necessary resources to carry out their missions which include time, tools, people and technology.

The first step in creating your team is to focus on leadership. The leader of your team needs to set an example by prioritizing experimentation and making it a part of your organization’s values. Your leader needs to value and encourage experimentation by creating a safe environment for testing where failures are seen as learning opportunities. CRO organizations need to create a culture of collaboration and communication where everyone works together to achieve a common goal.

It’s important to keep in mind that experimentation requires a lot of collaboration. By having a vast team equipped with different skills, you’ll need to facilitate communication between different teams, such as designers, developers, marketers, and data analysts.

This means that everyone needs to be aware of the goals and deliverables of each experiment, the roles of each stakeholder, the project timeline, and certainly if there are changes to the roadmap. This requires constant and open communication to keep everyone prepared. Each team member needs to be able to trust their teammates to perform certain tasks and have confidence in their own individual role.

With open communication and frequent regroups to check progress and share ideas, you can ensure that everyone is aligned and working towards the same objectives. Sharing results builds trust between team members and gives everyone an opportunity to celebrate wins, support each other through the learning opportunities and create a positive environment where feedback is welcome.

What is the ideal CRO team structure?

When picking the ideal structure for your CRO team, you have to keep in mind that this will vary depending on your organization’s size, goals, and resources at hand.

A small CRO team following the centralized model will need to have individuals responsible for covering all core responsibilities – from ideation to implementation to examination. Ideally, this would include:

CRO Manager
UX/UI Designer
Data Analyst
Web Developer
Content Specialist

To continue CRO team expansion, a medium-sized or large team should adopt the positions above and some or all positions listed below:

Product Manager
Product Designer
Data Scientist
Content Designer
Content Writer
Conversion Rate Optimizer/Tester
Technical Web Analyst
Website Animation Specialist

The skills needed to perform CRO are vast. A person equipped to be a great addition to your CRO team will most likely have a background in one of the following areas:

Chief Data Officer
Full Stack Developer
Functional Designer
Digital Marketing Specialists
Data Scientist (Specializing in CRO)
Web Analyst

Keep in mind that a CRO team is typically a cross-functional team and team members may be involved in other projects simultaneously. As each organization is completely unique, there are no hard and fast rules for the “perfect” team. Your ideal structure may shift as you go, reminding you of the importance of flexibility.

Rapid CRO growth

To put the rapid growth of CRO teams into context, let’s take a quick look at one global leader in the premium cosmetics industry: Shiseido.

Even though Shiseido already had a CRO team in place, they wanted to grow and turn their constricted experimentation strategy into an intuitive and scalable optimization program. They went from running four tests per year to over 10 tests per month using AB Tasty and expanded their team accordingly to cover more ground and expand their experimentation goals. Growth can happen quickly when setting new priorities and adopting a new mindset. See how Shiseido revitalized its experience optimization strategy with AB Tasty.

Steps for successful CRO implementation

Mindset shift

Building a culture of experimentation is crucial for a successful CRO organization. There needs to be a mindset shift towards data-driven decision-making, embracing bold decisions and viewing failure as an opportunity to learn and improve.

One of the most significant obstacles in establishing this culture is the fear and apprehension linked to failure. CRO teams need to recognize that failure is a natural part of the experimentation process and that every failed experiment provides valuable insights and learnings. By embracing what doesn’t work, CRO teams can create a culture that encourages experimentation and embraces risks.

All data derived from tests is valuable for building out future steps. The sooner an organization can adapt to this mentality, the more stable its CRO foundation will be.

LOOKING FOR MORE about the culture of experimentation?

Listen to the 1000 Experiments Club Podcast. The only podcast that interviews industry experts who have run over 1,000 experiments.

Set goals for your CRO team

CRO teams need to define exactly what they want to achieve through experimentation and how they will measure success. With this being said, data should be at the heart of all experimentation. Decisions should be made based on data collected and not only a gut feeling. By setting goals and assigning metrics to track progress, CRO teams can stay focused on their vision to achieve their objectives and track their progress.

Define the challenges of CRO implementation

There will be challenges to any success story. It’s important to address the potential challenges that may arise early on to keep your team prepared for any tough moments.

Barriers to continual success could include time restrictions, lack of adequate resources, employees with sub-par attitudes, pressure from HIPPOS, technology or anything that could potentially interfere with your roadmap.

After setting your goals and defining the next steps on your roadmap, it’s easier to outline the barriers that may prevent you from achieving those objectives, such as technical limitations or budget constraints.

Outline the team’s roles and responsibilities

Next, define the team’s roles and responsibilities. All team members should be aware of their personal objectives and how their work contributes to the overall success of the project (and their impact on the organization).

This includes identifying who will be responsible for testing, analyzing data, creating content, and making technical improvements to the website or app. Especially if team members have cross-functional roles where their time is divided, their responsibilities during each project should be clearly defined.

Standardize the A/B test process with your CRO Team

To standardize the A/B test process in your organization, there needs to be coordination of all digital teams around A/B tests and your overall CRO strategy. Your testing roadmap should outline the experiments your team will conduct, the hypotheses they will test, and the metrics they will use to measure success. By developing a testing plan, CRO teams can ensure that their experiments are aligned with their goals and that they are testing the right elements of the website or landing page.

With your new CRO team, it’s important to always start with identifying the most valuable tests at the right time. By brainstorming with your team to identify multiple elements, you will have various high-value optimization paths available to you when your team has the bandwidth.

When implementing a test, you must have a team ready to create the design and content for the test and another team available to put it all into production.

As a post-launch follow-up plan, you will need to develop an optimization plan to cater to the results.

Implement the winning variation – If your variation shows better results when compared to the original, plan for adequate time in your roadmap to incorporate any permanent changes.
Develop a new variation – Let’s say your variation wasn’t more influential than the original version. You’ve learned more about your audience that you can use in the future. If you’ve found what doesn’t work, leave room in your plans to go back to the drawing board to find a variation that resonates better with your audience.
Accept the original version – If you and your team are happy with the performance of the original version of your webpage, it’s time to move on to the next priority on your optimization list.
Re-challenge the winning variation – Consumer preferences are constantly changing. What worked 6 months ago might not resonate with your audience in the same way down the road. Plan time in your roadmap for more challenges to see continued success.

To promote communication, your experimentation roadmap and the results of each experiment should be accessible to everyone and promote transparency. This keeps your team aligned to standardize your process.

In CRO, you need to be adaptable. You won’t know the outcome of a test until it’s over (you don’t want to develop a bias by trying to guess the results either!). Based on the results, you and your team need to be ready to react quickly to follow the next steps of whichever path you choose.

A centralized CRO team built for sustainable growth

Developing a CRO team that’s built to grow and build a sustainable culture of experimentation is not the easiest task. There is always room for trial and error when figuring out what works best for your organization.

With a mindset shift, a well-equipped team, and a clear understanding of goals, barriers, and team roles, your organization will be prepped to carry out your winning strategy. With these elements in place, your organization can continuously test and optimize all digital e-commerce channels, leading to increased conversions, higher customer satisfaction, and ultimately, better business results.