Category: A/B Testing

Article

Oct 8, 2024
8min read

Mastering Mobile Optimization: Tactics for Higher Commerce Conversion

AB Tasty

AB Tasty

Share on

LinkedIn Facebook Twitter

In today’s mobile-first world, where smartphones dominate more than half of global web traffic, optimizing for mobile has never been more crucial. Mobile usage surpassed desktop in the US in 2022 and in the UK in 2023, signaling a clear shift in consumer behavior. Brands are now urged to design with mobile in mind first, adapting for desktop as needed, rather than the reverse. This shift may seem daunting for teams, but it’s a necessary evolution to meet the expectations of today’s users.

Whether your customers are researching products or making purchases, their mobile experience can make or break their journey with your brand. While it’s clear that more shopping is done on mobile devices than on desktop, the real question remains: how significant is mobile shopping overall? Today’s mobile-savvy consumer isn’t just using their device for convenience, but to blend their in-store and online shopping into one seamless experience. In fact, nearly 80% of shoppers globally use their smartphones to browse  a retailer’s website while shopping in-store, and 74% use the store’s app. However, only 33% of consumers prefer making purchases on their phones, with 49% reporting a smoother experience on desktop or tablet. This highlights just how important it is for brands to enhance their mobile offerings for a seamless experience across all devices.

To delve into the complexities of mobile optimization Mary Kate, AB Tasty’s Head of Growth Marketing for North America, teamed up with Allie Tkachenko, a UI/UX Strategist at WPromote, for a webinar on mastering mobile. AB Tasty’s platform enables brands to deliver personalized customer experiences, while Wpromote helps design and optimize engaging web experiences that convert.  They emphasize a key message: mobile optimization isn’t just about resizing for a smaller screen – it’s about creating an intuitive, seamless journey that aligns with today’s mobile-first consumer’s behaviors and expectations. 

It’s critical that mobile  websites  excel in areas like speed, navigation, and user-friendliness. Let’s dig into three actionable strategies from the webinar to help your brand stay ahead and deliver an improved mobile experience for your customers. 

1. Maximizing limited space

One of the biggest challenges in mobile design is maximizing limited screen space without overwhelming users. The key is to keep crucial content above the fold—on mobile, this means placing essential elements like navigation bars, CTAs, and product highlights in a prominent position, visible without scrolling. This is particularly important on search landing pages, the homepage, and other high-traffic areas. A well-organized and streamlined navigation system that helps users quickly find what they need can lead to higher engagement and reduced bounce rates.

While desktops offer ample space to break down navigation into detailed categories, mobile design requires a more simplified structure due to space constraints. Consider grouping categories under broader buckets like “Top Categories” or similar, allowing users to easily explore the site without feeling overwhelmed by too many options. Another key strategy is leveraging responsible design, such as implementing sticky navigation bars or menus that stay visible as users scroll. This approach, widely adopted across industries, ensures easy access to important links and minimizes the effort required to navigate the site.

AB Tasty in action

The UX team at Clarins wanted to make their product more visible on their category pages. In the original layout, filtering and sorting functions were stacked, removing space from the second row of products appearing. After testing a column layout for the filtering and sorting menus, the team saw a significant improvement—bounce rates decreased, and clicks to products increased by 34%.

Optimizing screen space solutions

  • Keep key elements above the fold 
  • Simplified navigation 
  • Use responsive design  
Watch the whole webinar below or get our Mobile Optimization Guide

2. The thumb zone

The “Thumb Zone” refers to the area of the screen that is easiest for users to reach with their thumbs, typically the lower portion of the screen. Since most users interact with their phones one-handed, placing critical CTAs, buttons, and interactive elements within this zone is important for accessibility and ease of use. 

Consider this: a navigation bar that starts at the top of the page but shifts responsively to the bottom as the user scrolls. This keeps it in an expected spot initially, avoiding any disruption to the user’s flow, and then moves it to a more reachable area as they continue browsing.

Another thing to keep in mind is sizing. Whether it’s buttons, images, form fields, or menu links, the size of these elements plays a huge role in usability. You can’t just shrink them to save space—you have to ensure they’re “tappable” so users can easily interact. While reachability is key, think about what doesn’t need to be within reach, like informational banners or logos. You can place those outside the thumb zone, saving prime space for interactive elements.

Brands that prioritize the thumb zone in their mobile designs see improved user engagement and lower frustration levels. This small shift can make a significant difference in usability and customer satisfaction. 

AB Tasty in action

The team at Club Med, a leading travel and hospitality brand, observed that their original mobile site displayed a navigation bar at the top of the page, which would disappear as users scrolled down. To increase user engagement with different category offerings, they created a variation of the mobile homepage with a sticky navigation bar which remained at the bottom of the screen while scrolling.

The results of the A/B test revealed a 12% increase in click rates, a 12% increase in access to the transaction funnel, and a 2% decrease in the bounce rate for users showing the variation with the sticky navigation bar. This approach effectively makes information more physically accessible.

Optimizing the thumb zone

  • Bottom Navigation 
  • Sizing 
  • Reachability  

3.  Improving processes 

Lengthy forms and cumbersome checkout processes are major obstacles to conversion in mobile digital experiences. Mobile users expect a seamless, fast journey, and frustration with complex forms often leads to abandoned carts. Streamlining these processes—especially form fills and checkouts—can reduce friction and improve conversions. We’ve all experienced the annoyance of having to redo a form, fearing progress might be lost, which can lead to users abandoning the process entirely. Key areas for optimization include simplifying checkout by offering guest checkout options and exploring one-click payment methods.

Search and product discovery also present unique challenges on mobile devices due to limited screen space. With condensed menus and site navigation, users often rely heavily on the search function. Optimizing your search results pages to help users quickly find specific products can drastically improve the user experience. The space constraints of mobile mean that every element, including search results, should guide users efficiently to what they’re looking for.

Lastly, page load speed plays a vital role in retaining users. A slow-loading site can deter users, leading them to abandon your site altogether. Reducing load times is crucial for keeping users engaged. Understanding your audience and continuously optimizing these processes will help ensure your site meets their needs and encourages conversions.

AB Tasty in action

Travel insurance company, DirectAsia, needed users to fill out a form to generate an insurance quote. The team observed that customers were not completing the forms as smoothly as expected. To address this, they implemented a variation in the test where bolded check marks appeared to validate each completed field. This change created a sense of progress for users as they navigated the form and alleviated any uncertainty about needing to go back to correct errors.

As a result of this test, DirectAsia achieved a 1.97% increase in quote generations and a 7.78% increase in transaction rates. By reassuring users throughout the form-filling process, DirectAsia successfully guided more customers through their quote generation form.

Optimizing mobile processes

  • Checkout 
  • Search and discovery 
  • Speed & image loading 

Wrapping up

Mobile optimization is about much more than making your website look good on a smaller screen; it’s about crafting a seamless, user-friendly experience that enhances the customer journey. Whether you’re focusing on improving site speed, optimizing design for better accessibility, or streamlining complex processes, the suggestions above provide a solid foundation for mastering mobile optimization. By understanding the nuances of mobile behavior and catering to the needs of your users, your brand can create a frictionless experience that drives conversions and fosters customer loyalty.

Stay ahead in the mobile-first era by ensuring your website design and processes align with the expectations of today’s consumers. AB Tasty can help achieve this goal by providing innovative tools and data-driven testing to enhance your mobile strategy. As mobile usage continues to grow, so does the importance of providing a smooth, engaging, and conversion-focused experience. 

If you want to get all the details. – watch the webinar below.

Share on

LinkedIn Facebook Twitter

Article

Sep 24, 2024
6min read

Ideas Worth Keeping: The Top 5 Subscription Trends to A/B Test

AB Tasty

AB Tasty

Share on

LinkedIn Facebook Twitter

Platform partner blog

This guest blog was written by Kit Heighway, Director of Optimization, at Daydot,  a digital agency that specializes in crafting exceptional experiences to drive measurable revenue growth. They are experts in Conversion Rate Optimization, Performance UX Design, and Customer Lifecycle Optimization for Subscription, eCommerce, and Non-Profit brands. 

Let’s imagine you are relaxing at home after a long day’s work, when the doorbell rings. You jump up, eager to see what’s waiting behind the door – a new clothing item, a cooking kit, or perhaps a treat for your pet? The excitement is real, and it’s all thanks to your recent subscription box sign-up. 

Subscription boxes have quickly become consumer’s favorite way to shop from brands they love. In fact, the global subscription box market is projected to exceed $75 billion by 2025.* With so many brands wanting to get in on the action, the key question is: how can you participate in a way that resonates with your audience? A/B testing subscription features is a great starting place. 

The Daydot team dive into the subscription box world and try out 5 of the most popular subscription purchase journeys. In the article below, we share what features we loved the most, giving you our round-up of the best features to test on your digital subscription journey. 

The subscription journeys reviewed: Abel and Cole, Bloom and Wild, Gin-box, Dear-bump, Bella & Duke, Perky Blenders, Butternut Box. 

5 Subscription Trends to Test

1. Combining expected USPs with what makes your business different 

In 2024 certain features have become standard expectations in the world of subscription services. Phrases like “Free delivery,” “Home delivery,” or “Cancel anytime” are no longer points of differentiation – they’re baseline consumer expectations. However, these essentials still matter, and this is where the savvier brands really stand out, by mixing their unique personality with those expected USPs. 

Take Abel & Cole, for example. As the leader in organic products, they don’t settle for the dull “Free delivery to your door.” Instead, they integrate their brand personality into the message with “Get your ethical food delivery dropped to your door.” It’s a small tweak, but it adds a layer of authenticity and makes a difference in standing out.

Abel and Cole: “Get your ethical food delivery dropped to your door” 

Test ideas:
  • Expected USP copy improvements

2. Cancelation reassurance throughout the user journey 

Subscription cancellation will happen but how you handle it can make or break the customer experience. A recent study shows that over 25% of consumers prioritize easy, penalty-free cancellations when choosing a service.* People value the assurance that they can leave without any hassle. 

The best brands excel in this area by offering clear, upfront reassurance about cancellation. From the product listing page, where they confidently state, “There’s no commitment – you can skip or cancel at any time,” to the basket page with a gentle reminder, “Delivered weekly, but you can cancel anytime.” Right before the final step, they reinforce the message: “Count on us for reliable weekly delivery, with the flexibility to skip, pause, or cancel anytime.” 

By making cancellation easy and transparent, these brands turn a potentially stressful decision into another positive touchpoint, helping customers feel in control from start to finish.

Test ideas: 
  • Homepage cancelation reassurance 
  • Product listing/details cancelation reassurance 
  • Basket cancelation reassurance 
  • Checkout cancelation reassurance 
  • Cancelation messaging tone formal vs friend

3. Showcasing how subscriptions could fit into users’ real lives 

For physical subscriptions, the key is to make them feel tangible and exciting in the early stages of the purchase journey. In a digital world getting something tangible is a huge draw. 

How can you showcase that experience on a website without sending samples? 

Butternut Box achieves this with a fun, heartwarming video on their landing page. It walks potential users through the excitement of the box arriving at the door, the thrill of unboxing, and of course, a happy dog enjoying their treat. It’s more than just a video—it’s a mini-experience that brings the product to life.

And Abel & Cole? They’re leading the way again, by adding recipe ideas and videos directly on the product page. It’s far more engaging than a simple image gallery, sparking users’ imaginations and making their experience more immersive.

Test ideas: 
  • An unboxing or arrival video 
  • A social media feed showing real users enjoying your subscription 
  • Previews of activities you can do with the subscription (like recipe guides, or dog games) 

4. Remembering that users don’t just subscribe for themselves 

It’s easy to overlook that many users aren’t subscribing for their own needs – they’re often gifting a subscription to someone special. Whether it’s for a child heading off to college or a new colleague at work, recurring subscription businesses miss this opportunity by sticking to a one-size-fits-all approach. 

Perky Blenders, however, has mastered the art of gifting by offering flexible three, six, and twelve-month subscription options for their premium, freshly roasted coffee. 

Test ideas: 
  • Gifting subscription journey 
  • Business gifting journey 
  • Personalization based on gift giving intent (supporting a friend, new home, starting uni, new parent etc) 

5. Not assuming that subscriptions will last forever 

Consumers want subscriptions to be as hassle-free as possible. Hidden or complicated cancellation processes can be a major turnoff. 

Surprisingly, more businesses aren’t testing fixed, limited-time prepaid subscriptions. Some customers don’t want to commit to an ongoing plan, no matter how easy cancellation is. Offering a set subscription period could disrupt the subscription box journey in a big way. 

Bloom & Wild are ahead of the curve here, letting customers pick between three, six, and twelve month subscriptions without any automatic renewal. 

Test ideas: 
  • 3 month fixed term subscription 
  • 6 month fixed term subscription 
  • X time fixed term subscription (reflecting a particular life-stage your product may be purchased for)

Wrapping up 

These five innovative strategies are helping subscription leaders enhance their customer journeys, boost conversions and drive revenue growth. Now is a good time to evaluate your own subscription flow and consider integrating some of these ideas. But don’t just copy and paste – Remember, about 80-90% of digital ideas flop because they weren’t tested first. That’s why Experimentation is essential before implementing. It allows you to identify what  clicks with your users and ensures that you invest in features that deliver results, rather than relying on assumptions. 

Share on

LinkedIn Facebook Twitter

Article

Sep 13, 2024
5min read

Unlocking Hidden Revenue – A/B Testing within Single Page Applications

AB Tasty

AB Tasty

Share on

LinkedIn Facebook Twitter

If your organization is having trouble successfully running A/B tests in areas of your site where customers are going through the purchase flow, the issue may be due to Single Page Applications (SPAs) on your site. As customers move through the process, your A/B testing tool might not recognize their progress in an SPA environment.

There is enormous value in A/B testing critical areas of the web experience that are often operating in SPA environments, such as an eCommerce checkout. 

This guest blog post was written by Jason Boal, The Principal Analyst & Optimization Strategist at 33 Sticks – a leading American analytics agency. Let’s address this common issue and uncover ways to overcome this to unlock hidden revenue on your site.

1. What is a SPA and how can I tell if the web experience uses one?

In a Single-Page Application (SPA) environment, content is loaded dynamically without requiring a full page refresh or reload. User interactions occur on a single page, with new content being loaded as the user navigates. Gmail is a prime example of an SPA. At a high level, an SPA functions similarly to a standard client-server interaction, but the key difference lies in what is returned to the browser.

To determine if you are operating in a Single-Page Application (SPA) environment on your site, pay attention to whether the page reloads as you interact online. If you see the page load indicator—such as the spinning icon in the browser tab (in Chrome)—it means the page is reloading, and you are likely using a traditional multi-page application (MPA).

Many websites are hybrids, meaning that only certain sections, like the checkout process, function as an SPA. To find out which parts of your site are SPAs, you can ask your development team for clarification.

2. Does my testing tool work within SPA environments and what do I do if it doesn’t?

Visual editors are becoming extremely popular in the A/B testing space, for many reasons. One  is that Marketers are developing and launching more tests compared to the DEV team. If your testing tool has a difficult time loading and testing content in the visual editor, the reason could be that  the tool is either not equipped or not set up to properly handle SPAs. This often happens in a secure checkout flow, where the customer is required to step through items like shipping address, billing, etc. The page you are attempting to A/B test on will not properly load in the visual editor and you will receive an error message.

FIGURE 1 – VISUAL EDITOR SPA ERROR MESSAGE

Ask your vendor if your testing tool can detect changes in the DOM and if it has a mechanism to look for timing. 

Here are two challenges that some A/B testing tools face:

  1. Visual Editors: Some A/B testing tools rely on the initial page load to determine what content to modify. These tools may struggle when content needs to change without a page reload. For example, if your test content is on page 3 of your site’s checkout flow, which is an SPA, the tool might not detect the need to inject content changes because there are no page loads as users navigate through the checkout flow.
  2. Timing: As content on the page changes, it can be tricky for an A/B testing tool to insert test variations at the right moment. Variations can’t be applied before the content starts loading, but waiting until the content has fully loaded can result in users seeing the content change, a phenomenon known as “flicker.”

AB Tasty has extensive experience creating A/B tests in Single-Page Application (SPA) environments. We recommend implementing a delay on our tag’s execution so that it only triggers when the page is fully ready. This is achieved using a proprietary locking mechanism. This is just one example of how AB Tasty stands out in the A/B testing industry.

3. How do I take it to the next level?

Once you’ve unlocked A/B testing in SPAs, it is time to brainstorm testing ideas and develop a strategic roadmap to uncover ways to increase revenue for your organization. Here are a few ideas to help you jump-start that process!

  • Test various methods of updating cart quantity.
  • Test product detail page functions such as color variant selection methods.
  • Test buy box functions such as stock amount and store information.
  • Test different shipping messages based on cart value.
  • Test reordering flow steps.
  • Test navigation patterns or menu structures to optimize user flow within the SPA.
  • A/B test various UI/UX elements like buttons, forms, or interactive features specific to your SPA.
  • Test personalization strategies based on user behavior and interactions within the SPA.

Key Takeaways

  • There is enormous value in A/B testing critical areas of the web experience that are often operating in SPA environments, such as an eCommerce checkout flow. This is usually the last stage of any digital customer journey and vital to get right.
  • Determine whether or not your site leverages SPAs anywhere on your site.
  • Dig into your testing tool to ensure it can properly load test content changes with SPA environments.
  • Understand what other AB testing tools are out there and how they handle SPAs.
  • Develop an optimization roadmap based on your new knowledge!

Share on

LinkedIn Facebook Twitter

Article

Sep 9, 2024
5min read

Failing Forward for Experimentation Success | Shiva Manjunath

AB Tasty

AB Tasty

Share on

LinkedIn Facebook Twitter

Shiva Manjunath shares how debunking best practices, embracing failure, and fostering a culture of learning can elevate experimentation to new heights.

In this episode of The 1000 Experiments Club, guest host and AB Tasty’s Head of Marketing, John Hughes, sat down with Shiva Manjunath, Senior Web Product Manager of CRO at Motive and Host of the podcast From A to B. Shiva’s journey through roles at Gartner, Norwegian Cruise Line, Speero, Edible, and now Motive, has made him a passionate advocate for the transformative power of experimentation.

During their conversation, Shiva discussed the pitfalls of following “best practices” blindly, the importance of creating an environment where failure is seen as a step toward success, and how companies can truly build a culture of experimentation.

Here are some of the key takeaways.

The myth of ‘Best Practices’

Too often, the so-called experimentation best practices become a checkbox exercise, rather than a thoughtful strategy.

“If you’re focused on best practices, you’re likely missing the point of true optimization,” Shiva notes. 

He recounted a situation at Gartner where simplifying a form—typically hailed as a best practice—actually led to a sharp drop in conversions. His point? Understanding user motivation and context is far more important than relying on one-size-fits-all rules. It’s this deeper, more nuanced approach to experimentation that drives real results.

“If what you believe is this best practice checklist nonsense, all CRO is just a checklist of tasks to do on your site. And that’s so incorrect,” Shiva emphasized, urging practitioners to move beyond surface-level tactics and truly understand their audience.

Embracing failure in experimentation

A major theme of the discussion was the pivotal role failure plays in the journey to success. Shiva was candid about his early experiments, admitting that many didn’t go as planned. But these “failures” were crucial stepping stones in his development.

“My first ten tests were all terrible. They all sucked,” Shiva admitted, underscoring that even the most seasoned experts start with mistakes. He stressed that organizations must create an environment where employees can experiment freely, learn from their mistakes, and continue to improve.

“If you’re penalized for running a losing test, you’re not in a culture of experimentation,” Shiva insists.

Organizations that punish failure are stifling innovation. Instead, Shiva advocates for an environment where employees can test, learn, and iterate without fear. “The idea that you have the flexibility to discuss failures and focus on, ‘Well, I ran this test. It lost. Now, what do we do next?’—that’s a culture of experimentation.”

Scaling experimentation maturity

Shiva also explored the varying levels of experimentation maturity within organizations. Many companies claim to have a “culture of experimentation,” but few truly practice it at scale. Shiva emphasized the importance of making experimentation accessible to everyone in the organization, not just a select few.

Reflecting on the loss of Google Optimize, Shiva acknowledged its role as a gateway into the world of experimentation. “I got into experimentation through Google Optimize,” Shiva recalled, recognizing the tool’s importance in lowering the barrier to entry for newcomers. He urged companies to lower barriers to entry and enable more people to engage with experimentation, thereby fostering a more mature and widespread culture of testing.

The role of curiosity and data in experimentation

Another critical point Shiva raised was the importance of curiosity in experimentation. He believes that genuine curiosity drives the desire to ask “why” and dig deeper into user behavior, which is essential for effective experimentation.

“If you’re not genuinely curious about the why behind many things, I don’t know if experimentation is the field for you,” Shiva stated, underscoring curiosity as a crucial soft skill in the field.

Shiva also highlighted the foundational role of being data-driven in any experimentation strategy. However, he cautioned that having data isn’t enough—it must be effectively used to drive decisions.

“If you’re in a business setting and the business looks at your program and this is zero test wins, right? And then after two years, they would rightfully say ‘is this the way it’s supposed to go?’” Shiva remarked, pointing out that data-driven decisions are key to sustaining a culture of experimentation.

What else can you learn from our conversation with Shiva Manjunath?

  • Why it’s crucial to critically evaluate industry buzzwords and ensure they align with real practices.
  • How true personalization in experimentation goes beyond just adding a user’s name.
  • The need for thorough analysis to genuinely support data-driven decisions.
  • Shiva’s take on the future of experimentation after Google Optimize and how companies can adapt.

About Shiva Manjunath

Shiva Manjunath is the Senior Web Product Manager of CRO at Motive and Host of the podcast From A to B. His insatiable curiosity about user behavior and deep passion for digital marketing have made him a standout in the world of experimentation. With experience at top companies like Gartner, Norwegian Cruise Line, and Edible, Shiva is dedicated to demystifying CRO and pushing the boundaries of what’s possible in the field.

About 1,000 Experiments Club

The 1,000 Experiments Club is an AB Tasty-produced podcast hosted by John Hughes, Head of Marketing at AB Tasty. Join John as he sits down with the experts in the world of experimentation to uncover their insights on what it takes to build and run successful experimentation programs.

Share on

LinkedIn Facebook Twitter

Article

Aug 29, 2024
8min read

A/B Testing: It’s Not Just About the Outcome

AB Tasty

AB Tasty

Share on

LinkedIn Facebook Twitter

A/B testing is often seen as the magic bullet for improving e-commerce performance. Many believe that small tweaks—like changing the color of a “Buy Now” button—will significantly boost conversion rates. However, A/B testing is much more complex. 

Random changes without a well-thought-out plan often lead to neutral or even negative results, leaving you frustrated and wondering if your efforts were wasted. 

Success in A/B testing doesn’t have to be defined solely by immediate KPI improvements. Instead, by shifting your focus from short-term gains to long-term learnings, you can turn every test into a powerful tool for driving sustained business growth. 

This guest blog was written by Trevor Aneson, Vice President Customer Experience at 85Sixty.com, a leading digital agency specializing in data-driven marketing solutions, e-commerce optimization, and customer experience enhancement. In this blog, we’ll show you how to design A/B tests that consistently deliver value by uncovering the deeper insights that fuel continuous improvement. 

Rethinking A/B Testing: It’s Not Just About the Outcome 

Many people believe that an A/B test must directly improve core e-commerce KPIs like conversion rates, average order value (AOV), or revenue per visitor (RPV) to be considered successful. This is often due to a combination of several factors: 

1. Businesses face pressure to show immediate, tangible results, which shifts the focus toward quick wins rather than deeper learnings. 

2. Success is typically measured using straightforward metrics that are easy to quantify and communicate to stakeholders.

3. There is a widespread misunderstanding that A/B testing is a one-size-fits-all solution, which can lead to unrealistic expectations. 

However, this focus on short-term wins limits the potential of your A/B testing program. When a test fails to improve KPIs, you might be tempted to write it off as a failure and abandon further experimentation. However, this mindset can prevent you from discovering valuable insights about your users that could drive meaningful, long-term growth. 

A Shift in Perspective: Testing for Learnings, Not Just Outcomes 

To maximize the success and value of your A/B tests, it’s essential to shift from an outcome-focused approach to a learning-focused one. 

Think of A/B testing not just as a way to achieve immediate gains but as a tool for gathering insights that will fuel your business’s growth over the long term. 

The real power of A/B testing lies in the insights you gather about user behavior — insights that can inform decisions across your entire customer journey, from marketing campaigns to product design. When you test for learnings, every result — whether it moves your KPIs or not — provides you with actionable data to refine future strategies. 

Let’s take a closer look at how this shift can transform your testing approach. 

Outcome-Based Testing vs. Learning-Based Testing: A Practical Example 

Consider a simple A/B test aimed at increasing the click-through rate (CTR) of a red call-to-action (CTA) button on your website. Your analytics show that blue CTA buttons tend to perform better, so you decide to test a color change. 

Outcome-Based Approach 

Your hypothesis might look something like this: “If we change the CTA button color from red to blue, the CTR will increase because blue buttons typically receive more clicks.”

In this scenario, you’ll judge the success of the test based on two possible outcomes: 

1. Success: The blue button improves CTR, and you implement the change. 2. Failure: The blue button doesn’t improve CTR, and you abandon the test. 

While this approach might give you a short-term boost in performance, it leaves you without any understanding of why the blue button worked (or didn’t). Was it really the color, or was it something else — like contrast with the background or user preferences — that drove the change? 

Learning-Based Approach 

Now let’s reframe this test with a focus on learnings. Instead of testing just two colors, you could test multiple button colors (e.g., red, blue, green, yellow) while also considering other factors like contrast with the page background. 

Your new hypothesis might be: “The visibility of the CTA button, influenced by its contrast with the background, affects the CTR. We hypothesize that buttons with higher contrast will perform better across the board.” 

By broadening the test, you’re not only testing for an immediate outcome but also gathering insights into how users respond to various visual elements. After running the test, you discover that buttons with higher contrast consistently perform better, regardless of color. 

This insight can then be applied to other areas of your site, such as text visibility, image placement, or product page design. 

Key Takeaway: 

A learning-focused approach reveals deeper insights that can be leveraged far beyond the original test scenario. This shift turns every test into a stepping stone for future improvements. 

How to Design Hypotheses That Deliver Valuable Learnings

Learning-focused A/B testing starts with designing better hypotheses. A well-crafted hypothesis doesn’t just predict an outcome—it seeks to understand the underlying reasons for user behavior and outlines how you’ll measure it. 

Here’s how to design hypotheses that lead to more valuable insights: 1. Set Clear, Learning-Focused Goals 

Rather than aiming only for KPI improvements, set objectives that prioritize learning. For example, instead of merely trying to increase conversions, focus on understanding which elements of the checkout process create friction for users. 

By aligning your goals with broader business objectives, you ensure that every test contributes to long-term growth, not just immediate wins. 

2. Craft Hypotheses That Explore User Behavior 

A strong hypothesis is specific, measurable, and centered around understanding user behavior. Here’s a step-by-step guide to crafting one: 

● Start with a Clear Objective: Define what you want to learn. For instance, “We want to understand which elements of the checkout process cause users to abandon their carts.” 

● Identify the Variables: Determine the independent variable (what you change) and the dependent variable (what you measure). For example, the independent variable might be the number of form fields, while the dependent variable could be the checkout completion rate. 

● Explain the Why: A learning-focused hypothesis should explore the “why” behind the user behavior. For example, “We hypothesize that removing fields with radio buttons will increase conversions because users find these fields confusing.” 

3. Design Methodologies That Capture Deeper Insights 

A robust methodology is crucial for gathering reliable data and drawing meaningful conclusions. Here’s how to structure your tests:

● Consider Multiple Variations: Testing multiple variations allows you to uncover broader insights. For instance, testing different combinations of form fields, layouts, or input types helps identify patterns in user behavior. 

● Ensure Sufficient Sample Size & Duration: Use tools like an A/B test calculator to determine the sample size needed for statistical significance. Run your test long enough to gather meaningful data but avoid cutting it short based on preliminary results. 

● Track Secondary Metrics: Go beyond your primary KPIs. Measure secondary metrics, such as time on page, engagement, or bounce rates, to gain a fuller understanding of how users interact with your site. 

4. Apply Learnings Across the Customer Journey 

Once you’ve gathered insights from your tests, it’s time to apply them across your entire customer journey. This is where learning-focused testing truly shines: the insights you gain can inform decisions across all touchpoints, from marketing to product development. 

For example, if your tests reveal that users struggle with radio buttons during checkout, you can apply this insight to other forms across your site, such as email sign-ups, surveys, or account creation pages. By applying your learnings broadly, you unlock opportunities to optimize every aspect of the user experience. 

5. Establish a Feedback Loop 

Establish a feedback loop to ensure that these insights continuously inform your business strategy. Share your findings with cross-functional teams and regularly review how these insights can influence broader business objectives. This approach fosters a culture of experimentation and continuous improvement, where every department benefits from the insights gained through testing. 

Conclusion: Every Test is a Win 

When you shift your focus from short-term outcomes to long-term learnings, you transform your A/B testing program into a powerful engine for growth. Every

test—whether it results in immediate KPI gains or not—offers valuable insights that drive future strategy and improvement. 

With AB Tasty’s platform, you can unlock the full potential of learning-focused testing. Our tools enable you to design tests that consistently deliver value, helping your business move toward sustainable, long-term success. 

Ready to get started? Explore how AB Tasty’s tools can help you unlock the full potential of your A/B testing efforts. Embrace the power of learning, and you’ll find that every test is a win for your business.

Share on

LinkedIn Facebook Twitter

Article

Jul 18, 2024
4min read

Transaction Testing With AB Tasty’s Report Copilot

Hubert Wassner

Hubert Wassner

Share on

LinkedIn Facebook Twitter

Transaction testing, which focuses on increasing the rate of purchases, is a crucial strategy for boosting your website’s revenue. 

To begin, it’s essential to differentiate between conversion rate (CR) and average order value (AOV), as they provide distinct insights into customer behavior. Understanding these metrics helps you implement meaningful changes to improve transactions.

In this article, we’ll delve into the complexities of transaction metrics analysis and introduce our new tool, the “Report Copilot,” designed to simplify report analysis. Read on to learn more.

Transaction Testing

To understand how test variations impact total revenue, focus on two key metrics:

  • Conversion Rate (CR): This metric indicates whether sales are increasing or decreasing. Tactics to improve CR include simplifying the buying process, adding a “one-click checkout” feature, using social proof, or creating urgency through limited inventory.
  • Average Order Value (AOV): This measures how much each customer is buying. Strategies to enhance AOV include cross-selling or promoting higher-priced products.

By analyzing CR and AOV separately, you can pinpoint which metrics your variations impact and make informed decisions before implementation. For example, creating urgency through low inventory may boost CR but could reduce AOV by limiting the time users spend browsing additional products. After analyzing these metrics individually, evaluate their combined effect on your overall revenue.

Revenue Calculation

The following formula illustrates how CR and AOV influence revenue:

Revenue=Number of Visitors×Conversion Rate×AOV

In the first part of the equation (Number of Visitors×Conversion Rate), you determine how many visitors become customers. The second part (×AOV) calculates the total revenue from these customers.

Consider these scenarios:

  • If both CR and AOV increase, revenue will rise.
  • If both CR and AOV decrease, revenue will fall.
  • If either CR or AOV increases while the other remains stable, revenue will increase.
  • If either CR or AOV decreases while the other remains stable, revenue will decrease.
  • Mixed changes in CR and AOV result in unpredictable revenue outcomes.

The last scenario, where CR and AOV move in opposite directions, is particularly complex due to the variability of AOV. Current statistical tools struggle to provide precise insights on AOV’s overall impact, as it can experience significant random fluctuations. For more on this, read our article “Beyond Conversion Rate.”

While these concepts may seem intricate, our goal is to simplify them for you. Recognizing that this analysis can be challenging, we’ve created the “Report Copilot” to automatically gather and interpret data from variations, offering valuable insights.

Report Copilot

The “Report Copilot” from AB Tasty automates data processing, eliminating the need for manual calculations. This tool empowers you to decide which tests are most beneficial for increasing revenue.

Here are a few examples from real use cases.

Winning Variation:

The left screenshot provides a detailed analysis, helping users draw conclusions about their experiment results. Experienced users may prefer the summarized view on the right, also available through the Report Copilot.

Complex Use Case:


The screenshot above demonstrates a case where CR and OAV have opposite trends and need a deeper understanding of the context.

It’s important to note that the Report Copilot doesn’t make decisions for you; it highlights the most critical parts of your analysis, allowing you to make informed choices.

Conclusion

Transaction analysis is complex, requiring a breakdown of components like conversion rate and average order value to better understand their overall effect on revenue. 

We’ve developed the Report Copilot to assist AB Tasty users in this process. This feature leverages AB Tasty’s extensive experimentation dashboard to provide comprehensive, summarized analyses, simplifying decision-making and enhancing revenue strategies.

Share on

LinkedIn Facebook Twitter

Article

Jul 16, 2024
5min read

The Past, Present, and Future of Experimentation | Bhavik Patel

Maddie Ostrander

Maddie Ostrander

Share on

LinkedIn Facebook Twitter

What is the future of experimentation? Bhavik Patel highlights the importance of strategic planning and innovation to achieve meaningful results.

A thought leader in the worlds of CRO and experimentation, Bhavik Patel founded popular UK-based meetup community, CRAP (Conversion Rate, Analytics, Product) Talks, seven years ago to fill a gap in the event market – opting to cover a broad range of optimization topics from CRO, data analysis, and product management to data science, marketing, and user experience.

After following his passion throughout the industry from acquisition growth marketing to experimentation and product analytics, Bhavik landed the role of Product Analytics & Experimentation Director at product measurement consultancy, Lean Convert, where his interests have converged. Here he is scaling a team and supporting their development in data and product thinking, as well as bringing analytical and experimentation excellence into the organization.

AB Tasty’s CMO Marylin Montoya spoke with Bhavik about the future of experimentation and how we might navigate the journey from the current mainstream approach to the potentialities of AI technology.

Here are some of the key takeaways from their conversation.

The evolution of experimentation: a scientific approach.

Delving straight to the heart of the conversation, Bhavik talks us through the evolution of A/B testing, from its roots in the scientific method, to recent and even current practices – which involve a lot of trial and error to test basic variables. When projecting into the future, we need to consider everything from people, to processes, and technology.

Until recently, conversion rate optimization has mostly been driven by marketing teams, with a focus on optimizing the basics such as headlines, buttons, and copy. Over the last few years, product development has started to become more data driven. Within the companies taking this approach, the product teams are the recipients of the A/B test results, but the people behind these tests are the analytical and data science teams, who are crafting new and advanced methods, from a statistical standpoint.

Rather than making a change on the homepage and trying to measure its impact on outcome metrics, such as sales or new customer acquisition, certain organizations are taking an alternative approach modeled by their data science teams: focusing on driving current user activity and then building new products based on that data.

The future of experimentation is born from an innovative mindset, but also requires critical thinking when it comes to planning experiments. Before a test goes live, we must consider the hypothesis that we’re testing, the outcome metric or leading indicators, how long we’re going to run it, and make sure that we have measurement capabilities in place. In short, the art of experimentation is transitioning from a marketing perspective to a science-based approach.

Why you need to level up your experiment design today.

While it may be a widespread challenge to shift the mindset around data and analyst teams from being cost centers to profit-enablement centers, the slowing economy might have a silver lining: people taking the experimentation process a lot more seriously. 

We know that with proper research and design, an experiment can achieve a great ROI, and even prevent major losses when it comes to investing in new developments. However, it can be difficult to convince leadership of the impact, efficiency and potential growth derived from experimentation.

Given the current market, demonstrating the value of experimentation is more important than ever, as product and marketing teams can no longer afford to make mistakes by rolling out tests without validating them first, explains Bhavik. 

Rather than watching your experiment fail slowly over time, it’s important to have a measurement framework in place: a baseline, a solid hypothesis, and a proper experiment design. With experimentation communities making up a small fraction of the overall industry, not everyone appreciates the ability to validate, quantify, and measure the impact of their work,  however Bhavik hopes this will evolve in the near future.

Disruptive testing: high risk, high reward.

On the spectrum of innovation, at the very lowest end is incremental innovation, such as small tests and continuous improvements, which hits a local maximum very quickly. In order to break through that local maximum, you need to try something bolder: disruptive innovation. 

When an organization is looking for bigger results, they need to switch out statistically significant micro-optimizations for experiments that will bring statistically meaningful results.

Once you’ve achieved better baseline practices – hypothesis writing, experiment design, and planning – it’s time to start making bigger bets and find other ways to measure it.

Now that you’re performing statistically meaningful tests, the final step in the evolution of experimentation is reverse-engineering solutions by identifying the right problem to solve. Bhavik explains that while we often focus on prioritizing solutions, by implementing various frameworks to estimate their reach and impact, we ought to take a step back and ask ourselves if we’re solving the right problem.

With a framework based on quality data and research, we can identify the right problem and then work on the solution, “because the best solution for the wrong problem isn’t going to have any impact,” says Bhavik.

What else can you learn from our conversation with Bhavik Patel?

  • The common drivers of experimentation and the importance of setting realistic expectations with expert guidance.
  • The role of A/B testing platforms in the future of experimentation: technology and interconnectivity.
  • The potential use of AI in experimentation: building, designing, analyzing, and reporting experiments, as well as predicting test outcomes. 
  • The future of pricing: will AI enable dynamic pricing based on the customer’s behavior?

About Bhavik Patel

A seasoned CRO expert, Bhavik Patel is the Product Analytics & Experimentation Director at Lean Convert, leading a team of optimization specialists to create better online experiences for customers through experimentation, personalization, research, data, and analytics.
In parallel, Bhavik is the founder of CRAP Talks, an acronym that stands for Conversion Rate, Analytics and Product, which unites CRO enthusiasts with thought leaders in the field through inspiring meetup events – where members share industry knowledge and ideas in an open-minded community.

About 1,000 Experiments Club

The 1,000 Experiments Club is an AB Tasty-produced podcast hosted by John Hughes, Head of Marketing at AB Tasty. Join John as he sits down with the experts in the world of experimentation to uncover their insights on what it takes to build and run successful experimentation programs.

Share on

LinkedIn Facebook Twitter

Article

May 23, 2024
5min read

Mutually Exclusive Experiments: Preventing the Interaction Effect

Hubert Wassner

Hubert Wassner

Share on

LinkedIn Facebook Twitter

What is the interaction effect?

If you’re running multiple experiments at the same time, you may find their interpretation to be more difficult because you’re not sure which variation caused the observed effect. Worse still, you may fear that the combination of multiple variations could lead to a bad user experience.

It’s easy to imagine a negative cumulative effect of two visual variations. For example, if one variation changes the background color, and another modifies the font color, it may lead to illegibility. While this result seems quite obvious, there may be other negative combinations that are harder to spot.

Imagine launching an experiment that offers a price reduction for loyal customers, whilst in parallel running another that aims to test a promotion on a given product. This may seem like a non-issue until you realize that there’s a general rule applied to all visitors, which prohibits cumulative price reductions – leading to a glitch in the purchase process. When the visitor expects two promotional offers but only receives one, they may feel frustrated, which could negatively impact their behavior.

What is the level of risk?

With the previous examples in mind, you may think that such issues could be easily avoided. But it’s not that simple. Building several experiments on the same page becomes trickier when you consider code interaction, as well as interactions across different pages. So, if you’re interested in running 10 experiments simultaneously, you may need to plan ahead.

A simple solution would be to run these tests one after the other. However, this strategy is very time consuming, as your typical experiment requires two weeks to be performed properly in order to sample each day of the week twice.

It’s not uncommon for a large company to have 10 experiments in the pipeline and running them sequentially will take at least 20 weeks. A better solution would be to handle the traffic allocated to each test in a way that renders the experiments mutually exclusive.

This may sound similar to a multivariate test (MVT), except the goal of an MVT is almost the opposite: to find the best interaction between unitary variations.

Let’s say you want to explore the effect of two variation ideas: text and background color. The MVT will compose all combinations of the two and expose them simultaneously to isolated chunks of the traffic. The isolation part sounds promising, but the “all combinations” is exactly what we’re trying to avoid. Typically, the combination of the same background color and text will occur. So an MVT is not the solution here.

Instead, we need a specific feature: A Mutually Exclusive Experiment.

What is a Mutually Exclusive Experiment (M2E)?

AB Tasty’s Mutually Exclusive Experiment (M2E) feature enacts an allocation rule that blocks visitors from entering selected experiments depending on the previous experiments already displayed. The goal is to ensure that no interaction effect can occur when a risk is identified.

How and when should we use Mutually Exclusive Experiments?

We don’t recommend setting up all experiments to be mutually exclusive because it reduces the number of visitors for each experiment. This means it will take longer to achieve significant results and the detection power may be less effective.

The best process is to identify the different kinds of interactions you may have and compile them in a list. If we continue with the cumulative promotion example from earlier, we could create two M2E lists: one for user interface experiments and another for customer loyalty programs. This strategy will avoid negative interactions between experiments that are likely to overlap, but doesn’t waste traffic on hypothetical interactions that don’t actually exist between the two lists.

What about data quality?

With the help of an M2E, we have prevented any functional issues that may arise due to interactions, but you might still have concerns that the data could be compromised by subtle interactions between tests.

Would an upstream winning experiment induce false discovery on downstream experiments? Alternatively, would a bad upstream experiment make you miss an otherwise downstream winning experiment? Here are some points to keep in mind:

  • Remember that roughly eight tests out of 10 are neutral (show no effect), so most of the time you can’t expect an interaction effect – if no effect exists in the first place.
  • In the case where an upstream test has an effect, the affected visitors will still be randomly assigned to the downstream variations. This evens out the effect, allowing the downstream experiment to correctly measure its potential lift. It’s interesting to note that the average conversion rate following an impactful upstream test will be different, but this does not prevent the downstream experiment from correctly measuring its own impact.
  • Remember that the statistical test is here to take into account any drift of the random split process. The drift we’re referring to here is the fact that more impacted visitors of the upstream test could end up in a given variation creating the illusion of an effect on the downstream test. So the gain probability estimation and the confidence interval around the measured effect is informing you that there is some randomness in the process. In fact, the upstream test is just one example among a long list of possible interfering events – such as visitors using different computers, different connection quality, etc.

All of these theoretical explanations are supported by an empirical study from the Microsoft Experiment Platform team. This study reviewed hundreds of tests on millions of visitors and saw no significant difference between effects measured on visitors that saw just one test and visitors that saw an additional upstream test.

Conclusion

While experiment interaction is possible in a specific context, there are preventative measures that you may take to avoid functional loss. The most efficient solution is the Mutually Exclusive Experiment, allowing you to eliminate the functional risks of simultaneous experiments, make the most of your traffic and expedite your experimentation process.

References:

https://www.microsoft.com/en-us/research/group/experimentation-platform-exp/articles/a-b-interactions-a-call-to-relax/

 

Share on

LinkedIn Facebook Twitter

Article

May 14, 2024
6min read

The Truth Behind the 14-Day A/B Test Period

Hubert Wassner

Hubert Wassner

Share on

LinkedIn Facebook Twitter

The A/B testing method involves a simple process: create two variations, expose them to your customer, collect data, and analyze the results with a statistical formula. 

But, how long should you wait before collecting data? With 14 days being standard practice, let’s find out why as well as any exceptions to this rule.

Why 14 days?

To answer this question we need to understand what we are fundamentally doing. We are collecting current data within a short window, in order to forecast what could happen in the future during a more extended period. To simplify this article, we will only focus on explaining the rules that relate to this principle. Other rules do exist, which mostly correlate to the number of visitors, but this can be addressed in a future article.

The forecasting strategy relies on the collected data containing samples of all event types that may be encountered in the future. This is impossible to fulfill in practice, as periods like Christmas or Black Friday are exceptional events relative to the rest of the year. So let’s focus on the most common period and set aside these special events that merit their own testing strategies.

If the future we are considering relates to “normal” times, our constraint is to sample each day of the week uniformly, since people do not behave the same on different days. Simply look at how your mood and needs shift between weekdays and weekends. This is why a data sampling period must include entire weeks, to account for fluctuations between the days of the week. Likewise, if you sample eight days for example, one day of the week will have a doubled impact, which doesn’t realistically represent the future either.

This partially explains the two-week sampling rule, but why not a longer or shorter period? Since one week covers all the days of the week, why isn’t it enough? To understand, let’s dig a little deeper into the nature of conversion data, which has two dimensions: visits and conversions.

  • Visits: as soon as an experiment is live, every new visitor increments the number of visits.
  • Conversions: as soon as an experiment is live, every new conversion increments the number of conversions.

It sounds pretty straightforward, but there is a twist: statistical formulas work with the concept of success and failure. The definition is quite easy at first: 

  • Success: the number of visitors that did convert.
  • Failures: the number of visitors that didn’t convert.

At any given time a visitor may be counted as a failure, but this could change a few days later if they convert, or the visit may remain a failure if the conversion didn’t occur. 

So consider these two opposing scenarios: 

  • A visitor begins his buying journey before the experiment starts. During the first days of the experiment he comes back and converts. This would be counted as a “success”, but in fact he may not have had time to be impacted by the variation because the buying decision was made before he saw it. The problem is that we are potentially counting a false success: a conversion that could have happened without the variation.
  • A visitor begins his buying journey during the experiment, so he sees the variation from the beginning, but doesn’t make a final decision before the end of the experiment – finally converting after it finishes. We missed this conversion from a visitor who saw the variation and was potentially influenced by it.

These two scenarios may cancel each other out since they have opposite results, but that is only true if the sample period exceeds the usual buying journey time. Consider a naturally long conversion journey, like buying a house, measured within a very short experiment period of one week. Clearly, no visitors beginning the buying journey during the experiment period would have time to convert. The conversion rates of these visitors would be artificially in the realm of zero – no proper measurements could be done in this context. In fact, the only conversions you would see are the ones from visitors that began their journey before the variation even existed. Therefore, the experiment would not be measuring the impact of the variation. 

The delay between the effective variation and the conversion expedites the conversion rate. In order to mitigate this problem, the experiment period has to be twice as long as the standard conversion journey. Doing so ensures that visitors entering the experiment during the first half will have time to convert. You can expect that people who began their journey before the experiment and people entering during the second half of the experiment period will cancel each other out: The first group will contain conversions that should not be counted, and some of the second group’s conversions will be missing. However, a majority of genuine conversions will be counted.

That’s why a typical buying journey of one week results in a two-week experiment, offering the right balance in terms of speed and accuracy of the measurements.

Exceptions to this rule

A 14-day experiment period doesn’t apply to all cases. If the delay between the exposed variation and the conversion is 1.5 weeks for instance, then your experiment period should be three weeks, in order to cover the usual conversion delay twice. 

On the other hand, if you know that the delay is close to zero, such as in the case of a media website, where you are trying to optimize the placement of an advertisement frame on a page where visitors only stay a few minutes, you may think that one day would be enough based on the this logic, but it’s not. 

The reason being that you would not sample every day of the week, and we know from experience that people do not behave the same way throughout the week. So even in a zero-delay context, you still need to conduct the experiment for an entire week.

Takeaways: 

  1. Your test period should mirror the conditions of your expected implementation period.
  2. Sample each day of the week in the same way.
  3. Wait an integer number of weeks before closing an A/B test.

Respecting these rules will ensure that you’ll have clean measures. The accuracy of the measure is defined by another parameter of the experiment: the total number of visitors. We’ll address this topic in another article – stay tuned.

Share on

LinkedIn Facebook Twitter

Article

Feb 2, 2024
10min read

A/A Testing: What is it and When Should You Use it?

Hubert Wassner

Hubert Wassner

Share on

LinkedIn Facebook Twitter

A/A tests are a legacy from the early days of A/B testing. It’s basically creating an A/B test where two identical versions of a web page or element are tested against each other. Variation B is just a copy of A without any modification.

One of the goals of A/A tests is to check the effectiveness and accuracy of testing tools. The expectation is that, if no winner is declared, the test is a success. Whereas detecting a statistical difference would mean a failure, indicating a problem somewhere in the pipeline.

But it’s not always that simple. We’ll dive into this type of testing and the statistics and tech behind the scenes. We’ll look at why a failed A/A test is not a proof of pipeline failure, and that a successful A/A test isn’t a foolproof sanity check.

What is tested during an A/A test?

Why is there so much buzz around A/A testing? An A/A test can be a way to verify two components of an experimentation platform: 

  1. The statistical tool: It may be possible that the formulas chosen don’t fit the real nature of the data, or may contain bugs.
  2. The traffic allocation: The split between variations must be random and respect the proportions it has been given. When a problem occurs, we talk about Sample Ratio Mismatch (SRM); that is, the observed traffic does not match the allocation setting. This means that the split has some bias impacting the analysis quality.
    Let’s explore this in more detail.

Statistical tool test

Let’s talk about a “failed” A/A test

The most common idea behind A/A tests is that the statistical tool should yield no significant difference. It is considered a “failed” A/A test if you detect a difference in performance during an A/A test. 

However, to understand how weak this conclusion is, you need to understand how statistical tests work. Let’s say that your significance threshold is 95%. This means that there is still a 5% chance that the difference you see is a statistical fluke and no real difference exists between the variations. So even with a perfectly working statistical tool, you still have one chance in twenty (1/20=5%) that you will have a “failed” A/A test and you might start looking for a problem that may not exist.

With that in mind, an acceptable statistical procedure would be to perform 20 A/A tests and expect to have 19 that yield no statistical difference, and one that does detect a significant difference. And even in this case, if 2 tests show significant results, it’s a sign of a real problem. In other words, having 1 successful A/A test is in fact not enough to validate a statistical tool. To validate it fully, you need to show that the tests are successful 95% of the time (=19/20).

Therefore, a meaningful approach would be to perform hundreds of A/A tests and expect ~5% of them to “fail”. It’s worth noting that if it “fails” less than 5% of the time it’s also a problem, maybe indicating that the statistical test simply says “no” too often, leading to a strategy that never detects any winning variation. So one A/A “failed” test doesn’t tell much in reality. 

What if it’s a “successful A/A test”? 

A “successful” A/A test (yielding no difference) is not proof that everything is working as it should. To understand why, you need to check another important tool in an A/B test: the sample size calculator.

In the following example, we see that from a 5% conversion rate, you need around 30k visitors per variation to reach the 95% significance level if a variation yields a 10% MDE (Minimal Detectable Effect).

But in the context of an A/A test, the Minimal Detectable Effect (MDE) is in fact 0%. Using the same formula, we’ll plug 0% as MDE.

At this point, you will discover that the form does not let you put a 0% here, so let’s try a very small number then. In this case, you get almost 300M visitors, as seen below.

In fact, to be confident that there is exactly no difference between two variations, you need an infinite number of visitors, which is why the form does not let you set 0% as MDE.

Therefore, a successful A/A test only tells you that the difference between the two variations is smaller than a given number but not that the two variations perform exactly the same.

This problem comes from another principle in statistical tests: the power. 

The power of a test is the chance that you discover a difference if there is any. In the context of an A/A test, this refers to the chance you discover a statistically significant discrepancy between the two variations’ performance. 

The more power, the more chance you will discover a difference. To raise the power of a test you simply raise the number of visitors.

You may have noticed that in the previous screenshots, tests are usually powered at 80%. This means that even if a difference exists between the variations in performance, 20% of the time you will miss it. So one “successful” A/A test (yielding no statistical difference) may just be an occurrence of this 20%. In other words, having just one successful A/A test doesn’t ensure the efficiency of your experimentation tool. You may have a problem and there is a 20% chance that you missed it. Additionally, reaching 100% of power will need an infinite number of visitors, making it impractical.

How do we make sure we can trust the statistical tool then? If you are using a platform that is used by thousands of other customers, chances are that the problem would have already been discovered. 

Because statistical software does not change very often and it is not affected by the variation content (whereas the traffic allocation might change, as we will see later), the best option is to trust your provider, or you can double-check the results with an independent provider. You can find a lot of independent calculators on the web. They only need the number of visitors and the number of conversions for each variation to provide the results making it quick to implement.

Traffic allocation test

In this part, we only focus on traffic, not conversions. 

The question is: does the splitting operation work as it should? We call this kind of failure a SRM or Sample Ratio Mismatch. You may ask yourself how a simple random choice could fail. In fact, the failure happens either before or after the random choice. 

The following demonstrates two examples where that can happen:

  • The variation contains a bug that may crash some navigators. In this case, the corresponding variation will lose visitors. The bug might depend on the navigator and then you will end up with bias in your data.
  • If the variation gives a discount coupon (or any other advantage), and some users find a way to force their navigator to run the variation (to get the coupon), then you will have an excess of visitors for that variation that is not due to random chance, which results in biased data.


It’s hard to detect with the naked eye because the allocation is random, so you never get sharp numbers. 

For instance, a 50/50 allocation never precisely splits the traffic in groups with the exact same size. As a result, we would need statistical tools to check if the split observed corresponds with the desired allocation. 

SRM tests exist. They work more or less the same way as an A/B test except that the SRM formula indicates whether there is a difference between the desired allocation and what really happened. If there is indeed an SRM, then there is a chance that this difference is not due to pure randomness. This means that some data is lost or bias occurred during the experiment entailing trust for future (real) experiments.

On the one hand, detecting an SRM during an A/A test sounds like a good idea. On the other hand, if you think operationally it might not be that useful because the chance of a SRM is low.  

Even if some reports say that they are more frequent than you may think, most of the time it happens on complex tests. In that sense, checking SRM within an A/A test will not help you to prevent having one on a more complex experiment later. 

If you find a Sample Ration Mismatch on a real experiment or in an A/A test, the following actions remain the same: find the cause, fix it, and restart the experiment. So why waste time and traffic on an A/A test that will give you no information? A real experiment would have given you real information if it worked fine on the first try. If a problem does occur, we would detect it even in a real experiment since we only consider traffic and not conversions.

A/A tests are also unnecessary since most trustworthy A/B testing platforms (like AB Tasty) do SRM checks on an automated basis. So if an SRM occurs, you will be notified anyway. 

So where does this “habit” of practicing A/A tests come from?

Over the years, it’s something that engineers building A/B testing platforms have done. It makes sense in this case because they can run a lot of automated experiments, and even simulate users if they don’t have enough at hand, performing a sound statistical approach to A/A tests. 

They have reasons to doubt the platform in the works and they have the programming skills to automatically create hundreds of A/A tests to test it properly. Since these people can be seen as pioneers, their voice on the web is loud when they explain what an A/A test is and why it’s important (from an engineering perspective).

However, for a platform user/customer, the context is different as they’ve paid for a ready-to- use and trusted platform and can start a real experiment as soon as possible to get a return on investment. Therefore, it makes little sense to waste time and traffic on an A/A test that won’t provide any valuable information.

Why sometimes it might be better to skip A/A tests

We can conclude that a failed A/A test is not a problem and that a successful one is not  proof of sanity. 

In order to gain valuable insights from A/A tests, you would need to perform hundreds of them with an infinite number of visitors. Moreover, an efficient platform like AB Tasty does the corresponding checks for you.

That’s why, unless you are developing your own A/B testing platform, running an A/A test may not give you the insights you’re looking for. A/A tests require a considerable amount of time and traffic that could otherwise be used to conduct A/B tests that could give you valuable insights on how to optimize your user experience and increase conversions. 

When it makes sense to run an A/A test

It may seem that running A/A tests may not be the right call after all. However, there may be a couple of reasons why it might still be useful to perform A/A tests. 

First is when you want to check the data you are collecting and compare it to data already collected with other analytics tools but keep in mind that you will never get the exact same results. The reason is that most of the metric definitions vary on different tools. Nonetheless this comparison is an important onboarding step to ensure that the data is properly collected.

The other reason to perform an A/A test is to know the reference value for your main metrics so you can establish a baseline to analyze your future campaigns more accurately. For example, what is your base conversion rate and/or bounce rate? Which of these metrics need to be improved and are, therefore, a good candidate for your first real A/B test?

This is why AB Tasty has a feature that helps users build A/A tests dedicated to reach these goals and avoids the pitfalls of “old school”  methods that are not useful anymore. With our new A/A test feature, A/A test data is collected in one variant (not two); let’s call this an “A test”. 

This allows you to have a more accurate estimation of these important metrics as the more data you have, the more accurate the measurements are. Meanwhile, in a classic A/A test, data is collected in two different variants which provides less accurate estimates since you have less data for each variant.

With this approach, AB Tasty enables users to automatically set up A/A tests, which gives better insights than classic “handmade” A/A tests.

Share on

LinkedIn Facebook Twitter