Tag: EN

Article

Jun 22, 2023

5min read

A New Chapter for Flagship as it Merges with the AB Tasty Website

AB Tasty

We are excited to share that as a part of our ongoing strategy to optimize how you access AB Tasty’s suite of experimentation and personalization tools, Flagship by AB Tasty is now evolving to join the AB Tasty brand and website.

This doesn’t mean your favorite experience rollout and feature management tools are disappearing, but rather it’s part of a new exciting chapter for AB Tasty with the goal to make all our features available in one place under one name.

We have merged the AB Tasty and Flagship websites. All resources and landing pages previously hosted on Flagship’s website (flagship.io) can now be found in one location on the AB Tasty website (abtasty.com).

This branding evolution means the Flagship name will be phased out and then retired. While we feel a little nostalgic for the old name, the end goal is to make it easier to get access to the AB Tasty umbrella of solutions and features and join them together to keep our promise of being your go-to-platform for improving and optimizing the customer experience.

If you have questions about what this change means for you, you’ve come to the right place. Below we will dig into what is changing, helpful links and resources and some general FAQs.

As always, our team of AB Tasty magic makers are available to answer any additional questions that might pop up along the way. If you have any more questions after reading this, don’t hesitate to send us an email at hello@abtasty.com and we will update this page as needed.

How are AB Tasty and Flagship related?

AB Tasty and Flagship have always been the same company, just with different names for the server-side solutions and client-side solutions.

AB Tasty’s experimentation suite enables brands to carry out client-side A/B testing and personalizations in order to provide a richer digital experience and boost conversions.

Meanwhile, Flagship by AB Tasty is also built to provide richer experiences that convert through risk-free feature management, server-side experimentation and personalization. Again, same company, just different ways of helping brands provide the best experience for their customers.

What do you mean when you say merge? Will the Flagship website be gone for good?

Yes, everything on the Flagship website (flagship.io) has moved over to the AB Tasty website (abtasty.com). This means links to existing landing pages and resources are all redirected to AB Tasty, and any new resources will be posted directly on AB Tasty from here on out. Easily access resources like e-books, blogs, guides and more by clicking on the resources tab above or following the link here.

Why are we merging the Flagship and AB Tasty websites and names?

From the start, our focus has always been on what we do best, which is giving clients the tools they need to validate ideas while maximizing impact, minimizing risk and accelerating time to market.

Marketing teams and tech teams are working more closely together than ever before to bring new features to market to stay competitive. Our customer-first approach means we want to make our features more accessible and find the tools you need for all your experimentation and personalization needs. For this reason, we have decided to bring Flagship to the AB Tasty website and to position it as AB Tasty’s Feature Experimentation and Experience Rollouts rather than as a separate solution.

Many of our client-side clients have evolved their experimentation needs to the point where they are running more advanced experiments and rolling out more advanced features. For our clients who are ready to start server-side experimentation, this change makes it much easier and faster to find all the information and support they need about all our features, including our server-side functionality, in one place.

What will happen to all the resources (blog posts, guides, e-books, etc.) on the Flagship.io?

As mentioned above, the Flagship content is now migrated and all links from flagship.io are redirected to the AB Tasty. From there, all our resources from guides to blog posts and e-books about feature management, experimentation and more can all be found on the AB Tasty website.

You’ll find your favorite content can be easily accessed here if you filter for the “Rollouts” and “Feature Experimentation” topics.

How can I log into my Flagship account? And where can I access the documentation and SDK libraries?

You can access your accounts by visiting abtasty.com and clicking the login button in the top right-hand corner.

All our documentation and SDKs will have the same links as before. You can access them below:

How will the merger affect existing customers of both Flagship and AB Tasty and the support they receive?

All our clients, regardless of whether they are using AB Tasty or Flagship or both, will not be affected. You can continue to use our platform for all your experimentation needs without any changes.

Likewise, you can expect to receive the same level of support and have access to the same dedicated team for client- and/or server-side experiments as before.

As always, your CSM will inform you in a timely manner when/if there any changes to the platform occur.

How will the merger affect new customers? Where can I sign up for a demo for AB Tasty’s Feature Experimentation and Rollouts?

If you’re new and you’d like to try out AB Tasty’s Feature Experimentation or Experience Rollouts, click the banner below or click the “Get a demo” button on the top right-hand corner of the page to explore how server-side experiments can positively impact your business.

A very special thank you to our customers and our partners for supporting us in this exciting evolution of AB Tasty. Your feedback and support helps shape important changes such as these, and we are grateful for it.

Have any additional questions about AB Tasty? Send us an email at hello@abtasty.com to let us know and stay tuned for more exciting updates and information still to come!

You might also like...

See all

Article

6min read

Minimal Detectable Effect: The Essential Ally for Your A/B Tests

Hubert Wassner

May 6, 2025

Article

2min read

How We Made AB Tasty’s Feature Experimentation SDK OpenFeature-Compatible — And Even Easier to Adopt

Julie Dumont

Apr 30, 2025

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Subscribe to
our Newsletter

Article

10min read

The Ultimate Guide to Experience Rollouts Using Feature Flags

Rowan Haddad

In modern software development, DevOps teams have shifted their attention on the continuous delivery of features to keep up with fast-changing market and consumer demands.

Teams now more than ever have to be in the driver’s seat when it comes to delivering these features and to whom.

This is why feature flags (or feature toggles) have become the ultimate tool to manage the release of new features.

What are experience rollouts?

When we talk about experience rollouts, we’re referring to the risk-free deployment of features that improve and optimize the customer experience.

This could be in the form of progressive deployments where features are gradually released to reduce the risk of big bang releases or by targeting new features to the most relevant users in order to personalize their experience.

But how do you ensure you’re delivering optimal experiences without negatively impacting the user experience? How can you minimize risk when rolling out new features and ensure that they actually meet your customers’ needs and expectations?

The answer to both these questions is feature flags.

Feature flags are a great solution to allow you to continuously deliver new features while limiting user access to these features, thereby reducing risk.

By decoupling deployment from release, feature flags give teams the power to choose who to send new features to and when. Thus, teams can continuously develop and deliver new features without having to make them available to all users.

What are feature flags?

Let’s start with the most basic definition of feature flags.

Feature flags are a software development tool that enable teams to turn functionalities on or off in order to safely test new features by separating code deployment from feature release.

They can also be referred to as feature toggles as they allow you to toggle a feature on or off by hiding that feature behind a flag and then deciding who to make this feature visible for.

This is particularly useful when you’re looking to personalize the customer experience according to the type of user. This means you can enable features to only target certain users to display the right content to the right audience at the right time, while tracking their performance over time.

With AB Tasty Rollouts, you can configure personalization campaigns, for example, to personalize the user experience for new visitors on mobile only and show them discount codes as a welcome offer. Therefore, you can define the targeted users and the flag (with its value) that will activate the discount code for a particular scenario; in this case, new users on mobile while monitoring the relevant KPIs.

Feature flags can be leveraged across different use cases. This is because there are many different types and categories of feature flags as seen in the image below and which one you choose depends on the purpose of using the flag in the first place.

For example, release toggles support dev teams as they write new features while experiment toggles are primarily used by product and marketing teams to facilitate A/B testing.

For this reason, feature flags can be used across a wide variety of use cases by multiple teams across an organization, especially when you have a feature management solution to manage all your flags.

In particular, feature flags give teams a very granular level of control and risk management over code, which can be important when modifying backend features that have a wide-ranging impact on how your system performs.

The following section will provide further details on what the term “experience rollouts” entails and discuss how feature flags can help you regain control of how you roll out experiences to your customers at the flip of a switch.

Progressive deployment and rollouts

Perhaps one of the greatest benefits of feature flags is their ability to mitigate risk when it comes to new feature releases.

This is because feature flags empower teams to release their features to any user groups of their choice.

Therefore, teams can safely test out their new features on a preselected number of users, whether with internal or external users, to validate functionality and gather feedback in order to make any necessary changes and optimize future feature releases. By continuously iterating features in real time during the release process, companies can provide more value to their customers and ensure customer satisfaction.

Sophisticated feature flagging functionalities give you the ability to closely monitor metrics that indicate how a new feature is performing and how well-received it is by users.

This way, should anything go wrong with a release, teams can minimize the blast radius and any negative impact due to a faulty feature. This also gives them the time necessary to address the issue by disabling the flag before releasing it to everyone else.

The best thing about progressive deployments and rollouts is that teams are essentially in the driver’s seat: they have control over who sees what and when, allowing them to maintain the momentum of CI/CD but with less risk.

Another great advantage of progressive rollouts is that it increases both the velocity of the development lifecycle and testing as teams can roll out releases in phases, they can quickly test on their chosen user group, make the iterations necessary and then run more tests.

Rollbacks

Just as you can roll out new features and experiences to your users, you can also easily roll back these features whenever needed with the help of feature flags.

This means that if anything goes wrong with any feature you’ve rolled out to your chosen users, you can quickly disable the flag so that users no longer have access to the feature.

Releasing new features to real-world users is always a risky endeavor and can cause real harm to your brand’s user relationships but it doesn’t have to be.

Now, after any feature release, teams can isolate the faulty or buggy individual feature(s) that have and perform a targeted rollback on them. With advanced third-party feature management platforms, you can roll back a feature in real-time by just toggling a single field with just one click.

AB Tasty is one such tool that allows you to roll out new features to subsets of users by assigning specific flag values to different user segments and comes with an automatic triggered rollback in case something goes wrong.

The automatic rollback option enables you to stop the deployment of a feature and to revert all the changes that have been made in order to ensure that your feature isn’t breaking your customer experience. This is done by defining the business KPI you want to track that would indicate the performance of your feature.

When this KPI is set, you will then associate a value (in %) which, if exceeded or reached, will trigger the rollback. To make the rollback significant, you must define a specific number of visitors from which the rollback comparison will be triggered.

When the conditions are met, the progressive rollout feature will be rolled back, which means that no more targeted users will see the feature.

Targeting

We’ve talked a lot about how you can use feature flags to allow certain users to see a feature while hiding it from others.

When you do a targeted rollout, you’re basically releasing new features to a predefined set of users rather than opting for the riskier big bang release.

Here’s a look at some targeting scenarios where feature flags do their best work:

Alpha and beta testing
A/B testing
Managing entitlements
Blocking users
Canary deployments/percentage rollouts
Ring deployments

There are many ways teams can progressively deploy and roll out features to a select audience. With the help of feature flags, teams can manage and streamline these deployment methods to perform highly granular user targeting.

AB Tasty Rollouts allows you to target users based on certain identifying attributes like beta testers, age group, or any other user attributes you have access to.

Furthermore, our integrations with third-party tools such as Segment, GA4, Mixpanel and Heap means that you can also target your test and personalization use cases with audiences built in these tools and then exporting these user groups or cohorts to AB Tasty to target them.

Flag management

To truly reap the benefits of feature flags, you have to know how to manage them effectively. Otherwise, you will end up with so many flags in your system that you start to lose track of which flag does what. This could ultimately lead to the most dangerous pitfall of feature flags: technical debt.

At that point, your code could become too complex that it will be difficult to manage and could negatively affect the quality of your code.

This is why feature management and feature management solutions are so essential today for modern software development teams. With such solutions, teams have access to advanced features to enable them to work with feature flags at scale and avoid the most common problems associated with them.

AB Tasty is one solution packed full of features that can help you avoid the dreaded technical debt with a clear, easy-to-use dashboard that all your teams, from development to product, can easily use to efficiently track and manage feature flags usage across your organization no matter how far along you are in your feature flag journey.

Furthermore, flags can be controlled from another platform using AB Tasty’s Remote Control API allowing teams to work from just one tool without having to log onto the platform. This saves a lot of time and effort as you can perform all AB Tasty tasks directly with API calls, including managing your projects, use cases, variations, variation groups, users, targeting keys, and your flags.

Experience rollouts with feature flags

As we’ve seen, the idea of experience rollouts revolves around rolling out your best features to end-users. This is when feature flags become the most powerful tool to ensure you’re only releasing optimal features that provide the best customer experience possible.

This is because feature flags give you the ability to progressively deploy and roll out new features to gather feedback from the users – giving you the most relevant feedback to iterate and optimize your releases. This will help your teams to make more informed, data-driven decisions to drive your conversion rates, ultimately aligning the user experience with business objectives.

Consequently, when you finally do a full release, you’re confident that you’re releasing features that provide the most value to your customers and so will have the best impact on your business in terms of revenue and conversions.

Do you want to deliver first-in-class customer experiences? Click on the “Get a Demo” button at the top to see for yourself what feature flags can do for your own experience rollouts.

You might also like...

See all

Article

6min read

Minimal Detectable Effect: The Essential Ally for Your A/B Tests

Hubert Wassner

May 6, 2025

Article

2min read

How We Made AB Tasty’s Feature Experimentation SDK OpenFeature-Compatible — And Even Easier to Adopt

Julie Dumont

Apr 30, 2025

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Subscribe to
our Newsletter

Article

Jun 20, 2023

9min read

A/B, Split or Multivariate Test: How to Choose the Right One

Anthony Brebion

In the fast-paced world of digital marketing, settling for anything less than the best user experience is simply not an option.

Every marketing strategy has room for improvement and unlocking more comes from recognizing hidden opportunities.

With analytics data and a little bit of creativity, you can uncover some valuable insights on how to optimize your conversion rate on your website or campaign landing pages. However, achieving structured and streamlined data from your assumptions requires diligent testing.

Marketing professionals have steadily used different testing methodologies such as A/B testing, split testing, multivariate testing and multipage testing to increase conversion rates and enhance digital performance.

Experimenting and testing are essential as they eliminate opinions and bias from the decision-making process, ensuring data-driven decisions.

With the availability of many diverse testing options, it can be challenging to find your starting point. In this article, we’ll dive into the specifics of different forms of testing to help you navigate this testing landscape.

What is A/B testing?

flowers-366155_1280

A/B testing is a method of website optimization where you are comparing two versions of the same page: variation A and variation B. For the comparison, it’s common to look at the conversion rates and metrics that matter to your business (clicks, page views, purchases, etc) while using live traffic.

It’s also possible to do an A/B/C/D test when you need to test more than two content variations. The A/B/C/D method will allow you to test three or more variations of a page at once instead of testing only one variation against the control version of the page.

When to use A/B tests?

A/B tests are an excellent method to test radically different ideas for conversion rate optimization or small changes on a page.

A/B testing is the right method to choose if you don’t have a large amount of traffic to your site. Why is this? A/B tests can deliver reliable data very quickly, without a large amount of traffic. This is a great approach to experimentation to maximize test time to achieve fast results.

If you have a high-traffic website, you can evaluate the performance of a much broader set of variations. However, there is no need to test 20 different variations of the same element, even if you have adequate traffic. It’s important to have a strategy when approaching experimentation.

Want to start testing? AB Tasty is the best-in-class experience optimization platform that empowers you to create a richer digital experience – fast. From experimentation to personalization, this solution can help you activate and engage your audience to boost your conversions.

Split testing vs A/B testing

A/B tests and split tests are essentially the same concept.

“A/B” refers to the two variations of the same URL where changes are made “live” using Javascript on the original page. SaaS tools that provide you with a visual editor, like AB Tasty, allow you to create these changes quickly without technical knowledge.

Meanwhile, “split” refers to the traffic redirection towards one variation or another, each hosted on its own URL and fully redesigned in the code.

You can consider A/B tests to work the same as split tests.

The variation page may differ in many aspects depending on the testing hypothesis you put forth and your industry goals (layout, design, pictures, headlines, sub-headlines, calls to action, offers, button colors, etc.).

In any case, the number of conversions on each page’s variation is compared once each variation gets enough visitors.

In A/B tests, the impact of the design as a whole is tracked, not individual elements – even though many design elements might be changed on variations simultaneously.

TIP: Keep in mind that testing is all about comparing the performances of variations. It’s recommended not to make too many changes between the control and variation versions of the page at the same time. You should limit the number of changes to better understand the impact of the results. In the long term, a continuous improvement process will lead to better and lasting performance.

What is multivariate testing?

magic-cube-378543_1280

Multivariate tests or multi-variant tests are the same as A/B tests in their core mechanism and philosophy. The difference is that multivariate testing allows you to compare a higher number of variables and the interactions between each other. In other words, you can test and track changes to multiple sections on a single page.

For multivariate testing, you need to identify a few key page sections and then create variations for those sections specifically. You aren’t creating variations of a whole page as you do while A/B testing.

TIP: Use multivariate testing when several element combinations on your website or landing page are called into question.

Multivariate testing reveals more information about how these changes to multiple sections interact with one another. In multivariate tests, website traffic is split into each possible combination of a page – where the effectiveness of the changes is measured.

It’s very common to use multivariate testing to optimize an existing website or landing page without making a significant investment in redesign.

Although this type of testing can be perceived as an easier way of experimentation – keep in mind that multivariate testing is more complicated than traditional A/B testing.

Multivariate tests are best suited for more advanced testers because they give many more possibilities of combinations for visitors to experience on your website. Too many changes on a page at once can quickly add up. You don’t want to be left with a very large number of combinations that must be tested.

Multivariate test example

Let’s say that you’ve decided to run a multivariate test on one of your landing pages. You choose to change two elements on your landing page. On the first variation, you swap an image for a video, and on the second variation, you swap the image for a slider.

For each page variation, you add another version of the headline. This means that now you have three versions of the main content and two versions of the headline. This is equal to six different combinations of the landing page.

	Image	Video	Slider
Headline 1	Combination 1	Combination 2	Combination 3
Headline 2	Combination 4	Combination 5	Combination 6

After only changing two sections, you quickly have six variations. This is where multivariate testing can get tricky.

When to use multivariate testing?

Multivariate tests are recommended for sites with a large amount of daily traffic. You will need a site with a high volume of traffic to test multiple combinations, and it will take a longer time to obtain meaningful data from the test.

mvt — AB Tasty’s reporting allows you to weigh up each element’s impact on the conversion rate

The multivariate testing method will allow you to incrementally improve an existing design, while the test results can be used to apply to a larger website or landing page redesign.

What is multipage testing?

Multipage testing is an experimentation method similar to standard A/B testing. As we’ve discussed, in A/B testing, changes can be made to one specific page or to a group of pages.

If the changed element appears on several pages, you can choose whether or not to change it on each page. However, if the element is on several pages but it’s not identical, appears at a different place or has a different name, you’ll have to set up a multipage test.

Multipage tests allow you to implement changes consistently over several pages.

This means that multipage tests allow you to link together variations of different pages and are especially useful when funnel testing.

In multipage tests, site visitors are funneled into one funnel version or the other. You need to track the way visitors interact with the different pages they are shown so you can determine which funnel variation is the most effective.

You must ensure that the users see a consistent variation of changes throughout a set of pages. This is key to getting usable data and allows one variation to be fairly tested against another.

Multipage test example

Let’s say you want to conduct a multipage test with a free shipping coupon displayed in the funnel at different places. You want to run the results of this test against the original purchase funnel without a coupon.

For example, you could offer visitors a free shipping coupon on a product category page – where they can see “Free shipping over €50” as a static banner on the page. Once the visitor adds a product to the shopping cart, you can show them a new dynamic message according to the cart balance – “Add €X to your cart for free shipping”.

In this case, you can experiment with the location of the message (near the “Proceed to checkout” button, near the “Continue shopping” button, near the shipping cost for his order or somewhere else) and with the call-to-action variations of the message.

This kind of test will help you understand visitors’ purchase behavior better – i.e. how does the placement of a free shipping coupon reduce shopping cart abandonment and increase sales? After enough visitors come to the end of the purchase funnel through the different designs, you will be able to compare the effect of design styles easily and effectively.

How to test successfully?

Remember that the pages being tested need to receive substantial traffic so the tests will give you some relevant data to analyze.

Whether you use A/B testing, split testing, multivariate testing or multipage testing to increase your conversion rate or performance, remember to use them wisely.

Each type of test has its own requirements and is uniquely suited to specific situations, with advantages and disadvantages.

Using the proper test for the right situation will help you get the most out of your site and the best return on investment for your testing campaign. Even though testing follows a scientific method, there is no need for a degree in statistics when working with AB Tasty.

You might also like...

See all

Article

6min read

Minimal Detectable Effect: The Essential Ally for Your A/B Tests

Hubert Wassner

May 6, 2025

Article

2min read

How We Made AB Tasty’s Feature Experimentation SDK OpenFeature-Compatible — And Even Easier to Adopt

Julie Dumont

Apr 30, 2025

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Subscribe to
our Newsletter

Article

Jun 7, 2023

8min read

10 Generative AI Ideas for Your Experimentation Roadmap

Emily Healy

Artificial intelligence has been a recurring theme for decades. However, it’s no longer science fiction – it’s a reality.

Since OpenAI launched its own form of generative AI, ChatGPT, in November 2022, the world has yet to stop talking about its striking capabilities. It’s particularly fascinating to see just how easy it is to get results after interacting with this bot which is comprised of deep-learning algorithms for natural language processing.

Even Google quickly followed by launching a new and experimental project, Gemini, to revolutionize its own Search. By harnessing the power of generative AI and the capacity of large language models, Google is seeking to take its search process to the next level.

Given the rapid growth of this technological advancement over the past few months, it’s time that we talk about generative AI in the context of A/B testing and experimentation.

Whether you’re curious about how AI can impact your experiments or are ready for inspiration we’ll discuss some of our ideas around using AI for A/B testing, personalization, and conversion rate optimization.

What is generative AI?

Generative AI is a type of artificial intelligence that doesn’t have programming limitations, which allows it to generate new content (think ChatGPT). Instead of following a specific, pre-existing dataset, generative AI learns from indexing extensive data, focusing on patterns and using deep learning techniques and neural networks to create human-like content based on its learnings.

The way algorithms capture ideas is similar to how humans gather inspiration from previous experiences to create something unique. Based on the large amounts of data used to craft generative AI’s learning abilities, it’s capable of outputting high-quality responses that are similar to what a human would create.

However, some concerns need to be addressed:

Biased information: Artificial intelligence is only as good as the datasets used to train it. Therefore if the data used to train it has biases, it may create “ideas” that are equally biased or flawed.
Spreading misinformation: There are many concerns about the ethics of generative AI and sharing information directly from it. It’s best practice to fact-check any content written by AI to avoid putting out false or misleading information.
Content ownership: Since content generated with AI is not generated by a human, can you ethically can claim it as your own idea? In a similar sense, the same idea could potentially be generated elsewhere by using a similar prompt. Copywriting and ownership are then called into question here.
Data and privacy: Data privacy is always a top-of-mind concern. With the new capabilities of artificial intelligence, data handling becomes even more challenging. It’s always best practice to avoid using sensitive information with any form of generative AI.

By keeping these limitations in mind, generative AI has the potential to streamline processes and revolutionize the way we work – just as technology has always done in the past.

10 generative AI uses for A/B testing

In the A/B testing world, we are very interested in how one can harness these technological breakthroughs for experimentation. We are brainstorming a few approaches to re-imagine the process of revolutionizing digital customer experiences to ultimately save time and resources.

Just like everyone else, we started to wonder how generative AI could impact the world of experimentation and our customers. Here are some ideas, some of them concrete and some more abstract, as to how artificial intelligence could help our industry:

DISCLAIMER: Before uploading information into any AI platform, ensure that you understand their privacy and security practices. While AI models strive to maintain a privacy standard, there’s always the risk of data breaches. Always protect your confidential information.

1. Homepage optimization

Your homepage is likely the first thing your visitors will see so optimization is key to staying ahead of your competitors. If you want a quick comparison of content on your homepage versus your competitors, you can feed this information into generative AI to give it a basis for understanding. Once your AI is loaded with information about your competitors, you can ask for a list of best practices to employ to make new tests for your own website.

2. Analyze experimentation results

Reporting and analyzing are crucial to progressing on your experimentation roadmap, but it’s also time-consuming. By collecting a summary of testing logs, generative AI can help highlight important findings, summarize your results, and potentially even suggest future steps. Ideally, you can feed your A/B test hypothesis as well as the results to show your thought process and organization. After it recognizes this specific thought process and desired results, it could aid in generating new test hypotheses or suggestions.

3. Recommend optimization barriers

Generative AI can help you prioritize your efforts and identify the most impactful barriers to your conversion rate. Uploading your nonsensitive website performance data gathered from your analytics platforms can give AI the insight it needs into your performance. Whether it suggests that you update your title tags or compress images on your homepage, AI can quickly spot where you have the biggest drop-offs to suggest areas for optimization.

4. Client reviews

User feedback is your own treasure trove of information for optimization. One of the great benefits of AI that we already see is that it can understand large amounts of data quickly and summarize it. By uploading client reviews, surveys and other consumer feedback into the database, generative AI can assist you in creating detailed summaries of your users’ pain points, preferences and levels of satisfaction. The more detailed your reviews – the better the analysis will be.

5. Chatbots

Chatbots are a popular way to communicate with website visitors. As generative AI is a large language model, it can quickly generate conversational scripts, prompts and responses to reduce your brainstorming time. You can also use AI to filter and analyze conversations that your chatbot is already having to determine if there are gaps in the conversation or ways to enhance its interaction with customers.

6. Translation

Language barriers can limit a brand that has a presence in multiple regions. Whether you need translations for your chatbot conversations, CTAs or longer form copy, generative AI can provide you with translations in real time to save you time and make your content accessible to all zones touched by your brand.

7. Google Adwords

Speed up brainstorming sessions by using generative AI to experiment with different copy variations. Based on the prompts you provide, AI can provide you with a series of ideas for targeting keywords and creating copy with a particular tone of voice to use with Google Adwords. Caution: be sure to double-check all keywords proposed to verify their intent.

8. Personalization

Personalized content can be scaled at speed by leveraging artificial intelligence to produce variations of the same messages. By customizing your copy, recommendations, product suggestions and other messages based on past user interactions and consumer demographics, you can significantly boost your digital consumer engagement.

9. Product Descriptions

Finding the best wording to describe why your product is worth purchasing may be a challenge. With generative AI, you can get more ambitious with your product descriptions by testing out different variations of copy to see which version is the most promising for your visitors.

10. Predict User Behavior

Based on historical data from your user behavior, generative AI can predict behavior that can help you to anticipate your next A/B test. Tailoring your tests according to patterns and trends in user interaction can help you conduct better experiments. It’s important to note that predictions will be limited to patterns interpreted by past customer data collected and uploaded. Using generative AI is better when it’s used as a tool to guide you in your decision-making process rather than to be the deciding force alone.

The extensive use of artificial intelligence is a new and fast-evolving subject in the tech world. If you want to leverage it in the future, you need to start familiarizing yourself with its capabilities.

Keep in mind that it’s important to verify the facts and information AI generates just as you carefully verify data before you upload. Using generative AI in conjunction with your internal experts and team resources can assist in improving ideation and efficiency. However, the quality of the output from generative AI is only as good as what you put in.

Is generative AI a source of competitive advantage in A/B testing?

The great news is that this technology is accessible to everyone – from big industry leaders like Google to start-ups with a limited budget. However, the not-so-great news is that this is available to everyone. In other words, generative AI is not necessarily a source of competitive advantage.

Technology existing by itself does not create more value for a business. Rather, it’s the people driving the technology who are creating value by leveraging it in combination with their own industry-specific knowledge, past experiences, data collection and interpretation capabilities and understanding of customer needs and pain points.

While we aren’t here to say that generative AI is a replacement for human-generated ideas, this technology can definitely be used to complement and amplify your already-existing skills.

Leveraging generative AI in A/B testing

From education to copywriting or coding – all industries are starting to see the impact that these new software developments will have. Leveraging “large language models” is becoming increasingly popular as these algorithms can generate ideas, summarize long forms of text, provide insights and even translate in real-time.

Proper experimentation and A/B testing are at the core of engaging your audience, however, these practices can take a lot of time and resources to accomplish successfully. If generative AI can offer you ways to save time and streamline your processes, it might be time to use it as your not-so-secret weapon. In today’s competitive digital environment, continually enhancing your online presence should be at the top of your mind.

Want to start optimizing your website? AB Tasty is the best-in-class experience optimization platform that empowers you to create a richer digital experience – fast. From experimentation to personalization, this solution can help you activate and engage your audience to boost your conversions.

You might also like...

See all

Article

6min read

Minimal Detectable Effect: The Essential Ally for Your A/B Tests

Hubert Wassner

May 6, 2025

Article

2min read

How We Made AB Tasty’s Feature Experimentation SDK OpenFeature-Compatible — And Even Easier to Adopt

Julie Dumont

Apr 30, 2025

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Subscribe to
our Newsletter

Article

Jun 5, 2023

10min read

How Long Should You Run an A/B Test?

Anthony Brebion

One of the most popular questions when starting with experimentation is: How long should an A/B test run before you can draw conclusions from it?

Determining the ideal A/B test duration can be a challenge for most businesses. You have to factor in your business cycles, traffic flow, the sample size needed and be aware of other business campaigns.

Even if you reach your sample size in a few days… is it okay to end your test then? How long should you really wait?

In this article, we will discuss potential mishaps if your testing cycle is too short, give insights into which factors you need to consider and share advice on finding the best duration for your A/B tests.

Looking for fast statistical reliability? At AB Tasty, we provide a free A/B test duration calculator, which also has capabilities for a sample size calculator.

What happens if you end an A/B test too soon?

The underlying question is a crucial one and can be summed up as follows: At what point can you end a test that appears to be yielding results?

The answer depends on the relevance of the analysis and on the actual benefits of the test.

In fact, it’s not all that unusual to see tests yield good results during the trial phase and no longer see those results once the modifications are introduced.

In most cases, a disappointing observation of this nature comes down to an error during the trial phase: the test was ended too soon and the results at that point were misleading.

Let’s look at an example that illustrates the nature of the problem.

How long to run an A/B test

The graph above shows the change in the conversion rate of two versions of a page that were the subject of a test. The first version appears to break away and perform extremely well. The discrepancy between the two versions is gradually eroded as time goes by – two weeks after the starting point there’s hardly any observable difference at all.

This phenomenon where the results converge is a typical situation: the modification made does not have a real impact on conversion.

There is a simple explanation for the apparent outperformance at the start of the test: it’s unusual for the samples to be representative of your audience when the test starts. You need time for your samples to incorporate all internet user profiles, and therefore, all of their behaviors.

If you end the test too soon and allow your premature data to be the deciding factor, your results will quickly show discrepancies.

How to determine the duration of your A/B test

Now that the problem has been aired let’s have a look at how you can avoid falling into this trap.

The average recommended A/B testing time is 2 weeks, but you should always identify key factors relevant to your own conversion goals to determine the best length for a test that will meet your goals.

Let’s discuss several criteria you should use as a foundation to determine when you can trust the results you see in your A/B testing:

The statistical confidence level
The size of the sample
The representativeness of your sample
The test period and the device being tested

1. The statistical confidence level

All A/B testing solutions show a statistical reliability indicator that measures the probability of the difference in the results observed between each sample not being a matter of chance.

This indicator, which is calculated using the Chi-squared test, is the first indicator that should be used as a basis. It is used by statisticians to assert that a test is deemed reliable when the rate is 95% or higher. So, it is acceptable to make a mistake in 5% of cases and for the results of the two versions to be identical.

Yet, it would be a mistake to use this indicator alone as a basis for assessing the appropriate time to end a test.

For the purposes of devising the conditions necessary to assess the reliability of a test, this is not sufficient. In other words, if you have not reached this threshold then you cannot make the decision. Additionally, once this threshold has been reached, you still need to take certain precautions.

It’s also important to understand what the Chi-squared test actually is: a way of rejecting or not rejecting what is referred to as the null hypothesis.

This, when applied to A/B testing, is when you say that two versions produce identical results (therefore, there’s no difference between them).

If the conclusion of the test leads you to reject the null hypothesis then it means that there is a difference between the results.

However, the test is in no way an indication of the extent of this difference.

2. The size of the sample

There are lots of online tools that you can use to calculate the value of Chi-squared by giving, as the input parameters, the four elements necessary for its calculation (within the confines of a test with two versions).

AB Tasty can provide you with our own sample size calculator for you to find the value of Chi-squared.

By using this tool, we have taken an extreme example in order to illustrate this exact problem.

Sample size required for A/B testing

In this diagram, the Chi-squared calculation suggests that sample 2 converts better than sample 1 with a 95% confidence level. Having said that, the input values are very low indeed and there is no guarantee that if 1,000 people were tested, rather than 100, you would still have the same 1 to 3 ratio between the conversion rates.

It’s like flipping a coin. If there is a 50% probability that the coin will land heads-up or tails-up, then it’s possible to get a 70 / 30 distribution by flipping it just 10 times. It’s only when you flip the coin a very large number of times that you get close to the expected ratio of 50 / 50.

So, in order to have faith in the Chi-squared test, you are advised to use a significant sample size.

You can calculate the size of this sample before beginning the test to get an indication of the point at which it would be appropriate to look at the statistical reliability indicator. There are several tools online that you could use to calculate this sample size.

In practice, this can turn out to be difficult, as one of the parameters to be given is the % improvement expected (which is not easy to evaluate). But, it can be a good exercise to assess the pertinence of the modifications being envisaged.

Pro Tip: The lower the expected improvement rate, the greater the sample size needed to be able to detect a real difference.

If your modifications have a very low impact, then a lot of visitors will need to be tested. This serves as an argument in favor of introducing radical or disruptive modifications that would probably have a greater impact on the conversion.

img_548ab40f4fb20

3. The representativeness of your sample

If you have a lot of traffic, then getting a sufficiently large sample size is not a problem and you will be able to get a statistical reliability rate in just a few days, sometimes just two or three.

Having said that, ending a test as soon as the sample size and statistical reliability conditions have been met is no guarantee that results in a real-life situation are being reproduced.

The key point is to test for as long as you need to in order for all of your audience segments to be included.

Actually, the statistical tests operate on the premise that your samples are distributed in an identical fashion. In other words, the conversion probability is the same for all internet users.

But this is not the case: the probability varies in accordance with different factors such as the weather, the geographical location and also user preferences.

There are two very important factors that must be taken into account here: your business cycles and traffic sources.

Your business cycles

Internet users do not make a purchase as soon as they come across your site. They learn more, they compare, and their thoughts take shape. One, two or even three weeks might elapse between the time they are the subject of one of your tests and the point at which they convert.

If your purchasing cycle is three weeks long and you have only run the test for one week, then your sample will not be representative. As the tool records visits from all internet users, they may not record the conversions of those that are impacted by your test.

Therefore, you’re advised to test over at least one business cycle and ideally two.

Your traffic sources

Your sample must incorporate all of your traffic sources including emails, sponsored links and social networks. You need to make sure that no single source is over-represented in your sample.

Let’s take a concrete situation: if the email channel is a weak source of traffic but significant in terms of revenue and you carry out a test during an email campaign, then you are going to include internet users who have a stronger tendency to make a purchase in your sample.

This would no longer be a representative sample. It’s also crucial to know about major acquisition projects and, if possible, not to test during these periods.

The same goes for tests during sales or other significant promotional periods that attract atypical internet users. You will often see less marked differences in the results if you re-do the tests outside these periods.

It turns out that it’s quite difficult to make sure that your sample is representative, as you have little control over the kind of internet users who take part in your test.

Thankfully, there are two ways of overcoming this problem.

The first is to extend the duration of your test more than is necessary in order to get closer to the normal spread of your internet users.
The second is to target your tests so that you only include a specific population group in your sample. For example, you could exclude all internet users who have come to you as a result of your email campaigns from your samples, if you know that this will distort your results. You could also target only new visitors so that you do not include visitors who have reached an advanced stage in their purchasing process (AKA visitors who are likely to convert regardless of which variation they see).

4. Other elements to keep in mind

There are other elements to bear in mind in order to be confident that your trial conditions are as close as they can be to a real-life situation: timing and the device.

Conversion rates can vary massively on different days of the week and even at different times of the day. Therefore, you’re advised to run the test over complete periods.

In other words, if you launch the test on a Monday morning then it should be stopped on a Sunday evening so that a normal range of conversions is respected.

In the same way, conversion rates can vary enormously between mobiles, tablets and desktop computers. So with devices, you’re advised to test your sites or pages specifically for each device. This is easy to accomplish by using the targeting features to include or exclude the devices if your users show very different browsing and purchasing behavior patterns.

These elements should be taken into account so that you do not end your tests too soon and get led astray by a faulty analysis of the results.

They also explain why certain A/A tests carried out over a period of time that is too short, or during a period of unusual activity, can present differences in results and also differences in statistical reliability, even when you may not have made any modifications at all.

The ideal A/B test duration

Running and A/B testing requires a thorough consideration of various factors such as your personal conversion goals, statistical significance, sample size, seasonality, campaigns, traffic sources, etc. All factors deserve equal attention when determining the best practices for your business.

Just remember to be patient, even if you reach your sample size early. You may be surprised by the ending results.

As A/B testing is an iterative process, continuous experimentation and conversion rate optimization will lead to better results over time.

You might also like...

See all

Article

6min read

Minimal Detectable Effect: The Essential Ally for Your A/B Tests

Hubert Wassner

May 6, 2025

Article

2min read

How We Made AB Tasty’s Feature Experimentation SDK OpenFeature-Compatible — And Even Easier to Adopt

Julie Dumont

Apr 30, 2025

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Subscribe to
our Newsletter

Article

Jun 1, 2023

4min read

5 Mistakes to Avoid When Selecting an EOP

AB Tasty

If you’re an e-commerce company, you know better than anyone how important it is to optimize your website and have the best possible user experience.

You’ve heard about experience optimization platforms (EOPs) and how they can improve your website’s performance, enhance customer loyalty and increase order size. We know you’re excited – and so are we!

But, before you dive headfirst into the dizzying world of EOP selection, let’s go through the most common mistakes e-commerce companies make so you can avoid them!

Mistake #1: Focusing on quantity over quality

You might be tempted to select an EOP that offers a ton of features, capabilities, and integrations. After all, more is better… right? Not necessarily.

When it comes to experience optimization, quality > quantity.

It’s especially important to make sure that what the EOP provides is right for YOUR company and its goals.

It’s easy to be seduced by an EOP that boasts 50 integrations or hundreds of features. But, a platform that overwhelms you with options and doesn’t deliver real value will end up costing you precious time and resources.

Opt for the platform that offers the most effective features that help you achieve your specific goals.

Mistake #2: Not considering customer success

You’ve worked hard to hire the smartest people for the job and have full faith in your employees. They can probably figure out any problem that comes up. But, when it comes to experience optimization, things don’t always go as planned.

This is why it’s so important to choose a platform that offers excellent customer support.

Look for a platform that not only provides responsive support but also has a community of users with people you genuinely enjoy working with. A platform like this can offer strategic guidance in alignment with your goals.

Mistake #3: Overlooking scalability

We know that part of the reason why you’re looking for an EOP is to grow your customer base. One of the best ways you can do that is by selecting an EOP that can grow with you.

Not every EOP is designed to scale, so make sure you’re choosing one that accommodates your company’s growth goals and can accompany you along the way.

Look for a platform that offers scalability within its plans and can support increased traffic, as well as new features.

Mistake #4: Ignoring mobile and app optimization

More and more customers are choosing to shop from their phones, which means mobile optimization for all devices is critical.

A poorly optimized mobile experience can lead to lost sales – and we know you can’t afford that!

Choose a platform that prioritizes mobile optimization and delivers a seamless experience across all devices, whether it’s web or app.

Omnichannel is the way of the future. A poor mobile experience can leave a bad taste in a customer’s mouth.

Mistake #5: Neglecting data and analytics

Robust data and analytics help you make better business decisions and are absolutely essential for effective experience optimization.

Go for a platform that relies on solid statistical models, creates reports based on your needs as an e-commerce company, and lets you choose the KPIs that are most relevant to your business.

A platform that delivers valuable insights will enable you to make smarter, data-driven decisions, which can lead to higher ROI and revenue.

Selecting the right EOP for e-commerce

Selecting the right EOP is more crucial than ever for e-commerce companies.

We know you have a lot of options out there, big goals to achieve, and even bigger dreams of where you want to take your company next. AB Tasty is here to help guide you through the confusing, often complicated process of selecting an EOP that fits with your business now and in the future.

By choosing a company that addresses the five areas above, you can rest assured that you’re on the right track.

Looking for a solution that addresses all five of these areas? AB Tasty is the best-in-class experience optimization platform that empowers you to create a richer digital experience – fast. From experimentation to personalization, this solution can help you activate and engage your audience to boost your conversions.

You might also like...

See all

Article

6min read

Minimal Detectable Effect: The Essential Ally for Your A/B Tests

Hubert Wassner

May 6, 2025

Article

2min read

How We Made AB Tasty’s Feature Experimentation SDK OpenFeature-Compatible — And Even Easier to Adopt

Julie Dumont

Apr 30, 2025

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Subscribe to
our Newsletter

Article

May 30, 2023

8min read

Digital Consumer Engagement: Activate Your Mid-Market Audience

Emily Healy

One of the biggest challenges for mid-market businesses is keeping your website visitors engaged. At this point, you’ve grown your brand enough to have a significant amount of traffic, but are in search of the right strategy to nudge your passive visitors into paying customers.

When you have a significant number of passive visitors, enticing them to take action is crucial to growing your commercial activity, achieving your business objectives and creating loyalty.

After working diligently to get traffic on your website, how do you engage these visitors?

In this article, we will answer that burning question with some practical strategies to ultimately improve your conversions.

What are passive users?

Passive users are visitors on your website that do not take any action.

These individuals may browse through some content on your site, but they leave before impacting your conversion rates in a positive way.

They are simply just browsing your web pages instead of filling out forms, subscribing to newsletters, engaging with CTAs, or making a purchase.

How to identify passive users

To engage passive users, you have to identify them.

Website analytics tools, such as Google Analytics, can help you identify these particular users by recognizing patterns when you’re analyzing your users’ behavior.

With analytics tools, you can see:

Which pages are more likely to attract passive visitors
The number of passive visitors on your website
Which pages attract active users
The actions taken on each page
The amount of time visitors spend on each page
Where your visitors come from
And more

Note: Many analytics tools provide more metrics than you can manage. In other words, it’s easy to get lost in the sea of data and metrics. It’s important to identify key performance indicators to measure your user behavior consistently.

This information can help you understand user behavior and recognize what motivates visitors to take action if you analyze it correctly. Once you can identify these motivators, you can begin to create a strategy to employ similar tactics on other pages for consistency in your engagement.

Identifying and understanding your digital consumers’ behavior is the first step to reaching your conversion goals.

Why you need to engage your audience

Converting passive visitors into paying customers is crucial for success. A higher conversion rate means more revenue and growth opportunities.

To recap: in order to engage your audience, you will need to understand them based on the data you collect from their behavior. This collection of information will help you develop best practices and develop guidelines for your future endeavors. Not to mention, you will also see what doesn’t work.

Mastering this first step to engaging your audience will, at the same time, help you to create a more seamless and attractive digital customer experience.

Whether your goal is to convert more leads on your landing page, increase your sign-up form conversions, increase brand awareness or nudge visitors to add items to their shopping cart, you must capture your audience to make these objectives a reality.

As your company increases conversions, your brand reputation will grow in parallel. A great reputation attracts new customers and creates further brand loyalty.

Now the question is, how do you engage your users in the mid-market?

How to engage visitors with widgets

Unfortunately, maximizing your digital consumer engagement won’t happen overnight. Based on your consumer research on your website, you will need to create a plan to optimize your digital customer journey.

Offering incentives such as discounts, free trials or exclusive offers is one great way to engage passive website visitors. By doing this you are providing a reason for users to take action.

How can you achieve such engagement at a mid-market level? Leveraging widgets.

With little to no-code needed, widgets are the best tools you can use for your website optimization strategy. Deploying flexible, visually appealing, and impactful components, such as widgets, is a foolproof way to increase digital consumer engagement.

Here are a few ideas and advice on how you can implement widgets to boost your engagement:

Alert and notify with a pop-in

An exit-intent pop-in is one popular widget to set up on your page that can have a real impact on retaining your visitors before they leave. By displaying important information that your visitor might have missed, you are giving them another opportunity to benefit from your message.

Le Slip Français, a French clothing manufacturer, displayed a seasonal exit pop-in with free delivery during Valentine’s Day, leading to a 22% increase in clicks.

Leverage geotargeting banners

Having multiple regions requires you to differentiate your communication. With geo-targeted banners, you can maximize your chances of a conversion by sending the right message to the right digital consumer in the right location.

This type of widget can be used in a variety of ways such as creating a customizable banner to offer discounts or services at the closest physical store to the user, which could entice them to go to your location.

Be aware of overuse

It’s important to think of the right time and place for your information. Put yourself in the shoes of your digital audience and ask yourself what information would be enticing and what would be overwhelming.

Creating space with your messages is essential to avoid burdening visitors with too many pop-ins and banners. Even if you’re offering discounts for various reasons, be sure not to go overboard.

Test what works and what doesn’t

Every audience is unique. What works for one company may not work for the next.

This is why A/B testing your new ideas is important to find out what your audience likes. Without testing, a bias based on your personal likes and dislikes can get in the way of data-driven decision-making. Experimentation is the only way to find what works for your customers without making prior assumptions.

Want to start enticing your customers with widgets and leveraging A/B testing on your website? AB Tasty is the best-in-class experience optimization platform that empowers you to create a richer digital experience – fast. From experimentation to adding widgets, this solution can help you activate and engage your audience to boost your conversions.

Benefits of engaging digital consumers with widgets

Providing your visitors with additional opportunities to engage with your website can bring a number of benefits to your business, which include:

Drive conversions

Your visitors have already communicated their interest in your products by visiting your webpage. Now they need to be convinced that your products or service will be worth it to nudge them down the funnel.

Find out what influences your digital consumers’ decision-making with different tactics:

A countdown timer creates excitement for digital consumers. Alerting visitors that there is a timely discount helps create a sense of urgency that will help them make a decision, fast.
Adding social proof to your product pages is a popular scarcity and urgency tactic used by many marketers to help draw attention to certain products. By letting visitors know that their peers are also interested in the same product, they will feel a sense of comfort knowing they aren’t alone.

As an example, NYX cosmetics leveraged AB Tasty’s social proof banner widget on their product pages to create desire. Without any coding knowledge, NYX created this test and saw a 2x increase in their transaction rate compared to the original version.

Increase basket size

Another benefit of engaging users on your website is increasing the average basket size.

By adding widgets that propose complimentary products to your consumer’s purchases, you are actively encouraging them to add more to their basket while highlighting benefits that they may receive.

Let non-subscribers know they are eligible for a discount if they upgrade or give visibility to a free shipping discount by adding a progress bar.

Decathlon, a French sporting goods retailer, implemented a progress bar using AB Tasty’s easy-to-use widgets. By offering this special “gift” of free shipping to customers who filled the progress bar, they saw a 10% increase in the transaction rate.

Build loyalty

Building loyalty is an ever-growing challenge with the growing market saturation. Post-purchase, you should focus your attention on building customer loyalty and creating new brand ambassadors.

One strategy to increase loyalty is by implementing widgets to collect information about their experience in the form of a pop-up survey. An exit pop-in is a great way to capture your customers’ attention while the transaction process is still fresh in their minds.

Unicef, an agency of the United Nations responsible for providing humanitarian and developmental aid to children worldwide, used an iframe widget to collect information about their digital consumer’s experience quickly and saw over 300 answers in a short period of time.

unicef survey

Engaging your digital audience

When you have fewer interactive elements, there’s nothing in place to entice visitors to stay on your website. This means ultimately ending up with more passive customers.

Meanwhile, placing easy-to-use widgets is a great way to grab your customers’ attention and give them the nudge they need to engage more with your website. Getting your digital consumers to make the switch from passive to active will eventually make a positive impact on the KPIs that matter to you most.

If you’re looking to drive conversions, increase your basket size or drive loyalty – widgets may be the perfect addition to your optimization roadmap.

You might also like...

See all

Article

6min read

Minimal Detectable Effect: The Essential Ally for Your A/B Tests

Hubert Wassner

May 6, 2025

Article

2min read

How We Made AB Tasty’s Feature Experimentation SDK OpenFeature-Compatible — And Even Easier to Adopt

Julie Dumont

Apr 30, 2025

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Subscribe to
our Newsletter

Article

May 25, 2023

10min read

When to Make the Leap from Client- to Server-Side Testing

Rowan Haddad

As companies start out their experimentation journey, they may find themselves experimenting with small changes on their website such as the design of a CTA and other small changes to explore how they perform and their impact on key KPIs such as conversion rate and transaction.

However, as an experimentation program matures, companies are more likely to want to delve deeper into more sophisticated types of testing which require more expertise and a more advanced tool.

This is the point when many companies are ready to take the plunge from client- to server-side experimentation.

In this article, we will explore when is the right time to make the shift to server-side testing and discuss the importance of running these types of tests by outlining the various scenarios where server-side experiments are more appropriate.

Client- vs server-side testing

Before we go deeper into server-side testing, we will quickly point out the differences between client- and server-side tests to understand why you need both types of testing as your experimentation and optimization program evolves.

In client-side testing is where experimentation occurs on the client side through Javascript that runs in the browser. Therefore, client-side tools enable you to create variations of your pages by changing the content sent by your server to users in the web browser. The user then gets one variation of your altered content based on your targeting rules.

Put simply, all the work happens at the level of the browser thanks to Javascript. Because of this, client-side testing is usually best for surface-level changes such as layout, design and colors to measure their performance and impact on key KPIs.

Meanwhile, all the work in server-side testing happens at the server level rather than the browser. In other words, it’s your server that randomly sends a user the modified variation. As a result, the experimentation tool works on the server instead of inside your users’ browsers.

Perhaps one of the biggest benefits of client-side tests is that it’s easy to implement and no special expertise is required to run these tests on the front-end of the website.

However, because of its advanced capabilities, server-side tests require technical expertise and coding skills and so developers are usually the ones running these tests on the back-end.

What are the benefits of server-side testing?

Before we go into when you should be using server-side tests, we will outline some of their benefits in order to better understand their use cases.

Omnichannel experimentation

Client-side solutions are usually limited to devices with web browsers, whether that’s on desktop, mobile or tablet. This means you cannot run experiments on mobile apps or connected devices.

Server-side solutions, for their part, allow you to experiment across multiple channels including mobile apps which significantly widens your playing field and opens up numerous opportunities to A/B test beyond web browsers so you can cover all your bases.

Enhanced performance

Perhaps one of the biggest advantages of server-side tests is the absence of the flicker effect, which is often a major downside associated with client-side solutions.

The flicker effect occurs when the original page loads and is replaced by the test variation and is usually seen by the user.

Unlike client-side tools that require adding a Javascript tag to your pages, you don’t need to add this tag in server-side tools. That’s because experiments are rendered on the server before being pushed to the client and so all the major work occurs at the server level- it’s not noticeable on the client side.

In other words, during a server-side test, the variation is retrieved from the server and delivered to the user’s browser. As a result, no modifications take place on the front-end or browser, so there’s no flickering effect

Endless testing opportunities

Your teams have a lot more options to work with when it comes to server-side testing as it enables you to modify all aspects of your site.

As server-side tests are rendered from the back-end server, you can test more complex dynamic content unlike with client-side tests where it’s not easy to test such content and may damage the user experience if done on the client-side.

In other words, you can build tests that are much more complex that go deeper within your tech stack beyond the scope of UI or cosmetic changes to unlock a whole new world of experimentation.

With such deep experimentation capabilities, teams can thoroughly test all facets of a product to validate its functionality such as evaluating its underlying features, algorithms and back-end logic.

The next section will go over these various use cases where you should consider running server-side testing instead.

When does it make sense to move to server-side testing?

As we’ve mentioned, server-side tests are usually used to run more advanced types of tests and experiment deeply within your tech stack to thoroughly explore how a product works.

Put simply, server-side experimentation solutions allow teams to conduct more robust and secure experiments that are focused on modifying a product’s functions.

Here are some use cases where server side testing is recommended over client-side testing:

Run experiments on your mobile app

As already mentioned, one of the key advantages of server-side testing is enabling omnichannel and cross-platform experimentation.

As client-side solutions rely on Javascript and cookies, it’s not possible to use them to test on native mobile apps and you’re limited to devices that have a default web browser.

This means to run experiments on your mobile app, you will need a more advanced server-side testing solution to handle mobile app technologies, which are more complex and vastly different from web technologies.

Moreover, because server-side testing works on both web applications and mobile apps, you can run the same test serving the same variations irrespective of which channel they’re using. This allows you to compare data from each channel and optimize the user experience across all the different touch points accordingly.

Finally, if you use feature flags to conduct server-side testing on mobile apps, you can bypass the tedious and time-consuming app store approval. Put simply, feature flags enable you to turn functionality on or off remotely without redeploying code to app stores and waiting for its approval or having to wait for all changes to be ready at the same time to release your own changes.

Test your search algorithms

Server-side A/B tests are efficient for testing deeper level modifications related to the backend and the architecture of your website.

This is the case with search algorithms which require modification to your existing code. E-commerce websites usually turn to server-side testing to ensure that customers can easily browse through their website and find the product they’re looking for.

Thus, search algorithms are key in getting customers to find the product they’re looking for by providing a seamless search experience that eventually gets them to purchase.

For example, you can test what is displayed to customers in their search results, whether this should be based on pricing, popularity or reviews or whether you need to prioritize products based on what customers purchased/liked/viewed in the past.

Server-side testing allows you to create such complex scenarios and rules to provide customers with more personalized reocmmendations and optimize their search experience on your site. These are more difficult to test through client-side solutions as search pages are based on the search query and so they are rendered dynamically.

Thus, server-side testing offers more comprehensive testing and allows you to experiment with multiple algorithms by modifying the existing code.

Optimize your product recommendations

Similarly, with server-side tools, you can test multiple product recommendation algorithms typically found at the bottom of product pages to measure which types of layouts generate the most sales or increase average order value. For example, is it better to promote similar products, the most popular products or those that were recently viewed?

Such recommendations are based on a variety of criteria, like browsing history and on your own PIM (Product Information Management) system and other data sources. Server-side testing will allow you to test these multiple algorithms to uncover the best performing selection of recommended products.

Test data-sensitive pages

Server-side testing is great for testing on pages where data security is vital as it ensures that the data remains safe internally within the server without worrying about a security threat.

This is ideal to test on e-commerce payment pages or for banking applications, for example, or any other web pages or apps that contain sensitive data.

You can also use feature flags to test out a new payment method on a subset of users and see how they adapt before rolling it out to everyone else.

Determine the ideal form length

This is especially important for SaaS businesses that rely on free trial and request demo forms to gather information from their visitors. Thus, they need to determine the best length for such forms without risking drop-offs but still being able to gather all the necessary information about a prospect.

Server-side testing is the way to go in this scenario as your forms are directly linked to how your database is structured. If a field is obligatory, you will not be able to hide it using JavaScript because the form’s validation will fail server-side.

As a result, testing the length and complexity of a form that has the highest positive impact on conversion rates should be done on the server-side.

This also applies to other websites that utilize such forms from visitors such as hotel reservation or insurance sites. Note that different sectors will require different and more or less information depending on the type and purpose of the information being gathered.

Test the limit for free shipping

E-commerce businesses should be able to determine the order value over which free shipping is possible. This is important as shipping costs are one of the major causes for cart abandonment.

Therefore, since shipping cost is one of the determining factors in a customer’s purchase decision, companies should test out various cart value thresholds to find out the optimal limit for free shipping to improve transaction rates.

Since shipping costs are usually rendered dynamically from the back-end server, you will need to test them on the server-side. Any modifications made should have an impact on all of the following steps, and should be managed server-side.

Validate your features

Server-side tests allow you to conduct feature testing to validate your product features by rolling out variations of your feature to different groups of users to evaluate its performance before going for a general release.

With the help of feature flags, you can run server-side A/B tests and segment your users by directing them to one variation or the other. If anything goes wrong with a variation, you can easily roll it back by disabling the flag before it further impacts the user experience.

Finally, according to the feedback generated from these users, you can then optimize your features and release them to all your users with the confidence that it matches customers requirements.

Server-side tests and feature flags

The best way to run server-side tests is through feature flags. By decoupling deployment from release, you can conduct server-side A/B tests by rolling out new features to a small group of users. You can then measure their performance on this group of users before rolling out to everyone else.

While server-side testing requires technical expertise and coding skills, it’s not only relevant to tech teams. Often, non-technical staff team up with product teams to define experiments, which are then executed by the engineers. Once implemented, an experiment can usually be controlled, monitored, and analyzed via a dashboard.

With the right feature management solution, all teams across an organization can run server-side tests with the help of an easy-to-use dashboard without facing any technical hassles.

Client- vs server-side testing: Context matters

In the end, it’s important to note that it’s not a question of whether server-side is better than client-side. They’re both complementary approaches and whichever one a company chooses depends on which is better suited for its optimization and larger business goals.

In other words, one type of testing doesn’t replace the other. It’s a matter of looking at the type of experiment you want to run and which type is more suited in that particular context and which teams are looking to run the experiment- marketers tend to favor and use client-side testing more often while product managers and developers usually opt for server-side testing for their experimentation needs. It also depends on the resources businesses have at hand and the maturity of their optimization programs.

To ensure your website is optimized and provides a seamless user experience, having both testing techniques at hand is key to surviving in a competitive digital world.

Both types of testing are indispensable to help you build great products that will bring in maximum revenue. The key is to use both together to heighten productivity and achieve maximum impact.

You might also like...

See all

Article

6min read

Minimal Detectable Effect: The Essential Ally for Your A/B Tests

Hubert Wassner

May 6, 2025

Article

2min read

How We Made AB Tasty’s Feature Experimentation SDK OpenFeature-Compatible — And Even Easier to Adopt

Julie Dumont

Apr 30, 2025

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Subscribe to
our Newsletter

Article

May 24, 2023

3min read

AB Tasty Teams Up with Google Cloud

John Hughes

We are thrilled to announce that AB Tasty is now an official Google Cloud Partner, the next step in the strengthening of our partnership with Google.

With this partnership, our clients and partners can now benefit from the combination of AB Tasty’s powerful digital experience tools and Google’s secure and reliable platform.

Whether you are looking to engage and activate your clients or create resilient digital products that deliver exceptional customer experiences and strong business results, AB Tasty’s suite of products is now easier than ever to access.

From experimentation to personalization, server-side testing, feature flags, search and product recommendations, our solutions are designed to help you achieve your goals and take your digital experiences further.

Google Cloud Marketplace

With AB Tasty available on the Google Cloud Marketplace (GCM), product and marketing teams can easily integrate AB Tasty’s digital experience tools into their existing Google Cloud vendor relationships, without the need for time-consuming vendor certification processes.

The streamlined marketplace transactions and standardized terms also provide added peace of mind, requiring less involvement from legal and contracting teams, and enabling product teams to deploy AB Tasty solutions quickly and efficiently.

It also means businesses can be more flexible and agile if they want access to our experimentation tools to quickly set up accounts and contracts through Google. Businesses with an annual commitment to Google Cloud can easily incorporate AB Tasty solutions into their existing packages and quotes, while also gaining access to established vendor relationships with Google Cloud.

Additionally, the partnership allows businesses to take advantage of Google Cloud’s flexible billing options, as GCM automatically breaks up annual commitments into monthly charges on their Google Cloud Platform (GCP) bill. This makes it easier for businesses to manage their digital marketing budgets and adjust them according to their needs.

Finally, it enables businesses to count their AB Tasty departmental spend against their broader GCP Cloud infrastructure committed spend, helping them maximize their overall investment in Google Cloud. Partnering with Google Cloud and AB Tasty provides businesses with a more streamlined and cost-effective approach to digital marketing.

Our Strengthened Relationship with Google

Following the announcement of our integration with Google Analytics 4 (GA4) and AB Tasty being chosen at one of Google’s vendors of choice after the sunset of Google Optimize, this news takes our relationship with Google to new heights:

“As the first platform to integrate bi-directional synchronization with Google Analytics 4, and being one of the three partners highlighted by Google for the Optimize sunset, AB Tasty has established strong ties with the tech giant.” said Remi Aubert, CEO and Co-founder of AB Tasty. “Moreover, as one of the top Google Cloud partners and the largest digital native on the platform, AB Tasty is the ideal partner for navigating the Google Optimize sunset.”

As AB Tasty continues to help clients improve each touchpoint of their customer experience journey, we’re pleased to cement our ties with Google so strongly. Our work with Google fuels the growth of our software for experimentation, personalization, recommendation and intelligent search. In an increasingly competitive market, this strategic alliance provides businesses with the competitive edge they need.

You might also like...

See all

Article

6min read

Minimal Detectable Effect: The Essential Ally for Your A/B Tests

Hubert Wassner

May 6, 2025

Article

2min read

How We Made AB Tasty’s Feature Experimentation SDK OpenFeature-Compatible — And Even Easier to Adopt

Julie Dumont

Apr 30, 2025

Article

7min read

Understanding the Impact of Tariffs on E-Commerce Pricing Elasticity

Edoardo Aliprandi

Apr 15, 2025

Subscribe to
our Newsletter

Article

May 22, 2023

7min read

A/B Test Hypothesis Definition, Tips and Best Practices

Emily Healy

Incomplete, irrelevant or poorly formulated A/B test hypotheses are at the root of many neutral or negative tests.

Often we imagine that doing A/B tests to improve your e-commerce site’s performance means quickly changing the color of the “add to cart” button will lead to a drastic increase in your conversion rate, for example. However, A/B testing is not always so simple.

Unfortunately, implementing random changes to your pages won’t always significantly improve your results – there should be a reason behind your web experiments.

This brings us to the following question: how do you know which elements to experiment with and how can you create an effective AB test hypothesis?

Determine the problem and the hypothesis

Far too few people question the true origins of the success (or failure) of the changes they put in place to improve their conversion rate.

However, it’s important to know how to determine both the problem and the hypothesis that will allow you to obtain the best results.

Instead of searching for a quick “DIY” solution, it’s often more valuable in the long term to take a step back and do two things:

Identify the real problem – What is the source of your poor performance? Is it a high bounce rate on your order confirmation page, too many single-page sessions, a low-performing checkout CTA or something more complex?
Establish a hypothesis – This could show the root of the problem. For example, a great hypothesis for A/B testing could be: “Our customers do not immediately understand the characteristics of our products when they read the pages on our e-commerce site. Making the information more visible will increase the clicks on the “add-to-cart” button.”

The second step may seem very difficult because it requires a capacity for introspection and a critical look at the existing site. Nevertheless, it’s crucial for anyone who wants to see their KPIs improve drastically.

If you’re feeling a bit uncomfortable with this type of uncertainty around creating an effective hypothesis, know that you’ve come to the right place.

What is an A/B test hypothesis?

Technically speaking, the word hypothesis has a very simple definition:

“A proposal that seeks to provide a plausible explanation of a set of facts and which must be controlled against experience or verified in its consequences.”

The first interesting point to notice in this definition is “the set of facts to be explained.” In A/B testing, a hypothesis must always start with a clearly identified problem.

A/B tests should not be done randomly, or you risk wasting time.

Let’s talk about how to identify the problem:

Web analytics data – While this data does not explain digital consumers’ behavior exactly, it can highlight conversion problems (identifying abandoned carts, for example) and help prioritize the pages in need of testing.
Heuristic evaluation and ergonomic audit – These analyses allow you to assess the site’s user experience at a lower cost using an analysis grid.
User tests – This qualitative data is limited by the sample size but can be very rich in information that would not have been detected with quantitative methods. They often reveal problems understanding the site’s ergonomics. Even if the experience can be painful given the potential for negative remarks, it will allow you to gather qualified data with precise insights.
Eye tracking or heatmaps – These methods provide visibility into how people interact with items within a page – not between pages.
Customer feedback – As well as analyzing feedback, you can implement tools such as customer surveys or live chats to collect more information.

The tactics above will help you highlight the real problems that impact your site’s performance and save you time and money in the long run.

A/B test hypothesis formula

Initially, making an A/B test hypothesis may seem too simple. At the start, you mainly focus on one change and the effect it produces. You should always respect the following format: If I change this, it will cause that effect. For example:

Changing (the element being tested) from ___________ to ___________ will increase/decrease (the defined measurement).

At this stage, this formula is only a theoretical assumption that will need to be proven or disproven, but it will guide you in solving the problem.

An important point, however, is that the impact of the change you want to bring must always be measurable in quantifiable terms (conversion rate, bounce rate, abandonment rate, etc.).

Here are two examples of hypotheses phrased according to the formula explained above and that can apply to e-commerce:

Changing our CTA from “BUY YOUR TICKETS NOW” to “TICKETS ARE SELLING FAST – ONLY 50 LEFT!” will improve our sales on our e-commerce site.
Shortening the sign-up form by deleting optional fields such as phone and mailing address will increase the number of contacts collected.

In addition, when you think about the solution you want to implement, include the psychology of the prospect by asking yourself the following:

What psychological impact could the problem cause in the digital consumer’s mind?

For example, if your problem is a lack of clarity in the registration process which impacts the purchases, then the psychological impact could be that your prospect is confused when reading information.

With this in mind, you can begin to think concretely about the solution to correct this feeling on the client side. In this case, we can imagine that one fix could be including a progress bar that shows the different stages of registration.

Be aware: the psychological aspect should not be included when formulating your test hypothesis.

Once you have gotten the results, you should then be able to say whether it is true or false. Therefore, we can only rely on concrete and tangible assumptions.

Best practice for e-commerce optimization based on A/B hypotheses

There are many testable elements on your website. Looking into these elements and their metrics can help you create an effective test hypothesis.

We are going to give you some concrete examples of common areas to test to inspire you on your optimization journey:

HOMEPAGE

The header/main banner explaining the products/services that your site offers can increase customers’ curiosity and extend their time on the site.
A visible call-to-action appearing upon arrival will increase the chance visitors will click.
A very visible “about” section will build prospects’ trust in the brand when they arrive on the site.

PRODUCT SECTIONS

Filters save customers a lot of time by quickly showing them what they are looking for.
Highlighting a selection of the most popular products at the top of the sections is an excellent starting point for generating sales.
A “find out more” button or link under each product will encourage users to investigate.

PRODUCT PAGES

Product recommendations create a more personal experience for the user and help increase their average shopping cart
A visible “add to cart” button will catch the prospect’s attention and increase the click rate.
An “add to cart and pay” button saves the customer time, as many customers have an average of one transaction at a time.
Adding social sharing buttons is an effective way of turning the product listing into viral content.

Want to start A/B testing elements on your website? AB Tasty is the best-in-class experience optimization platform to help you convert more customers by leveraging intelligent search and recommendations to create a richer digital experience – fast. From experimentation to personalization, this solution can help you achieve the perfect digital experience with ease.

CART PAGE

The presence of logos such as “Visa certified” enhances customer confidence in the site.
A very visible button/link to “proceed to payment” greatly encourages users to click.

PAYMENT

A single page for payment reduces the exit rate.
Paying for an order without registration is very much appreciated by new prospects, who are not necessarily inclined to share their personal information when first visiting the site.
Having visibility over the entire payment process reassures consumers and will nudge them to finalize their purchase.

These best practices allow you to build your A/B test hypotheses by comparing your current site with the suggestions above and see what directly impacts conversion performance.

The goal of creating an A/B test hypothesis

The end goal of creating an A/B test hypothesis is to identify quickly what will help guarantee you the best results. Whether you have a “winning” hypothesis or not, it will still serve as a learning experience.

While defining your hypotheses can seem complex and methodical, it’s one of the most important ways for you to understand your pages’ performance and analyze the potential benefits of change.