abtasty

Article

May 6, 2025
6min read

Minimal Detectable Effect: The Essential Ally for Your A/B Tests

Hubert Wassner

Hubert Wassner

Share on

LinkedIn Facebook Twitter

In CRO (Conversion Rate Optimization), a common dilemma is not knowing what to do with a test that shows a small and non-significant gain. 

Should we declare it a “loser” and move on? Or should we collect more data in the hope that it will reach the set significance threshold? 

Unfortunately, we often make the wrong choice, influenced by what is called the “sunk cost fallacy.” We have already put so much energy into creating this test and waited so long for the results that we don’t want to stop without getting something out of this work. 

However, CRO’s very essence is experimentation, which means accepting that some experiments will yield nothing. Yet, some of these failures could be avoided before even starting, thanks to a statistical concept: the MDE (Minimal Detectable Effect), which we will explore together.

MDE: The Minimal Detectable Threshold

In statistical testing, samples have always been valuable, perhaps even more so in surveys than in CRO. Indeed, conducting interviews to survey people is much more complex and costly than setting up an A/B test on a website. Statisticians have therefore created formulas that link the main parameters of an experiment for planning purposes:

  • The number of samples (or visitors) per variation
  • The baseline conversion rate
  • The magnitude of the effect we hope to observe

This allows us to estimate the cost of collecting samples. The problem is that, among these three parameters, only one is known: the baseline conversion rate. 

We don’t really know the number of visitors we’ll send per variation. It depends on how much time we allocate to data collection for this test, and ideally, we want it to be as short as possible. 

Finally, the conversion gain we will observe at the end of the experiment is certainly the biggest unknown, since that’s precisely what we’re trying to determine.

So, how do we proceed with so many unknowns? The solution is to estimate what we can using historical data. For the others, we create several possible scenarios:

  • The number of visitors can be estimated from past traffic, and we can make projections in weekly blocks.
  • The conversion rate can also be estimated from past data.
  • For each scenario configuration from the previous parameters, we can calculate the minimal conversion gains (MDE) needed to reach the significance threshold.

For example, with traffic of 50,000 visitors and a conversion rate of 3% (measured over 14 days), here’s what we get:

MDE Uplift
  • The horizontal axis indicates the number of days.
  • The vertical axis indicates the MDE corresponding to the number of days.

The leftmost point of the curve tells us that if we achieve a 10% conversion gain after 14 days, then this test will be a winner, as this gain can be considered significant. Typically, it will have a 95% chance of being better than the original. If we think the change we made in the variation has a chance of improving conversion by ~10% (or more), then this test is worth running, and we can hope for a significant result in 14 days.

On the other hand, if the change is minor and the expected gain is less than 10%, then 14 days will not be enough. To find out more, we move the curve’s slider to the right. This corresponds to adding days to the experiment’s duration, and we then see how the MDE evolves. Naturally, the MDE curve decreases: the more data we collect, the more sensitive the test becomes to smaller effects.

For example, by adding another week, making it a 21-day experiment, we see that the MDE drops to 8.31%. Is that sufficient? If so, we can validate the decision to create this experiment.

MDE Graph

If not, we continue to explore the curve until we find a value that matches our objective. Continuing along the curve, we see that a gain of about 5.44% would require waiting 49 days.

Minimum Detectable Uplift Graph

That’s the time needed to collect enough data to declare this gain significant. If that’s too long for your planning, you’ll probably decide to run a more ambitious test to hope for a bigger gain, or simply not do this test and use the traffic for another experiment. This will prevent you from ending up in the situation described at the beginning of this article, where you waste time and energy on an experiment doomed to fail.

From MDE to MCE

Another approach to MDE is to see it as MCE: Minimum Caring Effect. 

This doesn’t change the methodology except for the meaning you give to the definition of your test’s minimal sensitivity threshold. So far, we’ve considered it as an estimate of the effect the variation could produce. But it can also be interesting to consider the minimal sensitivity based on its operational relevance: the MCE. 

For example, imagine you can quantify the development and deployment costs of the variation and compare it to the conversion gain over a year. You could then say that an increase in the conversion rate of less than 6% would take more than a year to cover the implementation costs. So, even if you have enough traffic for a 6% gain to be significant, it may not have operational value, in which case it’s pointless to run the experiment beyond the duration corresponding to that 6%.

MDE graph

In our case, we can therefore conclude that it’s pointless to go beyond 42 days of experimentation because beyond that duration, if the measured gain isn’t significant, it means the real gain is necessarily less than 6% and thus has no operational value for you.

Conclusion

AB Tasty’s MDE calculator feature will allow you to know the sensitivity of your experimental protocol based on its duration. It’s a valuable aid when planning your test roadmap. This will allow you to make the best use of your traffic and resources.

Looking for a free and minimalistic MDE calculator to try? Check out our free Minimal Detectable Effect calculator here.

Share on

LinkedIn Facebook Twitter

Subscribe to
our Newsletter

bloc Newsletter EN

We will process and store your personal data to respond to send you communications as described in our  Privacy Policy.

Article

Apr 30, 2025
2min read

How We Made AB Tasty’s Feature Experimentation SDK OpenFeature-Compatible — And Even Easier to Adopt

Julie Dumont

Julie Dumont

Share on

LinkedIn Facebook Twitter

Why We Chose to Adopt OpenFeature

At AB Tasty, we believe a great product experience starts with smooth feature delivery and personalization. Our Feature Experimentation SDK empowers tech teams to control feature rollout and tailor interfaces to each visitor.

But in today’s complex, fast-moving ecosystems, interoperability is key. That’s where OpenFeature comes in.

What is OpenFeature?

OpenFeature is an open-source specification that defines a standardized API for feature flag management. It lets developers manage feature flags consistently across tools and platforms.

Why it matters:

  • Interoperability: A unified API across providers.
  • No vendor lock-in: Switch tools without rewriting business logic.
  • Thriving community: Backed by CNCF and designed for cloud-native development.

What We Built: The Official AB Tasty OpenFeature Provider

To ensure our SDK plays well with OpenFeature, we created an official provider:
@flagship.io/openfeature-provider-js

You can now use our feature flags in any OpenFeature-compliant setup, like this simple Node.js example:


JavaSvript
const express = require("express");
const { ABTastyProvider } = require("@flagship.io/openfeature-provider-js");
const { OpenFeature } = require("@openfeature/server-sdk");

const app = express();
app.use(express.json());

const provider = new ABTastyProvider("<ENV_ID>", "<API_KEY>");
await OpenFeature.setProviderAndWait(provider);

app.get("/item", async (req, res) => {
  const evaluationContext = {
    targetingKey: "visitor-id",
    fs_is_vip: true,
  };

  OpenFeature.setContext(evaluationContext);
  const client = OpenFeature.getClient();

  const fsEnableDiscount = await client.getBooleanValue("fs_enable_discount", false, evaluationContext);
  const fsAddToCartBtnColor = await client.getStringValue("fs_add_to_cart_btn_color", "blue", evaluationContext);

  res.json({
    product: {
      name: "AB Tasty Feature Experimentation T-shirt",
      price: 20,
      discountEnabled: fsEnableDiscount,
      btnColor: fsAddToCartBtnColor,
    },
  });
});

app.listen(3000, () => console.log("Server running on port 3000"));

Even Smarter: Our Built-In Codebase Analyzer

We’ve made onboarding even easier. Our CLI tool, which is also bundled with the AB Tasty VSCode Extension, includes a powerful codebase analyzer.

What it does:

  • Scans your codebase
  • Detects flags and usage from other providers (e.g., Optimizely, Kameleoon, etc.)
  • Identifies if you’re already using OpenFeature
  • Automatically generates corresponding flags in Flagship

Example: Already using OpenFeature with a competitor? Just plug in our CLI, and AB Tasty will detect your flags and preconfigure them for you — saving you hours of manual setup.

What This Means for You

By supporting OpenFeature, we offer:

  • Faster integration with your current stack
  • Greater flexibility in your feature flag strategy
  • Portability across cloud platforms
  • Alignment with open, modern dev practices

Quick Integration Guide

1. Install dependencies


Shell
npm install @openfeature/server-sdk
npm install @flagship.io/openfeature-provider-js

2. Initialize the provider


JavaScript
const { ABTastyProvider } = require("@flagship.io/openfeature-provider-js");
const { OpenFeature } = require("@openfeature/server-sdk");

const provider = new ABTastyProvider("<ENV_ID>", "<API_KEY>");
await OpenFeature.setProviderAndWait(provider);

3. Define the evaluation context


JavaScript
const evaluationContext = {
  targetingKey: "visitor-id",
  fs_is_vip: true,
};

OpenFeature.setContext(evaluationContext);

4. Evaluate your flags


JavaScript
const client = OpenFeature.getClient();

const fsEnableDiscount = await client.getBooleanValue(
  "fs_enable_discount",
  false,
  evaluationContext
);

const fsAddToCartBtnColor = await client.getStringValue(
  "fs_add_to_cart_btn_color",
  "blue",
  evaluationContext
);

Want to Learn More?

Check out these resources for more information about Feature Experimentation and OpenFeature.

Share on

LinkedIn Facebook Twitter