Tag: EN

Article

May 20, 2022

8min read

Building Customer-Centric Cultures With Data

Mary Kate Cash

We invite you to also read our previous article in this series, Measuring Your Digital Impact, or the series introduction.

This is the fifth part of our series on a data-driven approach to customer-centric marketing. We met with our partner Sophie D’Souza, Vice President of Optimization at Spiralyze, and Rémi Aubert, Co-CEO & Co-Founder of AB Tasty, who talk about what a customer-centric culture really means, why it’s so important for companies to foster one, the data that enables such a culture, and the challenges and benefits involved.

How would you define a customer-centric culture?

In this data series, we’ve discussed ways to use and analyze data, metrics, and experimentation to better understand your customers, meet their needs and forge emotional connections with them. All of these things contribute to the ultimate goal of building a customer-centric vision and culture for brands.

But what defines a customer-centric culture? For Sophie, “Being customer-centric means that the customer is at the nucleus of the business – the shared collection of values, expectations, practices, and decisions that guide and inform team members are centered around the customer and the needs of the customer. And a big part of achieving that is ensuring data isn’t siloed – it’s not segmented to any one department like upper management or customer success; it permeates every aspect of the company; formal and informal systems, behaviors, business decisions and values all revolve around the customer.”

In Rémi’s opinion, customer-centricity is also very much about “Prioritizing customers above prospects in your day-to-day work. It’s easiest when you’re a small business, but it’s vital to keep this spirit while you grow. Acquiring new customers is important, but we need to remember that our existing customers have already given us their trust. It’s our job to repay them for that with positive experiences, or at least excellent customer support so we can maintain positive experiences and turn any negative experiences into positive ones to ensure we retain them.

“Above all, being customer-centric means not being mercenary: it’s the foundation of organic growth, where word-of-mouth from satisfied customers spreads and turns prospects into new customers.”

Why is the democratization of data important?

“Data democratization is essential for building a customer-centric culture,” explains Sophie. “Shared, accessible data that isn’t siloed to any one department is the best way to gain customer knowledge. Equally important is a system for gathering, storing, interpreting, and acting upon this data whenever possible.”

“Constant product and website experimentation has shed light on the value of feedback – both qualitative and quantitative – and proven its value for providing insights to the organization. Companies now understand the meaning of a data-driven culture, and the dissemination of these insights across the entire organization is what drives customer-centricity.”

Rémi notes that during the last ten years, the emphasis has been on collecting data. “But today, we’re in a phase of interpreting data in order to act upon it – and this is a mature phase, we know the right KPIs to use to bring value; tomorrow, we’ll be able to automate this data, but few organizations have attained that capability yet.”

What types of data are needed to build a customer-centric culture?

“A customer-centric culture is a data-led business model, where both qualitative and quantitative data are essential – and experimentation plays a vital role,” says Sophie. “Quantitative data gives us brilliant direction. It’s often dictated by product centricity – how customers are interacting with products, and the actions they’re taking. Qualitative data, on the other hand, is dictated by customer needs. Pairing them will provide tons of valuable information. You can gather this from many different sources: engagement and community building (e.g., encouraging customers to leave reviews, asking questions on social channels, etc.).”

“But experimentation is a core part of this, allowing us to directly measure how individuals coming to our product or our website are interacting with us and what actions should accordingly be taken.”

Rémi agrees: “Even if we understand the quantitative aspect or the qualitative aspect of our data, we won’t be able to measure the impact of customer behavior if we’re not able to change those behaviors. This is where testing and personalization come into play.

“It’s fine to identify issues, but if we can’t propose solutions and measure their efficacy, we won’t be able to adapt our culture of customer centricity to new needs. The complementarity between quantitative and qualitative data is essential. Quantitative data helps us identify problems, while qualitative data usually helps find solutions.”

Sophie’s on board: “Experimentation lets us put the customer first because we can test different solutions based on the problems we’ve identified. So rather than rolling out an idea we’ve deemed internally to be the best, experimentation lets the customer guide our actions, and in that way, we know we’re responding to real needs.”

Are there problems associated with acquiring the necessary data?

Rémi says the main problem is related to faulty data collection: “We sometimes see biased data due to incomplete data collection. Biased data is useless. Another issue we often see is that of overcollection: people collect far more data than they need, then find themselves lost in a data deluge that’s impossible to analyze and from which they can’t extract insights. The enemy of good data is too much data because you can’t orchestrate it.”

“We’ve learned that too much data equals clutter and distraction,” says Sophie. “There’s a lack of central systems in place that are efficient enough to process that much data and make it actionable. Designing systems to capture the information we need at scale and disseminate it while minimizing variance by individual interpretation is the objective for businesses today.”

What are the challenges to achieving a customer-centric culture?

Rémi tells a story about a client from a top-tier luxury jeweler. “It’s very difficult for brands like that, which have strict graphic charts and editorial guidelines, to be customer-centric, as they have little flexibility for testing. These brands are very powerful: you can’t make the slightest modification without validation by the entire brand team. So even if you know you can improve the customer journey or experience on the website, you can’t implement any changes because brand policy prohibits it. The result? Even if you have data proving a given change will improve their customer satisfaction, brand ‘integrity’ won’t allow it.”

Sophie sees a lot of progress being made, but certain barriers remain. “To be a data-driven organization, you need an open mind and an experimentation mindset, because a customer-centric culture is premised on innovation and constant change to meet customer needs. A big challenge today is that not everyone in a given organization has a data-driven mindset, although website and product experimentation and personalization are paving the way to its adoption.”

Rémi and Sophie agree that in a data-driven organization, people at every level are empowered to contribute, because it’s data, not experience, that matters. A new hire can propose a test hypothesis just as valuable as one suggested by a CEO. This kind of democratization is happening at Hanna Andersson, a children’s clothing manufacturer where all employees have a voice and are encouraged to submit test ideas. The best ones are acted upon, as in this AB Tasty case study where a small change in product image led to big impact.

How does a customer-centric culture benefit businesses/brands?

According to research by Deloitte and Touche, customer-centric businesses are 60% more profitable than their product-focused counterparts. Companies that put the customer at the center of their organization enjoy increased customer lifetime value and reduced churn.

“There’s a plethora of concrete benefits, including increased retention, customer loyalty, referrals… Operational efficiency is a major benefit, and it’s fueled by experimentation. This means that we’re not just guessing, but spending our time where it’s most valuable: on meeting real customer needs.

“Then there’s innovation. When we receive customer feedback, whether online or off, the products are iterated upon accordingly. It allows us to be more creative with solutions for customer problems rather than small iterations.”

Rémi adds that there’s also an important internal benefit to being customer-centric. “When your experiments have been successful and you’ve increased customer satisfaction, your clients are happy and so are your teams. That boosts their confidence in the product they’ve developed. It’s very rewarding.”

Sophie enthusiastically agrees: “It rallies everyone around the customer. No matter what role you play in an organization, you can see the benefit of your work.”

You might also like...

See all

Article

5min read

Click Happens: Smarter Digital Experiences for 2025

Kate Feng

Jun 13, 2025

Article

8min read

A Practical Guide to the European Accessibility Act

Thomas Hein

May 23, 2025

Article

2min read

Your Domain, Your Rules: Domain Delegation by AB Tasty

AB Tasty

May 21, 2025

Subscribe to
our Newsletter

Article

May 19, 2022

11min read

Outcome-Driven Roadmap: Paving the Way for Better Feature Experimentation

Rowan Haddad

In our previous post within our ‘Feature Experimentation’ series, we listed some essential best practices when it comes to feature testing and experimentation.

We also mentioned the importance of having a roadmap in place to help guide your team so that they’re clear on the main business objectives so that they can build efficient tests.

This roadmap will essentially be the link between these objectives with a product manager’s ideas to be able to run experiments and track the right metrics.

As a product manager in a modern software development world, you know how critical it is to quickly deliver features or products to keep up with rapidly changing demands.

You also understand how important it is to continuously run experiments to make sure that your releases are ones your customers actually want.

Having a roadmap in place will enable you and your team to plan experiments that lead you to the best outcomes in line with your company’s overall strategy and objectives.

Therefore, in this post, we will guide you on how to build and create an experimentation roadmap so you can determine what you need to optimize and figure out which experiments you need to run.

A well-structured roadmap will help you and your team prioritize which actions to take and which experimentation ideas are viable to achieve your goals and objectives.

It outlines a clear strategy for your team to follow so everyone is on the same page working towards the same objectives.

Why do you need an experimentation roadmap?

In the previous section, we quickly went over the importance of having an experimentation roadmap in place to achieve your goals. Here we will go into more detail about why you need such a roadmap.

We are focusing specifically on an ‘experiment-driven’ roadmap, which is a roadmap that illustrates the experiments a product team is aiming to run in order to achieve certain outcomes in line with business objectives.

Such a roadmap helps to point out which features to add, change or remove depending on the pain point you’re trying to solve and the data generated from experiments and any additional data gathered from other sources such as through the discovery phase process or from interviews or product usage.

Put simply, it will act as a strategic plan of action that communicates to all teams how your short-term goals align with the organization’s long-term goals.

Before embarking on your experimentation journey, having a roadmap should be the first step for various reasons, including:

Better resource planning

Creating a clear plan or roadmap before starting your experiments will help you plan your resources in advance instead of finding out you’re running out of resources halfway through an experiment.

Furthermore, as you run more experiments, you’ll be able to better allocate resources to each of these experiments to obtain more accurate results.

It’s also very likely that as you go further into your experimentation journey, you might want to create more complex experiments and tests. When you have a roadmap in place, you will be able to plan in advance for such tests.

Thus, a roadmap will enable you to see the kind of resources, such as maximum budget, you have at your disposal to plan your experiments more efficiently.

Working towards a common objective

As already mentioned, an experimentation roadmap will clarify for your teams the kind of goals and objectives you’re aiming to achieve so that every team is aligned and experiments can be built around these goals.

For example, if you have a roadmap established, you and your team will know which tests you need to run first so you can start planning these tests and the kind of resources you’ll need, as per the point above. This will end up being a huge time-saver as you have a clear idea of which experiments would fulfil company objectives.

This is a more efficient way than running random tests which might not get you the results you’re looking for and that end up being a waste of time and resources.

Greater visibility and enhanced communication

Because everyone has a clear goal to organize their workflow around, this gives greater visibility to all teams within an organization on the testing processes that are being set up.

This, in turn, allows for better communication as everyone is on the same page on what needs to be done and the information they need to do it. In short, it helps to communicate and lay out to the teams involved what the plan is to achieve objectives.

This also helps establish a culture of experimentation within the organization as different teams share their insights and learnings from the experiments.

We’ve emphasised in our previous post in our Feature Experimentation series the importance of making experimentation a team effort and an experimentation roadmap facilitates this process.

By building this roadmap, you’re basically communicating to your teams that experimentation and optimization are taken seriously and are at the forefront of the business.

In short, from this section, we can conclude that an experimentation roadmap helps align teams, the company vision and its overall goals with the kind of experiments you run.

It helps put experimentation at the core of your business and puts it into perspective within the wider business objectives. In other words, it gives teams purpose and direction in their day-to-day work.

Keep your eye on the goal: Focus on the outcomes

In Agile product development, it is often the outcomes that matter foremost while putting users at the heart of the process to deliver concrete and measurable results.

What this means is that when drawing up your roadmap, you should focus on outcomes rather than just listing the features you’re looking to build within a certain timeframe and hence forcing you down a fixed path.

When you look at outcomes, your main focus is on understanding problems and finding innovative ideas to solve these problems through experimentation.

This is the essence of an experimentation or experiment-driven roadmap. When you run experiments, you are given the opportunity to explore various solutions by testing different features within your experiments and tracking their performance.

Consequently, your main focus will be on achieving outcomes rather than having a predefined list of features you think will be the solution to your pain points. Also referred to as a feature-driven roadmap, this roadmap instead focuses on a fixed list of features to be delivered over a given time period without taking into account the wider consumer or market context.

This means that the goal of an experimentation roadmap is to help you reach specific outcomes within various time frames, outcomes that fulfil both short- and long-term goals.

Thus, you start out with a problem or pain point you’re trying to solve and the experimentation roadmap gives your team insight and a clear strategy on how to solve that problem by testing out your ideas through different experiments till you get the outcome you’re looking for.

An example of a problem-focused roadmap can be seen in the image below, which includes a ‘discover and experiment’ phase to help tackle problems in each quarter:

This approach provides more autonomy for your teams and encourages them to share their own insights and knowledge which helps to instil a true culture of experimentation.

Focusing on the outcomes will enable product teams to be more flexible by making constant changes and reiterating to get to the outcome instead of committing in advance to features that they think will deliver the desired results.

Only once you’ve validated features through experiments should you proceed with releasing a feature that will actually deliver value.

The following image depicts a more simplified and straightforward version of this approach:

How to construct your experimentation roadmap: Step-by-step

In this section, we will discuss the steps to follow when creating your roadmap to help run efficient experiments and campaigns.

1. Outline business mission and vision

We stressed the importance of aligning short-term goals of your experiments with the long-term organizational goals.

For that, you will need to clearly outline the mission and vision in order to create quarterly objectives and key results.

Afterwards, it is imperative to establish your business goals, based on the vision and mission, as these will be the basis for all the future experiments you run (or don’t run).

2. Define goals

Once you have established the main business objectives, you will now need to consider why you are running these experiments in the first place. In other words, what the goals of these experiments are, i.e the why aspect of your experiments.

Your roadmap should contain specific, detailed descriptions of the kind of changes you want to see and define what success looks like for your company so you can set goals based on this information.

Your goals will usually be tied to your brand vision and overall business objectives. In other words, it will focus on the bigger picture so that it communicates clearly to all stakeholders the desired business outcomes.

Ironing out the details will then serve as the foundation of your experimentation framework and roadmap and act as a prerequisite for tactical planning.

Setting experiential goals will then help you formulate hypotheses as a starting point for any test you run. This is especially the case if you’re planning to run A/B tests, which usually start with a hypothesis that includes an assumption and the expected outcome.

3. Define experimentation KPIs

After you’ve defined your goals, you’ll need to determine the primary and secondary KPIs you need to track in order to gauge the success of your experiments.

These KPIs will depend on the kind of results you’re looking to get from these experiments so KPIs can be anything from access rate of the product page to the number of clicks on the CTA.

Other KPIs to consider depending on the goal of your experiment may include product usage, conversion rate and sign-up rate.

4. Outline the key features of your tests

An experimentation roadmap will help you identify key features in each of your tests or experiments including:

What to test
When to test
Who to test on
Who is running the test

These are all key features to include within your roadmap to help you keep track of your experiments and the people and teams involved in the experiment process.

This is especially important when you’re part of a large organization and different teams are involved in and running their own experiments. This also ensures any knowledge, insights and learnings from experiments are visible throughout the organization.

5. Prioritize tests

You will also need to determine which tests to run first depending on priority, usually by directing your attention to the ones that will bring you the highest return on investment (ROI).

For example, you may decide to run tests depending on your existing resources-you will need to consider how easy or complicated it is to put together a certain experiment depending on existing organizational resources- or you may choose to run experiments based on the results you’re seeking and plan accordingly.

However you choose to prioritize your experiments, try to prioritize them based on your desired outcomes so you don’t waste time or resources running tests on features that don’t give you the data or insights you need to optimize your products.

Remember that being part of an agile product team, it’s important to remain flexible. This means that the roadmap you built will not be set in stone. You should always leave room for changes as often unexpected findings may occur during experiments that may force you to shift gears.

There are many techniques for prioritizing roadmap items. One approach is referred to as the MoSCoW technique, which helps product teams to prioritize features. This approach is especially useful for Agile teams which tend to favor items that carry the highest value.

The below image depicts the requirements of this framework, where the top category ‘Must have’ are features that are absolutely critical to a project all the way to ‘Will not have’, which are features that are not worth the investment.

Experimentation roadmap: Your road to better products and/or features

What we can conclude from this post is that an experimentation roadmap serves as a shared source of truth that outlines the company vision, priorities and planned features and their progress.

Most importantly, a roadmap is a way to empower teams down a path of maximum impact by giving them the information they need to get started on their experiments.

We emphasized the idea of flexibility so it’s ok to introduce changes to your roadmap as you start your experiments.

This is the whole point of experimentation; you are conducting experiments without the certainty of knowing what the solution will be so it’s a learn-as-you-go process and then adjusting your roadmap accordingly.

The only thing that should remain a staple within your roadmap is how the goals of your experiments will achieve long-term business objectives.

You might also like...

See all

Article

5min read

Click Happens: Smarter Digital Experiences for 2025

Kate Feng

Jun 13, 2025

Article

8min read

A Practical Guide to the European Accessibility Act

Thomas Hein

May 23, 2025

Article

2min read

Your Domain, Your Rules: Domain Delegation by AB Tasty

AB Tasty

May 21, 2025

Subscribe to
our Newsletter

Article

May 18, 2022

5min read

1,000 Experiments Club: A Conversation With Jeremy Epperson of ConversionAdvocates

AB Tasty

Jeremy Epperson explains why startups should leverage conversion rate optimization to maximize growth.

Jeremy Epperson is about to change the way you approach growth in your business. The chief growth officer at ConversionAdvocates, a top-ranking CRO agency specialized in data analysis, takes a data-driven approach to identify the roadblocks in testing and optimize these processes for maximum effectiveness.

Over the past decade, he has launched CRO programs for 150+ growth-stage startups, creating a repeatable proven process for conversion rate optimization that can be implemented across different verticals and business sizes. By collating the insights gained from the different businesses, notably the common mistakes, Jeremy has gathered the expertise to facilitate CRO programs and avoid the steep learning curve that comes with launches.

In his conversation with AB Tasty’s VP Marketing Marylin Montoya, Jeremy delves into the granular level of data analysis and takes on topics that most people in CRO steer clear of.

Focus on customer experience optimization to catapult business growth

In today’s digital landscape, the old-school ideology of branding and push marketing is no longer an effective strategy. These days, customers have easy access to online reviews, forums and price comparison websites to inform their purchasing decision.

Rather than trying to control the customer journey, Jeremy recommends optimizing the experience of each of its four phases, using a data-driven, scientific-testing approach. This leads to the creation of different processes and reshapes the idea of optimization: The game-changing idea is that agility (allowing companies to move, learn and improve faster) can trump exorbitant budgets, thus allowing smaller companies to take market share from giants.

Passionate about being involved with teams on the ground level to “iteratively work through the entire process,” Jeremy touts CRO as the best mechanism and catalyst for growth, which challenges teams to rethink and rebuild processes and workflows, break down silos and build communication. Jeremy says this team-building aspect is more valuable from a CRO perspective than any individual winning test.

All data is equal: the value of wins, losses and flat tests in post-test analysis

When it comes to testing, certain results are deemed more “sexy” by marketers, and others are often swept under the carpet. However, Jeremy explains the utility of all test results, be that a win, a loss or a flat result, for informing how testing should evolve.

A string of inconclusive tests means that the testing has not been focused on what is actually blocking the conversion. “If we’re not targeted in on the things that are blocking them (users) from converting then we’re not going to see big movement in the conversion rates, so that’s really important,” says Jeremy.

When test results show big changes in the conversion rate, positive or negative, this indicates that an important part of the customer experience has been impacted. While winning tests are celebrated and losing tests shied away from, Jeremy advises that in both cases, the next step should be to double down on test variations to fully resolve the problem, creating at least three variations for each of those hypotheses.

Understand your customer and remove their purchasing roadblocks

Oftentimes, marketers, especially in smaller businesses, are reluctant to spend their budget on research and insights, opting for customer acquisition strategies involving ads and content. However, according to Jeremy, investing in research to better understand the customer can bring us closer to answering one question that’s key to creating the right growth strategy for your business: Why does your customer buy or not buy your product?

Research and testing can offer 360-degree insights into customer behavior such as their buying criteria, decision-making and their buying process in order to remove any conversion roadblocks. It could be as simple as creating an FAQ page to clarify primary questions, resulting in a 23% lift in lead conversion, as Jeremy exemplified.

Jeremy explains that businesses will naturally experience growth when they focus on offering a better customer experience, eliminating customer frustrations and roadblocks, which would otherwise cause them to abandon their purchase. This customer-centric mindset will actually have a direct positive impact on revenue and growth.

Jeremy Epperson

What else can you learn from our conversation with Jeremy Epperson

How to combine research and testing in CRO to double the average validated win rate
How to encourage teams to embrace the CRO process and cooperate across verticals
The inutility of customer personas and how to replace them
How to implement CRO for the first time

About Jeremy Epperson

Jeremy Epperson, chief growth officer at ConversionAdvocates, has worked in the field of startup growth and conversion rate optimization (CRO) for 14 years, as a consultant in his own businesses as well as part of digital agencies. Jeremy is passionate about researching, building and implementing processes to generate growth and has launched CRO processes within more than 155 growth-stage startups. He also specializes in customer journey mapping, CRO maturity assessments and marketing and customer research.

About 1,000 Experiments Club

The 1,000 Experiments Club is an AB Tasty-produced podcast hosted by Marylin Montoya, VP of Marketing at AB Tasty. Join Marylin and the Marketing team as they sit down with the most knowledgeable experts in the world of experimentation to uncover their insights on what it takes to build and run successful experimentation programs.

You might also like...

See all

Article

5min read

Click Happens: Smarter Digital Experiences for 2025

Kate Feng

Jun 13, 2025

Article

8min read

A Practical Guide to the European Accessibility Act

Thomas Hein

May 23, 2025

Article

2min read

Your Domain, Your Rules: Domain Delegation by AB Tasty

AB Tasty

May 21, 2025

Subscribe to
our Newsletter

Article

May 13, 2022

14min read

Migrating from Monolith to Microservices: How do Feature Flags Fit in?

Rowan Haddad

If you’re looking to get started on building an application, you may be wondering whether to design it as a monolith or build it as a collection of microservices. In fact, this has been a long-standing point of debate for many years among application architects.

So what is the difference between these two architectures and how do you decide which one to choose and which one is best for your organization?

While monolithic architectures have been used for many years, microservices seem to be taking over as it’s becoming a key driver of digital transformation.

Indeed, in a world where speed and agility are more important than ever, you may find that switching over to the more versatile microservices approach to build applications that are quicker to create and deploy the go-to-strategy to remain competitive and to be able to continuously deliver software without delay.

In this post, we will investigate the above questions by comparing monolithic and microservices application architectures to help you in your decision. We will also explain, since moving to microservices might be a risky endeavor, how feature flags may help reduce some of that risk.

Monolithic architecture

Before we move on to the migration process, we will quickly go through the definitions of these architectures and why one may take precedence over the other.

By definition, a monolith refers to a “large block of stone”. In the same way, a monolithic application is an application made up of one piece or block built as a single indivisible unit.

In that sense, in a typical monolith application, code is handled in one single, tightly knit codebase and so data is stored in a single database.

Although this type of application is considered to be the common and traditional method to build applications, it may cause some major problems and over time may become unmanageable.

The image below illustrates the makings of this architecture, which consists of a client-side user interface, server-side application and a database. They all function as a single unit and so changes are made in the code base and require an update of the entire application.

Monolithic Architecture Diagram — Source

Below, we will list some of the difficulties and drawbacks associated with this architecture, which prompts many to move to microservices.

Drawbacks of monolithic applications

Less scalability- components cannot be scaled independently; instead, the whole application will need to be scaled, not to mention that every monolith has scalability limitations.

Reliability issues- given how the components of a monolithic application are interdependent, any minor issue may lead to the breakdown of the entire application.

Tight coupling- the components of the application are tightly coupled inside a single execution meaning that changes are harder to implement. Furthermore, all code changes affect the whole system, which could significantly slow down the development process.

Flexibility- with monolithic applications, you will need to stick to a single technology as integrating any new technology would mean rewriting the entire application which is costly and time consuming.

Complexity- as a monolithic application scales up, it becomes too complicated to understand due to how the structure is tightly connected and becomes even harder to modify that eventually it may become too difficult to manage the complex system of code within the application.

Despite its drawbacks, monoliths do offer some advantages. Firstly, monolithic applications are simple to build, test and deploy. All source code is located in one place and can be quickly understood.

This offers the added advantage when it comes to debugging. As code is one place, any issues can be easily identified to be fixed.

As already mentioned, a monolithic approach has been in existence for a long time and since it’s become such a common method for developing apps, this means that engineering and development teams have the sufficient knowledge and skills to create a monolithic program.

Nonetheless, the many disadvantages of monolithic architecture has led to many businesses shifting to microservices.

Microservices architecture

Unlike a monolithic architecture, microservices architecture divides an application into smaller, independent units and breaks down an app into its core functions-each function is called a service.

Every application process is handled by these units as a separate service and each service is self-contained; this means that in the event that a service fails, it won’t impact the other services.

In other words, the application is developed as a collection of services, where each service has its own logic and database and the ability to execute specialized functions. The following image depicts how this architecture works:

You can look at each microservice as a way to break down an application into pieces or units that are easier to manage. In the words of Martin Fowler:

“In short, the microservice architectural style [1] is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API.”

In other words, microservices architecture is a way to design software applications as suites of independently deployable services that communicate with one another through specific methods, i.e by using well-defined APIs.

Microservices: The answer to accelerated application development and time to market?

More distributed systems architectures such as microservices are increasingly replacing the more traditional monolithic architecture. One of the main reasons is that systems designed with microservices architecture are easier to modify and scale.

Due to its distributed nature, developers can develop multiple microservices simultaneously.

Since services can be deployed independently, each service is a separate codebase that can be managed by a small development team, as can be seen in the image below, which illustrates the major differences between these two architectures:

Migrating monolith app to microservices — Source

This results in shortened development cycles so releases are ready for market faster.

Microservices, as a result, are used to speed up the application development process as this type of architecture enables the rapid delivery of large, complex applications on a frequent basis.

Moreover, since these services are deployed independently, a team can update an existing service without redeploying the entire application unlike monolithic architecture. This makes continuous deployment possible.

This also makes these types of applications less risky to work with than monolithic applications. Risk mitigation, then, is one of the key drivers for adoption of microservices.

This makes it easier to add new changes or functionalities to it than to a monolithic program. This means that updating the program is more straightforward and less troublesome.

With monolithic applications, even the most minor modifications require redeployment of the entire system and so feature releases could be delayed and any bugs require a significant amount of time to be fixed.

Thus, microservices fits within an Agile workflow as using such an approach makes it easier to fix bugs and manage feature releases. You can update a service without redeploying the entire application and roll back if something goes wrong.

Not to mention that a microservices architecture addresses the scalability limitations that come with monolithic architecture. Because of its smaller, autonomous parts, each element can be scaled independently so this process is more cost- and time-efficient.

Finally, each service can be written in a different language without affecting the other services. Developers are also unrestricted by the technology they choose so they can use a variety of technologies and frameworks instead of going for a standardized one-size fit all approach.

To sum up the differences…

The table below summarizes some of the major differences between the two architectures:

	Monolithic	Microservices
Deployment	Simple deployment of the entire system	More complex as there are independent services which need to be deployed independently
Scalability	Harder to scale; the whole system needs to be redeployed	Each element can be scaled independently without downtime
Testing	Easier to test: end-to-end testing	Harder to test; each component needs to be tested individually
Flexibility	Limited to single technology	Freedom of choice of tech stack
Security	Communication with a single unit and so security is handled in one place	Large system of standalone services communicating via network protocols raises security concerns
Adoption	Traditional way to build applications so easier to implement and develop as developers possess necessary skills	Specialized skills are required
Resiliency	Single point of failure- any issue can cause a breakdown in the entire application	A failure in one microservice doesn’t affect the other services

Tread carefully with microservices

In sum, a microservices architecture offers many advantages. Nonetheless, this type of architecture may not be suited for all companies so a proper evaluation will need to be made to choose the best approach for them depending on factors such as type of product or audience.

As a result, before moving onto the migration process, it is important to proceed carefully before attempting this migration as a microservices architecture is not without its cons.

Among some of the drawbacks of microservices include:

We’ve already mentioned how monolithic architectures have been used for a long time that many engineering teams have the knowledge and experience to create a monolithic program. Meanwhile, building a microservice application without the necessary skills could be a risky endeavor as a microservice architecture is a distributed system and so you would need to configure all the modules and database connections.

Just a monolithic application could become complex with time, standalone services that make up a microservice application could also lead to high developmental and operational complexities.

Because of the distributed system that makes up this architecture, testing such an application is more difficult because of the large number of deployable parts.

Debugging and deploying these large numbers of independently deployable components are also much more complex processes. (However, should any individual microservice become unavailable, the entire application will not be disrupted).

Testing, such as integration and end-to-end testing, can become difficult due to its distributed nature. This is in contrast to monolithic apps which consist of a single unit that makes it easier to run end-to-end testing.

In the end, transitioning to a microservices architecture will ultimately depend on the pain point you’re trying to solve.

You’ve got to ask yourself whether your current (monolithic) architecture is giving you trouble and whether actually migrating to microservices will help solve your issues.

Make the transition less risky: Feature flags and microservices

With the above in mind, DevOps teams might still want to make the transition from monolithic to microservices architecture due to its compatibility with Agile development workflows, that come with with lower risks and fewer errors.

During this process, teams will look to replace the old code and roll out the new code at once, which could be very risky.

Therefore, migration to a microservice-based ecosystem could turn out to be a challenging and time consuming process, especially for businesses with large and complex systems with monolithic architecture.

This is where feature flags come into play.

Feature flags are a great asset when it comes to releases and we’re not only referring to front-end releases but also when it comes to your architectural strategy.

Feature flags give you greater control over the release process by choosing when and to whom you will release products and features by separating deployment from release.

Thus, you can turn features on or off for certain users by simply wrapping them up in a feature flag without redeploying, lessening the risk associated with the release process.

Just as feature flags enable progressive delivery of features instead of a big bang release, the same idea applies when it comes to migrating to services: it’s best to do it one piece at a time instead of all at once.

The main idea is to slowly replace functionality in the system with microservices to minimize the impact of the migration.

You would essentially be making small deployments of your microservices by deciding who sees the new service instead of going ahead with a big bang migration.

This will be preceded by analyzing your current system to identify what you can start to migrate. You can experiment with functionalities within your customer journey to start migrating and gradually direct traffic to it via feature flags away from your monolith and then slowly kill off the old code.

There are other ways to go about the migration process- which often involve a roll out of the new code all at once- but feature flags lessen the risk usually associated with microservices releases through progressive rollout instead.

Split your monolith into microservices using feature flags

The key is to move from monoliths towards microservices in incremental ways. Think of it as if you’re untangling a knot that’s been tightly woven together and feature flags as the tools that will help you to gradually unravel this knot.

Start with identifying a functionality within your monolith to migrate to a microservice architecture. It could be a core or preferably an edge functionality such as a code that sends coupon or welcome emails to users in the case of an e-commerce platform, for example.

Proceed by building a microservice version of this functionality. The code that controls the functionality within the monolith will need to be diverted to where the new functionality lives, i.e within the microservice.

Then, wrap a feature flag around this microservice with the traffic going to the old version. Once the feature flag is turned on, the microservice code is turned on so you can direct traffic to the new version to test it.

Note that you should keep the existing functionality in place in the monolith application during the transition so you can then alternate between different versions or implementations of this functionality-the one in the monolith and the one in the new microservice.

If anything goes wrong, you will be able to revert traffic back to the monolith with the original functionality. Hence, you can switch between the two functionalities until you’re satisfied that the microservice is working properly.

Using a dedicated feature flag management tool, you can test the microservices to ensure everything is working as expected. Feature flags allow you to target certain users such as percentage rollouts (similar to a canary deployment), through IP address or whatever other user attributes you set.

If no issues come up, then you can turn the flag on for more users and continue to monitor the microservice to ensure that nothing goes wrong as you increase the traffic to it.

Should anything go wrong, you can roll back by turning the flag off (i.e kill switch) and delete the old application code.

Make sure you remove the flag once you no longer need it to avoid the accumulation of technical debt.

Then, you will repeat this process with each functionality and validate them with your target users using your feature flag management tool.

Remember, the whole point is to create these microservices progressively to ensure things go smoothly and with feature flags, you further decrease the risk of the migration process.

This is based on the idea of the ‘strangler fig’ pattern.

This term is inspired by a kind of plan, where in a similar way to the plant, the pattern describes a process of wrapping an old system with a new one, the microservice architecture, using an HTTP proxy to divert calls from the old monolith functionality to the new microservice..

This would allow the new system to gradually take over more features from the old system, as can be seen in the image below, where the monolith is ‘strangled’:

Progressively decompose a monolithic application

In this scenario, a feature flag can be applied to the proxy layer to be able to switch between implementations.

Conclusion

Monoliths aren’t all bad. They’re great when you’re just getting started with a simple application and have a small team; the only issue comes from their inability to support your growing business needs.

On the other hand, microservices are a good fit for more complex and evolving applications that need to be delivered rapidly and frequently and particularly when your existing architecture has become too difficult to manage.

There is no one-size fits all approach. It will eventually depend on the unique needs of your company and the capabilities of your team.

Should you decide to take the plunge and shift to microservices architecture, make sure that you have a feature management tool where you can track the flags in your system and how your features are performing.

AB Tasty’s server-side functionality is one such tool that allows you to roll out new features to subsets of users and comes with an automatic triggered rollback in case something goes wrong during the migration process.

The most important takeaway is to carefully consider whether you really need to migrate and if so, why. You must evaluate your options and think about the kind of outcome you’re hoping to achieve and whether a microservices architecture provides the right path to this outcome.

You might also like...

See all

Article

5min read

Click Happens: Smarter Digital Experiences for 2025

Kate Feng

Jun 13, 2025

Article

8min read

A Practical Guide to the European Accessibility Act

Thomas Hein

May 23, 2025

Article

2min read

Your Domain, Your Rules: Domain Delegation by AB Tasty

AB Tasty

May 21, 2025

Subscribe to
our Newsletter

Article

May 10, 2022

9min read

What Is Customer Experience Optimization? (EXO)

AB Tasty

Once upon a time, driving digital customer experience optimization (EXO) meant having a competitive edge. You went the extra mile, you won. Nowadays, everyone is focused on EXO to the point where it’s the minimum necessary to stay in the game.

“Experience” encompasses the entire user journey across all touchpoints that a consumer encounters when interacting with your brand. Be it website, app, tablet, mobile, bot-generated or in-store, the quality of these interactions will impact your customers’ purchasing decisions and their loyalty.

Deliver solid experiences and it will shape your brand reputation and increase your conversion rates – the key is to never stop moving. Remain stagnant, and you’ll be overtaken; but if you can figure out what your customers want, find the line between what they’re looking for and what you can offer, and then evolve your interactions on an ongoing basis, you can deliver superior experiences and business success.

Here at AB Tasty we believe that optimization is the bare minimum you should be delivering. In order to stay competitive and stay ahead, the work should never stop. Establishing a continuous feedback loop through experimentation and data gathering and analysis are what it takes to maximize customer experience and keep your competitive edge.

In this article, we’ll cover:

[toc]

Why is customer experience optimization so relevant?

At the base, no matter what the product or sales channel, any business will try to satisfy their customers. Customer centricity has been around longer than we might think, but customer experience optimization really started to take flight as technology advanced and brand touchpoints and interactions diversified.

Throw in the fact that data is more readily available, collectible and collected, and suddenly the means to understand your customers better than they understand themselves is out there for the taking.

Use the data you collect to tajke your customer experience to the next level — Use the data you collect to take your customer experience to the next level (Source)

Not convinced that it really matters? Think again. PwC’s Future of CX report found that one in three consumers will walk away from a brand after just one negative experience. Furthermore, 73% of consumers nominate their experience in brand interactions as an important factor in making purchasing decisions.

Is customer experience optimization truly essential?

Think about your own experiences when shopping online. How does it feel? Which brands do you gravitate towards and which ones just don’t seem to tickle your fancy? Do they see you as an individual, a real person, or are you just another transaction to them? It only takes a moment’s pause to consider your own experiences to understand why optimizing customer experiences is not just important, but essential.

As consumers, we make decisions about where to shop, which products to buy and which ones to keep buying based on our past experience of acquiring and consuming them. What’s more: the aforementioned Future of CX report from PwC found that customers are more likely to try additional products and services from a brand they trust, and that they’re even willing to pay more, too – up to 16% more depending on the product category. It’s also less expensive to encourage repeat business (customer loyalty) than to acquire new customers, so leveraging customer experience optimization to drive long-term brand affinity and customer lifetime value will pay for itself.

The three key ingredients to supercharge your customer experience optimization

1. Optimize your user experience

Know your customer journey and dial it up!

When a customer arrives on your site – whether they’re searching for products, comparing different options or just looking to learn more about your products – there are a number of steps they’ll go through to achieve their end goal. All of these add up to a path that they’ve taken through your website, and one that presents both opportunities and pitfalls when it comes to optimizing your site and meeting your customers’ needs. The more you can understand your user journey and implement improvements while removing frictions along the purchase funnel, the better your site will perform.

Gathering data about your customers’ behavior and preferences will give you the information you need to run experiments to discern the optimal setup using A/B testing. Not sure if your CTAs have the best wording? Test them! Trying to understand the best configuration for your landing page? Run an experiment! Have doubts about whether product images should be cropped or full body? We can examine that too!

Ultimately, you’re aiming to ensure that all roads lead to an increase in conversions – and driving UX optimization on an ever-changing customer pathway is necessary to keep you ahead of the game.

Continuously optimizing your user experience is essential for staying ahead of the curve (Source)

2. Improve your personalization efforts

Know your customers and tailor to their needs!

Tailoring a digital brand interaction to the unique needs of the person behind the screen builds customer loyalty and drives repeat business. In the experience economy, you’re selling your product plus the interaction with the brand and the purchase itself alongside it. The user experience when acquiring and consuming the product is just as important as the utility it performs. Accordingly, personalizing these digital exchanges with your consumers is key to long-term customer retention.

To better understand your customers on a personal level, building a solid data foundation allows you to best understand your users, identify their needs and deliver personalized experiences that will keep your shoppers returning again and again. After all, personalization is about getting to the root of what customers have shown you that they want and delivering against that.

Use the data you gather to tailor each user’s experience on your site (Source)

As with your customer journey, responding to ever-changing desires can be challenging, so knowing your customers intimately is crucial for personalization success. Get it right and the impact is high so don’t leave any stone unturned when exploring improvement opportunities.

3. Implement server-side testing and feature management

Bring in the tech teams to expand your optimization activities!

Server-side testing is where we bring in the heavy hitters. While A/B testing can be rapidly implemented by marketing teams, server-side experimentation requires the buy-in and expertise of tech teams and developers.

Collaboration between the two groups is essential to deliver seamless customer experiences where the front-end (client-side) lures in your customers and the back-end (server-side) runs smoothly to ensure an effortless shopping experience. For instance, presenting a promotional offer (front-end) will only deliver results if the payment gateway runs glitch-free and the page loading times are fast (back-end).

Lukas Vermeer, director of experimentation at Vista, champions the value of testing both sides. “A lot of the value from experimentation…comes from two things: One is not shipping the bad stuff – a huge value point for experimentation. The other amount of value [comes from] figuring out strategically, going forward, what you should invest in.”

Lukas Vermeer, a guest speaker in the “1000 Experiments Club” podcast, champions the value of testing both sides (Source)

If your business has reached a certain level of maturity and sophistication, maximizing both client- and server-side testing will ensure that your optimization approaches are working the hardest they possibly can to deliver improved business outcomes.

How can customer experience optimization apply to different sectors?

E-commerce

Delivering digital customer experience optimization through experimentation can drive transactions, increase conversion rates and optimize user experience as you test your site in an effort to offer a smoother purchasing experience that caters to your users’ every need.

B2B

Not every website is for purchasing then and there; sometimes site visits are an initial step on a longer journey. You can drive lead generation for purchases in areas like automotive, bedroom furniture or holiday rentals by optimizing site layout, CTAs, and access to product and store information.

Travel

Offering a range of solutions, from individual products (like hotel or transport bookings) right up to comprehensive packages that take care of every step of a holiday, is a particularity of the travel industry. When bundling items together into packages, finding that pricing sweet spot is especially key. Server-side testing is particularly relevant in this field and can give you the tools to both curate your product offering and increase bookings as well.

Conclusion

When it comes to digital customer experience optimization, improving continuously is essential to your strategy; here at AB Tasty, we can’t stress that enough!

With both technology and customer attitudes evolving every second, the only way to keep the pace is by continuously adapting your company’s own optimization practices to respond to customer demands and unlock increased value and continuing loyalty.

Living and breathing such an approach means setting up your marketing, product and technical teams for smooth cross-collaboration and a shared mission and objectives. Ensuring that they’re also sharing the same experimentation and development roadmap to unlock resources and roll out improvements at the right time will keep your business on the road to success.

You might also like...

See all

Article

5min read

Click Happens: Smarter Digital Experiences for 2025

Kate Feng

Jun 13, 2025

Article

8min read

A Practical Guide to the European Accessibility Act

Thomas Hein

May 23, 2025

Article

2min read

Your Domain, Your Rules: Domain Delegation by AB Tasty

AB Tasty

May 21, 2025

Subscribe to
our Newsletter

Article

May 5, 2022

10min read

Measuring Your Digital Impact

Mary Kate Cash

If you’d like, you can review our introduction to the Customer-Centric Data Series here or read the previous installment, Creating Emotional Connections with Customers Using Data.

For the fourth installment in our series on a data-driven approach to customer-centric marketing, we got together with Filip von Reiche, CTO of Integrated Customer Experiences at Wunderman Thompson, and Gaetan Philippot, Data Scientist at AB Tasty. We discussed the pros and cons of vanity metrics, how they’re different from actionable metrics, and the roles all types of metrics play when measuring a brand’s digital impact.

Let’s begin with digital transformation. What is it, and why have companies been so focused on it over the past few years?

Digital transformation, as defined by Salesforce, is the process of using digital technologies to create new – or modify existing – business processes, culture, and customer experiences to meet changing business and market requirements. It began in the late 20th century and underwent rapid acceleration in the first two decades of the 21st century, spreading across almost all industries.

Resisting digital transformation is risky. TechTarget tells the fateful story of Blockbuster LLC, a once-global entity with video rental stores throughout the US and the world. But its presence and relevance precipitously declined from about 2005, as Netflix harnessed emerging technologies and capitalized on consumer appetite for on-demand entertainment delivered by the then newly-available streaming services.

But digital transformation can also be seen as a buzzword, says Filip, “in the sense that people think it’s something they need to do. The original impetus behind digital transformation was that brands were trying to be more competitive – in how they grew their market share, how they were perceived, and so on. And digital transformation was the engine that enabled them to achieve these things, to react faster, and to be able to measure their impact.

“Initially, it was focused on giving brands an online presence, and of course, it has achieved that, but over time, it has acquired new uses. Its latest purpose is to help brands create personalized experiences by providing them with the right content and flow which allows them to have better conversations with their customers, and that leads to more conversions.”

For Gaetan, “Part of it is imitative: people say ‘Amazon is doing a thousand experiments a year, so we have to do the same,’ but not everyone has the vast resources of Amazon, or can hope for the same results.”

But if the objective is to have personalized brand experiences, Amazon isn’t a website where people want to spend much time. “On the contrary, people go to Amazon because they can get in, buy what they want, and get out fast. It’s totally impersonal,” explains Filip. “However, the reason I spend more time with a brand is because I want a specific product or service they offer, and I expect personalization from brands I’m engaged with.”

For personalization to be successful, there must be constant validation of your perceptions before going live with any website or campaign.“More than half of all campaigns that customers perform using AB Tasty have to do with personalization or experimenting with personalization,” remarks Gaetan.“They’re the foundation on which everything else is built.”

What are the differences between vanity metrics and actionable metrics?

The use of vanity metrics varies across different verticals at different levels and from client to client. The one constant is that vanity metrics are very alluring because they provide what Filip calls “A dopamine rush that lights up your brain – and in some cases, depending on what you’re trying to achieve with your personalization, that ‘rush’ might be sufficient. But ideally, you want to know what the long-range impact will be.”

The problem is that the impact is not always easily attainable. “Let’s take real estate as an example. It’s unfortunately not as simple as the target sees a personalized message, the target clicks, the target purchases a house. Wouldn’t that be great? But in reality, the lapse of time between that initial personalization and the purchase might be 30, 60, 90 days, or even longer. In some cases, you do need a vanity metric such as page likes, favorites, shares, etc., as an indicator to tell you where things are going, but it’s always better to have a conversion metric in the background to tell you what it all really means,” explains Filip.

“This is where more in-depth analytics come into play. If you have a customer who is engaged but not converting, you need to find out what the barrier is and find a way to get around it. If you can propose a solution using personalization that meets the consumer’s needs and knocks down that barrier, great. But you always have to respect the trust the consumer has placed in you by giving you the data you need for personalization. You can’t just pop out and say “Hi! We see you’re looking at our website! That’s creepy. But you can indicate that you, as a brand, are present and listening to your consumers’ needs. It’s a delicate balance.”

Can vanity metrics be transformed into actionable metrics?

It should be emphasized that the use of a “superficial” or vanity metric is always justified when there is a notable response, whether positive or negative, because it may prompt a company to want to dig deeper and analyze further; to do so, they turn to actionable metrics for answers.

Gaetan remarks, “But it’s important to remember that not everything is actionable immediately: sometimes the payoff will be further along. The value of each type of metric varies according to industry and also according to client maturity. For example, e-commerce clients that are just starting out will test all sorts of things before they learn which key metrics are the most useful and offer the best results for their businesses.”

“The entire metric discussion needs to begin as soon as you devise your personalization or testing strategy,” says Filip. “You’ll have a goal in mind: to achieve a certain type of awareness or engagement or a certain number of conversions, etc. Everything you test that you want to use as a measure of success must align with that goal. If a vanity metric can support that goal, then it’s sufficient. If the final conversion is needed to prove my point, then we need to figure out how to get it. Sometimes that can be more complicated and involve offline integrations, but that’s usually how it works.”

What questions should companies ask to find the right metrics to track?

For Filip, a vital question concerns the scope of the project you’re undertaking. Are you measuring an entire campaign or are you breaking it down into individual parts? A high-level scope is easier to measure, meaning fewer metrics are needed, generally speaking. A detailed scope is more complex, as measuring on an individual basis raises questions of how to determine identity, how to relate conversions back to specific individuals, etc., especially when using data from a Customer Data Platform (CDP).

But the most fundamental question is: ‘Should I be testing and personalizing my experiences?’ And Filip’s answer is “Hell yes! But there are lots of different paths to take to do these things. One way is to ask a company like Wunderman Thompson to help you in doing analysis, acting as a consultant to show you what’s working and what isn’t, where there are blockages, places for improvement, etc. (Sorry for the sales pitch).

“But if you’d rather appeal to consumers on your own, from a consumer experience point of view, you need to test to discover what the best way is to have a conversation with them. How can you show them you want to help them without being intrusive? It may help companies to think of this in terms of a retail store experience by asking themselves, ‘How do I, as a customer, want to be welcomed, assisted, guided?’ Understanding this is the best way to start their personalization framework.”

How is Customer Lifetime Value measured?

Customer lifetime value (CLTV) is the profit margin a company expects to earn over the entirety of its business relationship with the average customer. A CleverTap article explains further: “Because CLTV is a financial projection, it requires a business to make informed assumptions. For example, in order to calculate CLTV, a business must estimate the value of the average sale, average number of transactions, and the duration of the business relationship with a given customer. Established businesses with historical customer data can more accurately calculate their customer lifetime value.” A bit blunt, but that’s how it works.

A visual example of calculating customer lifetime value using sale, transactions, and retention metrics – all of which can be impacted by experimentation.

Customer Lifetime Value: What is it and How to Calculate | CleverTap

Source: CleverTap

Now, where to find this precious historical customer data?

“CDPs play an essential role in measuring CLTV because they can combine data from dozens of sources to retrace a customer’s entire history of interactions with a brand, from their web and mobile experiences to their in-store and support experiences. And with this data, you can measure how long you’ve been engaging with that customer, what the value of that engagement has been, what things you offer that they’re interested in,” says Filip.

“Obviously, if a consumer has been engaging with a particular brand for a very long time, they’re going to expect a certain level of personalization from you. They’re going to expect the warm embrace and friendly conversation you have with someone you’ve known for years, not just the quick hello and small talk you’d offer to someone you just met. And it’s worth offering this level of personalization because the better you know your customers, the longer you can continue your conversation with them, which results in loyalty and retention and hopefully, referrals.”

There are techniques to maximize CLTV, including segmenting, personalization, increasing marketing channels, cross-selling, and up-selling, to mention but a few.

In today’s economy, where the markets are crowded with competitors vying for the same customers, engagement and conversion are crucial to the success of any business.

Watch for the fifth installment in our Customer-Centric Data Series in two weeks!

You might also like...

See all

Article

5min read

Click Happens: Smarter Digital Experiences for 2025

Kate Feng

Jun 13, 2025

Article

8min read

A Practical Guide to the European Accessibility Act

Thomas Hein

May 23, 2025

Article

2min read

Your Domain, Your Rules: Domain Delegation by AB Tasty

AB Tasty

May 21, 2025

Subscribe to
our Newsletter

Article

Apr 29, 2022

12min read

Feature Experimentation Best Practices

Rowan Haddad

Welcome to the first post within our new ‘Feature Experimentation’ series, where we’ll be broaching different topics related to this modern and essential practice in modern product development.

In this series, we’ll be introducing various scenarios where you can reap the benefits of feature experimentation as well as other relevant guides to help you on your experimentation journey.

In this first post, we will list and discuss some essential best practices when it comes to feature experimentation to ensure that your experiments run smoothly and successfully.

Why running experiments should be a central part of your product development process

Running experiments has become a growing, popular trend and a necessity to develop high quality features and products.

Such experiments are key in helping you uncover usage patterns and to give you insight on how your users interact with your products.

Therefore, experiments are a great way, particularly for product managers and product teams, to validate product quality and to ensure that a product aligns with business objectives.

To measure the outcome of your experiments, metrics can be used to help gauge how your customers are reacting to the new feature and whether it meets their expectations.

This means that experiments help you build and optimize your products so you can make sure that you’re releasing products that can guarantee customer satisfaction.

Experiments are also a great way to learn and prioritize resources so that product teams can focus on the most impactful areas for further iteration.

Experiments can come in different forms and these include tests such as A/B testing and multi-armed bandits.

What exactly is feature experimentation?

We talked generally about experiments in the previous section but in this series we will focus on a specific type of experimentation.

As the name suggests, feature experimentation involves feature testing or running experiments on developed or modified features with live users in order to see whether they’re performing as intended.

When we talk about feature experimentation, we’re referring to certain areas within your product that may have issues and need further optimization and improvement.

These features are ones that define the functionality of your software which make the product as a whole more effective and the overall user experience better such as a sign-up flow, a referral program, a purchase funnel or pricing offers, for example.

In other words, features refer to complete parts of your product that often involve multiple stakeholders or teams and are tied to your internal processes or business logic.

These are the features that often have a major impact, positive or negative. As a result, such features need to be tested to avoid the risks associated with blindly launching them into the wild without a clear understanding how they will perform or what their impact will be on revenue and sales or on product usage, for example.

Thus, your team can compare different variations of features with users, instead of going for a full bang release, and see which one confirms your initial hypothesis and shows a positive impact.

This way, only your best features reach your customers after looking at the data that points to the better performing variation.

Experimentation will essentially give you the data you need to do exactly that. Once the winning feature is determined, it can then be rolled out to the rest of your users with the promise of a great user experience.

Some essential best practices for running impactful experiments

As we’ve just seen, feature experimentation and experimentation in general is an indispensable tool for any modern tech and product teams.

In this section, we will discuss some general best practices when it comes to running experiments so you can achieve the best results and avoid any missteps in your experimentation journey.

Create a culture of experimentation

This should go without saying but in order to get started with experimentation, you need to build and nurture a culture of experimentation within your organization.

Some factors will come into play during this process such as your company size, your team’s workflow and capabilities and the type of industry and market you’re operating in.

What this essentially means is that you primarily need to have a clear strategy and roadmap in place so that your teams are aware of the main business objectives to build efficient tests.

We will look into building an experimentation roadmap in another post within our Feature Experimentation series so stay tuned for that!

In the meantime, what is important to note is that this roadmap will serve as the key to link business objectives with product managers’ ideas in order to execute tests and experiments and to be able to set and track the right metrics.

Furthermore, having a culture of experimentation will enable you to make data-driven decisions.

The data gathered from your experiments will allow you to determine and measure the impact of your ideas to see how they resonate with your customers, enabling you to have a clearer understanding of your target audience’s needs.

Building such a culture means you will need to have the right tools in place to help you segment your audience accordingly and tools that will also help you to collect the appropriate metrics and to analyze the results.

Just as important is having and investing in the right people, management and infrastructure to get the most out of experimentation.

However, keep in mind that building this culture of experimentation doesn’t happen overnight.

It requires time and effort but with the right mindset, you can start nurturing this kind of culture within your organization and motivating your team to get started on their roadmaps.

Make it a team effort

To embrace experimentation as part of your company culture, all the relevant teams need to be involved in product or feature testing and not just engineers and developers.

It is important to remember that a good experiment comes as a result of well-defined, shared goals and metrics by all stakeholders.

For example, as mentioned previously, experimentation is a great way for product teams to test out their ideas so everyone needs to be part of the brainstoming process and to look at experiments as a learning experience even if they failed.

In fact, sometimes, it is failed experiments that give the best insight. Any data and learnings gathered from experiments, then, will need to be shared widely among teams so everyone gets a chance to review the results and take the necessary action.

Increasing experiment visibility will allow more people within an organization to clearly see the benefits and processes underlying this practice and highlighting the success and areas of improvement boosts engagement so that they can share their own inputs thereby further instilling a culture of experimentation.

Product managers, in turn, can empower the rest of the teams to be part of the decision-making process on how to improve and optimize products so experimentation becomes a collaborative effort.

It also holds them accountable for the experiments they run so that there is a shared sense of commitment. The earlier a team is involved, the more invested they’ll be in the experiment.

Make it easy

You want to build a culture of experimentation, great, but it’s also important not to make it too complex or a time consuming process that ends up discouraging your team from running their own experiments.

Remember, experimentation should be a collaborative effort, as mentioned previously. Often, experiments may involve cross-functional teams depending on the type and the scope of the experiment you’re looking to launch.

At the same time, there shouldn’t be too much dependence among teams. We already mentioned that every team, and not just development and engineering teams, should be able to run their own experiments.

Feature flags are one way to decrease risk of running experiments by decoupling release from deployment so that all teams feel confident enough to execute experiments. We will go into further detail on that later.

Set realistic experimentation goals

The goal of running experiments is to improve your product for your customers. The results gathered should give you sufficient data to enable you to make informed decisions to optimize your products.

To be able to obtain relevant data, you will need to have a specific goal or objective that will lead you to create a viable hypothesis that you can prove (or disprove).

This is why having a roadmap, as mentioned previously, will be important to allow you to focus your tests so you can get the right data with statistically significant results.

Also, remember that it’s not always possible to test everything. This means you will need to channel your testing energy into running experiments that are relevant to your goals and objectives.

Additionally, some companies may not have a high volume of traffic or users to be able to test everything. This is especially true for feature experiments. A feature needs to receive enough traffic when running A/B tests on this feature in order to generate efficient results.

In sum, good tests or experiments should be focused enough that they give you relevant results and data to improve your products to ultimately ensure customer satisfaction.

Learn from failure

If an experiment goes wrong for any reason and you don’t obtain the results you were expecting, this doesn’t mean that the experiment was a waste of time.

Failures when it comes to experimentation can be considered as a learning experience. This encourages your team to take more risks and boosts creativity.

As a result, implementing experimentation as part of your company culture, regardless whether your experiments turn out to be successful or not, means that it becomes embedded within your team’s natural workflow.

Also, remember knowing what not to do will actually help in improving your product by preventing you from implementing ideas that didn’t perform well so that you know it’s time to move on to the next idea.

Consider the metrics

If you want to make the most out of your experiments by making data-driven decisions then you need to carefully consider the metrics you will track to help you judge whether your feature was a success such as clicks, registrations or sales.

This is an essential best practice as good, efficient experiments are built around a specific goal or metric- the key is to keep a certain focus during experiments, as already mentioned, so as not to deviate from the original goal and lose sight of why you were conducting the experiment in the first place.

This all means that you need to basically tie your experiments to specific KPIs so you can track and analyze the impact of your experiments.

Choosing the right metrics serve as a baseline for your KPIs to enable you to track the results of your experiments so you can make sound decisions.

Target the right audience

This may seem like a no-brainer but to get the results you need to improve your products, you need to choose the right audience to give you those results.

Proper targeting will allow you to see what kind of changes you need to make to your feature variations and consequently, you will be able to tailor the user experience according to the needs of a specific set of users.

This way, product managers can gain valuable insight into their target audience by observing how they interact with different variations of a feature, allowing these managers to validate theories and assumptions about a certain audience.

There are many ways you can go about segmenting your audience, which includes by region, company, device, etc. It will ultimately depend on your own unique objectives.

Remember that to target the right audience, gather the data and analyze the results, you will need to have the appropriate tools at hand depending on your business objectives and teams’ preferences.

Consider the duration of the tests

With feature experimentation, you need to run these experiments for long enough time so you can gather enough data to yield statistically significant results.

Click here to read more about statistical significance and type 1 and type 2 errors which may occur during experiments.

This is important because statistical significance indicates that the results of your experiments can be attributed to a specific cause or trend and are not just a random occurrence.

Therefore, as you start to build your roadmap, you will need to include guidelines for the scheduling and duration of your tests in order to standardize workflows for your team.

However, keep in mind that having a sufficient sample size will be more important than the amount of time an experiment runs.

Use feature flags for safer experiments

For some, the idea of testing in production seems risky and stressful.

However, there is a way to run feature experiments safely without any headaches.

Feature flags are software development tools that decouple deployment from release giving you full control over the release process. In that sense, feature flags can be considered as the foundation of a good experiment.

Feature flags allow you to safely conduct experiments by turning on features for certain users and turning them off for everyone else. If anything goes wrong during your experiment, then you can easily turn off the faulty feature until it’s fixed.

Using feature flags alongside feature experimentation will help you maintain the continuous delivery momentum that is required from modern software development while minimizing the risk of disgruntled customers due to an unstable release.

Furthermore, once you have completed your experiment and obtained the results, you can implement the necessary changes through progressive rollout to further test how these new changes perform with users.

Therefore, through progressive delivery using feature flags, you can introduce changes slowly to your users to ensure a smooth user experience before releasing them to everyone else.

Embrace feature experimentation as part of your company DNA

Some of the biggest companies have achieved their market leadership position precisely because they have embraced experimentation as part of their culture.

Therefore, feature experimentation, when done right, will allow you to make more powerful decisions based on quantifiable data straight from your users.

This means that instead of making decisions on a whim, experimentation will demonstrate what works and what doesn’t based on mathematically-sound data.

Experimentation is one of the most important capabilities offered by many feature management tools.

Our own feature flagging solution, for example, offers an experiment platform that runs A/B tests to track the business impact of feature releases.

This means that everyone has the tools and confidence to take part in experimentation.

For product managers, in particular, it gives them the power to set up, monitor and release confidently without waiting on engineering teams to run the experiments for them through a simple, easy-to-use dashboard.

Our platform focuses specifically on more advanced server-side experiments that allow you to test deeper modifications tied to your back-end architecture using feature flags where you can then measure their impact on the user experience and business.

Flagship's Report Interface for Experimentation & A/B Tests

Find out how AB Tasty can help you transition seamlessly into the world of experimentation by signing up for a free trial.

You might also like...

See all

Article

5min read

Click Happens: Smarter Digital Experiences for 2025

Kate Feng

Jun 13, 2025

Article

8min read

A Practical Guide to the European Accessibility Act

Thomas Hein

May 23, 2025

Article

2min read

Your Domain, Your Rules: Domain Delegation by AB Tasty

AB Tasty

May 21, 2025

Subscribe to
our Newsletter

Article

Apr 22, 2022

9min read

How to Create a Modern Data Foundation for Experimentation

AB Tasty

Staying ahead of the game to deliver seamless brand experiences for your customers is crucial in today’s experience economy. Today we’ll dip our toe into the “how” by looking at the underlying foundation upon which all of your experiences, optimization and experimentation efforts will be built: data.

Data is the foundation experimentation is built on (Source)

Data is the technology that can power the experiences you build for your customers by first understanding what they want and how it’ll best serve your business to deliver this. It’s the special sauce that helps connect the dots between your interpretation of existing information and trends, and the outcomes that you hypothesize will address customer needs (and grow revenue).

If you’ve ever wondered whether the benefits of a special offer are sufficiently enticing for your customer or why you have so many page hits and so few purchases, then you’ve asked the questions the marketing teams of your competitors are both asking and actively working to answer. Data and experimentation will help you take your website to the next level, better understand your customers’ preferences, and optimize their purchasing journey to drive stronger business outcomes.

So, the question remains: Where do you start? In the case of e-commerce, A/B testing is a great way to use data to test hypotheses and make decisions based on information rather than opinions.

A/B testing helps brands make decisions based on data (Source)

“The idea behind experimentation is that you should be testing things and proving the value of things before seriously investing in them,” says Jonny Longden, head of the conversion division at agency Journey Further. “By experimenting…you only do the things that work and so you’ve already proven [what] will deliver value.”

Knowing and understanding your data foundation is the platform upon which you’ll build your knowledge base and your experimentation roadmap. Read on to discover the key considerations to bear in mind when establishing this foundation.

Five things to consider when building your data foundation

Know what data you’re collecting and why
Knowing what you’re dealing with when it comes to slicing and dicing your data also requires that you understand the basic types and properties of the information to which you have access. Firstly, let’s look at the different types of data:
- First-party data is collected directly from customers, site visitors and followers, making it specific to your products, consumers and operations.
- Second-party data is collected by a secondary party outside of your company or your customers. It’s usually obtained through data-sharing agreements between companies willing to collaborate.
- Third-party data is collected by entirely separate organizations with no consideration for your market or customers; however, it does allow you to draw on increased data points to broaden general understanding.
Data also has different properties or defining characteristics: demographic data tells you who, behavioral data tells you how, transactional data tells you what, and psychographic data tells you why. Want to learn more? Download our e-book, “The Ultimate Personalization Guide”!

‎‎ㅤ‎‎

Gathering and collating a mix of this data will then allow you to segment your audience and flesh out a picture of who your customers are and how to meet their needs, joining the dots between customer behavior and preferences, website UX and the buyer journey.

Chad Sanderson, head of product – data platform at Convoy, recommends making metrics your allies to ensure data collection and analysis are synchronized. Knowing what your business leaders care about, and which metrics will move the business forward, will ensure that your data foundation is relevant and set up for success.

‎ㅤ
Invest in your data infrastructure
Data is everywhere, in its myriad of forms and gathered from a multitude of sources. Even so, if you’re going to make use of it, you need a robust system for gathering, storing and analyzing it in order to best put it to work. Start by understanding how much first-party data you have the capacity to gather by evaluating your current digital traffic levels. How many people are visiting your site or your app? You can get this information using Google Analytics or a similar platform, and this will help you understand how sophisticated your data-leveraging practices can be and identify gaps where you might need to source supplementary data (second- and third-party).
Next, you’ll need to evaluate your infrastructure. Companies that are further on their data analytics journey will invest in customer data platforms (CDPs) that allow them to collect and analyze data – gathered from a variety of sources and consolidated into a central database – at a more granular level. Stitching together this data via a CDP helps you bring all the pieces together to form a complete picture of your customers and identify any gaps. This is a critical step before you leap into action. Chad Sanderson concurs. “[Start] with the business and what the business needs,” he advises. “Tailoring your… solution to that – whatever that is – is going to be a lot more effective.”‎
ㅤ
Get consent to build consumer trust
Data security is rightly of foremost concern to consumers. The very users from whom you want to gather that first-party data want to ensure that their private information remains secure. Getting their consent and being transparent about the inherent benefit to them if they agree to your request – be it through giveaways, exclusive offers, additional information or services – will give you the best chance of success. Demonstrating that you adhere to, and take seriously, various data compliance laws (such as GDPR) and good governance will also build trust in your brand and give you the opportunity to make it worth their while through improved UX and personalized experiences.

Build trust in your brand by respecting your users’ private information (Source)

‎ㅤ
Collect and discover insights to upgrade your customer strategy
We’ve already covered the fact that data is everywhere. As Chad Sanderson highlighted above, identifying immediate business needs and priorities – as well as focusing on quick wins and low-lift changes that can have a quick and high-level impact – can help you navigate through this minefield. It’s best to think of this section as a four-step process:
ㅤㅤ• Collect data as it flows into your CDP
ㅤㅤ• Transform or calibrate your data so that it can be compared in a
ㅤ ㅤlogical manner
ㅤㅤ• Analyze the data by grouping and categorizing it according to
ㅤ ㅤthe customer segments you’ve identified and benchmarking
ㅤ ㅤagainst business priorities
ㅤㅤ• Activate your insights by pushing the learnings back into
ㅤ ㅤyour platforms and/or your experimentation roadmap and really
ㅤ ㅤput this data to work
ㅤ
Turn your data into actions
It’s crunch time (no pun about numbers intended)! We’ve examined the different types of data and where to source them, how to be responsible with data collection and how to set up the infrastructure needed to consolidate data and generate insights. We’ve also covered the need to understand business priorities and core strategy to drive data collection, analysis and activation in the same direction. Now we need to put that data and those insights to work.
In the experience economy, where constant evolution is the name of the game, innovation and optimization are the key drivers of experimentation. Taking the data foundation that you’ve built and using it to fuel and nourish your experimentation roadmap will ensure that none of the hard work of your tech, marketing and product teams is in vain. Testing allows you to evaluate alternatives in real time and make data-driven decisions about website UX. It also ensures that business metrics are never far from reach, where conversion and revenue growth take center stage.
Use the data you’ve gathered to fuel your experimentation roadmap (Source)

Invest in a solid data foundation to maximize and scale

At AB Tasty, we apply the Bayesian approach to interpreting data and test results because in A/B testing, this method not only shows whether there is a difference between the tested options but also goes beyond that by calculating a measure of that difference. Being able to identify what that variance is allows you to best understand what you will gain by adopting a permanent change.

Collecting and analyzing data, and then leveraging the insights that you glean, are key to unlocking the next level of experience optimization for your customers and your business. An experimentation roadmap grounded in real-time responsiveness and long-term, server-side improvements will have a solid data foundation approach at its core, where understanding who you want to target and how to act drives success. Furthermore, if you invest in your data foundation – and the five core drivers we’ve explored above – you’ll be equipped to scale your experimentation and allow optimization to become a key business maximizer.

You might also like...

See all

Article

5min read

Click Happens: Smarter Digital Experiences for 2025

Kate Feng

Jun 13, 2025

Article

8min read

A Practical Guide to the European Accessibility Act

Thomas Hein

May 23, 2025

Article

2min read

Your Domain, Your Rules: Domain Delegation by AB Tasty

AB Tasty

May 21, 2025

Subscribe to
our Newsletter

Article

Apr 21, 2022

8min read

Creating Emotional Connections With Customers Using Data

Mary Kate Cash

Be sure to check out the series introduction, part 1 with Zion & Zion, and part 2 with Cro Metrics if you haven’t read them yet.

For the third blog in our series on a data-driven approach to customer-centric marketing, we talked with our partner Matt Wright, Director of Behavioral Science at Widerfunnel, and Alex Anquetil, Manager of North America Customer Success at AB Tasty, who discuss what emotional connection means in a marketing context, why it’s critical for brands to forge emotional connections with their customers, and how data can be used to both build and measure the efficacy of these connections.

What do we mean when we talk about creating an “emotional connection” in a marketing context?

Simply put, emotions are the driving force behind every purchase. People don’t buy from a given brand because they need a product they could easily find elsewhere, but because they feel an affinity, a sense of trust, well-being, or inclusion with or loyalty to that brand.

In such a crowded market, forging deep emotional connections with customers is essential for marketers to attract and retain customers today. Marketers can’t merely “appeal to emotions,” but need to understand their behaviors and motivations and ensure that their missions and messages align with customers’ emotions and needs.

Matt asks to reframe the question, “What’s the role of emotional decision-making in marketing? People build mental models around their emotions, experiences, and cultural associations. They think of some as ‘good’ or ‘bad’… they tie emotion to them. The key for marketers is to understand which emotions resonate with which group of people. And this is where A/B testing can help you find clues as to what works and what doesn’t. Creating strong emotional connections is paramount, and through experimentation, you can create them all throughout your sales funnel.”

“Our brains have limited bandwidth,” remarks Alex, “so we tend to save our resources for the important things. When we make a simple purchase, we take shortcuts. We grab what’s available from the wheel of our basic emotions – happiness, anger, surprise – to enable us to make quick decisions. If brands can leverage these emotions, whether positive or negative, and align their sales tactics to them, they can create frictionless experiences. The fact that every purchase is emotional is the reason why we don’t have ‘one perfect user interface,’ or ‘one ideal sales funnel:’ every brand, product, and user is different.”

Matt says, “That’s a great analogy. Usability is the foundation, but you need to build upon it. Even if your UI is ugly, in the right circumstances, it will convert. For example, if your website is for a charity, people don’t want you to spend your money on making it look beautiful. They want the money to go to the cause – so they may negatively judge you if you have a digital masterpiece for a website. But if you’re designing for a chic brand, people want it to look and feel exclusive. This is what A/B testing teaches us: it’s not about win or lose, it’s about gathering insights, which I think is often overlooked at a base level of experimentation.”

Why are emotional connections with customers so important for brands?

For Matt, emotion is especially important for positioning. “It’s not something people typically do experimentation around – I wish they did – because the data you can glean from testing things like value propositions or copywriting is extremely valuable for successfully positioning a product. Also, as customers move through their journeys, they’re going to have different emotions at different moments, including doubts, so give them signals to reassure them they’ve made the right decisions. By doing that, you’ll strengthen their loyalty to you.”

Alex thinks that first impressions matter, and if you don’t connect on the first day, you may not get another shot. “People look for meaning in what they buy, even when it’s something as banal as a pack of batteries. Utilitarian products can have ‘the right’ signals attached to them (think of the Energizer bunny, and the tradition and reliability attached to it). No one wants to buy products that have negative connotations. When it comes to clothing or luxury items, these are 100% emotional, and it’s essential for marketers to confer the correct image and status by selling to the right groups (because, of course, there are in-groups and out-groups by the brand’s standards) and by attaching the right emotions and motivators specific to each brand and product.

Should brands create different types of emotional connections for different audiences?

Again, Matt has a preliminary question to reposition how we approach the subject: “Is it worth it to build multiple experiences? The best way to decide is to start small then go deeper, and keep testing until the data leads you to a value proposition. If the data shows you it’s worth it, then build different approaches, yes.”

But Alex, who’s familiar with both the French and US markets, says yes right away. “When looking at short term and long term outcomes, I think there have to be different types of emotional connections for different cultural or geographical audiences. The question is, do you want the emotions to serve sales or marketing at all costs? In other words, do you want your value proposition to associate your brand with specific emotions? When brands expand to new markets, they may require different approaches. For example, certain French luxury brands sell product collections only in France and entirely different ones in the US. With perfume, US customers tend to buy larger bottles, while the French buy smaller ones, due to different cultural priorities and motivators.

Examples of motivators and leverage:

Source: HBR.org, “THE NEW SCIENCE OF CUSTOMER EMOTIONS,” NOVEMBER 2015, SCOTT MAGIDS, ALAN ZORFAS, AND DANIEL LEEMON

“You can analyze your own market data to find out what your highest-value group is and what their motivators are, then push that to the market and take everyone on your journey, or you can do it the other way around, and make sales your ultimate objective.”

Matt thinks the brand will usually lead and cites the example of Netflix. “There’s a debate going on right now to decide whether, in order to keep growing, Netflix should sell ads. Now, they can probably run an A/B test and find out they’ll make more money if they do sell ads, obviously. But how will that affect their brand image in the short, medium, and long term? They might not lose money, but on an emotional level, they might lose a lot of their historical appeal.

“When dabbling with emotions, it’s not as simple as just an A/B test. When making strategic decisions, experimentation can certainly help incrementally optimize things, but it can do bigger things, including help you make key decisions, better understand your customers, innovate, take risks… Not enough people realize the power of advanced testing. Companies that use them see exponential improvements.”

Talking about experimentation tools, Matt explains: “Early on in the industry, we talked about A/B testing in pretty much only an optimization win-or-lose mindset. And it’s so much more than that. When you make this investment, it’s going to help you make decisions, not just find tiny, incremental bits of revenue for your company. There’s a resourcing problem: conversion rate improvement isn’t the only thing you can do, there’s a huge range of other things you can achieve, and teams need more than a CRO manager to effect the full capabilities. It’s a key competitive differentiator.”

How can data be used to create emotional connections in marketing?

It’s a lot harder to target audiences today due to cookie policy changes and new regulations. But as Matt says (and everyone else agrees), “First-party data will lead to strong positioning and really good ads that connect with users. Because it’s owned by brands, it’s going to be the best quality data for testing hypotheses and segmenting data so brands can offer personalized, exclusive experiences.”

Alex puts it this way: “At the end of the day, you’re still going to be tracking conversions and clicks, so you need to do the groundwork in marketing. It’s more advanced than usability testing. To test for emotions, you have to do some groundwork and some guesswork. You need to know your brand; you need to work with market research. And when you find an emotion aligned with what you want your brand to represent, you need to identify a segment of high-potential customers. Then you find the motivators you associate with that segment, thanks to qualitative research and feedback; then you need to quantify all of that to see if you’re correct. Then you push motivators, measure results, see what boosts efficiency, retention, loyalty, customer lifetime value… and discover whether you’ve got a winning proposition.”

Matt grins: “I wouldn’t call any part of that approach ‘guesswork’. You’re simply combining qualitative with quantitative to come up with better hypotheses for testing. It’s the heart of good experimentation.”

The next installment in our Customer-Centric Data Series will be out in two weeks. Don’t miss it!

You might also like...

See all

Article

5min read

Click Happens: Smarter Digital Experiences for 2025

Kate Feng

Jun 13, 2025

Article

8min read

A Practical Guide to the European Accessibility Act

Thomas Hein

May 23, 2025

Article

2min read

Your Domain, Your Rules: Domain Delegation by AB Tasty

AB Tasty

May 21, 2025

Subscribe to
our Newsletter

Article

Apr 20, 2022

13min read

The Impact of Experimentation on Cumulative Layout Shift (CLS)

John Hughes

We teamed up with our friends at Creative CX to take a look at the impact of experimentation on Core Web Vitals. Read our guest blog from Creative CX’s CTO Nelson Sousa giving you insights into how CLS can affect your Google ranking, the pros and cons of experiments server and client side, as well as organisational and technical considerations to improve your site experience through testing, personalisation and experimentation.

What are Core Web Vitals?

Core Web Vitals (CWV) are a set of three primary metrics that affect your Google search ranking. According to StarCounter, the behemoth search engine accounts for 92% of the global market share. This change has the potential to reshape the way we look at optimising our websites. As more and more competing businesses seek to outdo one another for the top spots in search results.

One notable difference with CWV is that changes are focused on the user experience. Google wants to ensure that users receive relevant content and are directed to optimised applications. The change aims to minimise items jumping around the screen or moving from their initial position. The ability to quickly and successfully interact with an interface and ensure that the largest painted element appears on the screen in a reasonable amount of time.

Core Web Vitals

What is CLS?

Let’s imagine the following scenario:

You are navigated to a website. Click on an element. It immediately moves away from its position on the page. This is a common frustration. It means you click elsewhere on a page, or on a link, which navigates you somewhere else again! Forcing you to go back and attempt to click your desired element again.

You have experienced what is known as Cumulative Layout Shift, or for short, CLS; a metric used to determine visual stability during the entire lifespan of a webpage. It is measured by score, and according to Core Web Vitals, webpages should not exceed a CLS score of 0.1

CLS within Experimentation

When working with client-side experimentation, a large percentage of A/B testing focuses on making experimentation changes on the client side (in the browser). This is a common pattern, which normally involves placing a HTML tag in your website, so that the browser can make a request to the experimentation tool’s server. Such experimentation tools have become increasingly important as Tech teams are no longer the sole entities making changes to a website.

For many, this is a great breakthrough.

It means marketing and other less technical teams access friendly user interfaces to manipulate websites without the need of a developer.It also frees up time for programmers to concentrate on other more technical aspects.

One drawback for client-side, is certain elements can be displayed to the user before the experimentation tool has had a chance to perform its changes. Once the tool finally executes and completes its changes, it may insert new elements in the same position where other elements already exist. Pushing those other elements further down the page. This downward push is an example of CLS in action.

Bear in mind that this only affects experiments above the fold. Elements initially visible on the page without the need of scrolling.

So when should you check for CLS and its impact upon the application? The answer is up for debate. Some companies begin to consider it during the design phase, while others during the User Acceptance Testing phase. No matter what your approach is, however, it should always be considered before publishing an experiment live to your customer base.

Common CLS culprits

According to Google’s article on optimising CLS, the most common causes of CLS are:

Images without dimensions
Ads, embeds, and iframes without dimensions
Dynamically injected content
Web Fonts causing FOIT/FOUT
Actions waiting for a network response before updating DOM

Overall CLS Considerations

Team awareness and communication

Each variation change creates a unique CLS score. This score is a primary point in your prioritisation mechanism. It shapes the way you approach an idea. It also helps to determine whether or not a specific experiment will be carried out.

Including analysis from performance testing tools during your ideation and design phases can help you understand how your experiment will affect your CLS score. At Creative CX, we encourage weekly communication with our clients, and discuss CLS impact on a per-experiment basis.

Should we run experiments despite possible CLS impact?

Although in an ideal world you would look to keep the CLS score to 0, this isn’t always the case. Some experiment ideas may go over the threshold, but that doesn’t mean you cannot run the experiment.

If you have data-backed reasons to expect the experiment to generate an uplift in revenue or other metrics, the CLS impact can be ignored for the lifetime of the experiment. Don’t let the CLS score to deter you from generating ideas and making them come to life.

Constant monitoring of your web pages

Even after an experiment is live, it is vital to use performance testing tools and continuously monitor your pages to see if your experiments or changes cause unprecedented harmful effects. These tools will help you analyse your CLS impact and other key metrics such as First Contentful Paint and Time to Interactivity

Be aware of everyone’s role and impact

For the impact of experimentation on Web Core Vitals, you should be aware of two main things:

What is the impact of your provider?
What is the impact of modifications you make through this platform?

Experimentation platforms mainly impact two Web Vitals: Total Blocking Time and Speed Index. The way you use your platform, on the other hand, could potentially impact CLS and LCP (Largest Contentful Paint).

Vendors should do their best to minimize their technical footprint on TBT and Speed Index. There are best practices you should follow to keep your CLS and LCP values, without the vendor being held liable.

Here, we’ll cover both aspects:

Be aware of what’s downloaded when adding a tag to your site (TBT and Speed Index)

When you insert any snippet from an experimentation vendor onto your pages, you are basically making a network request to download a JavaScript file that will then execute a set of modifications on your page. This file is, by its nature, a moving piece: based on your usage – due to the number and nature of your experimentations, its size evolves.

The bigger the file, the more impact it can have on loading time. So, it’s important to always keep an eye on it. Especially as more stakeholders in your company will embrace experimentation and will want to run tests.

To limit the impact of experimenting on metrics such as Total Blocking Time and Speed Index, you should download strictly the minimum to run your experiment. Providers like AB Tasty make this possible using a modular approach.

Dynamic Imports

Using dynamic imports, the user only downloads what is necessary. For instance, if a user is visiting the website from a desktop, the file won’t include modules required for tests that affect mobile. If you have a campaign that targets only logged in users to your site, modifications won’t be included in the JavaScript file downloaded by anonymous users.

Every import also uses a caching policy based on its purpose. For instance, consent management or analytics modules can be cached for a long time. While campaign modules (the ones that hold your modifications) have a much shorter lifespan because you want updates you’re making to be reflected as soon as possible. Some modules can also be loaded asynchronously which has no impact on performance. For example, analytics modules used for tracking purposes.

To make it easy to monitor the impact on performance, AB Tasty also includes a tool, named “Performance Center”. The benefit of this is that you get a real time preview of your file size. It also provides on-going recommendations based on your account and campaign setup:

to stop campaigns that have been running for too long and that add unnecessarily weight to the file,
to update features on running campaigns, that have benefited from performance updates since their introduction (ex: widgets).

How are you loading your experimentation tool?

A common way to load an A/B testing platform is by inserting a script tag directly into your codebase, usually in the head tag of the HTML. This would normally require the help of a developer; therefore, some teams choose the route of using a tag manager as it is accessible by non-technical staff members.

This is certainly against best practice. Tag managers cannot guarantee when a specific tag will fire. Considering the tool will be making changes to your website, it is ideal for it to execute as soon as possible.

Normally it’s placed as high up the head tag of the HTML as possible. Right after any meta tags (as these provide metadata to the entire document), and before external libraries that deal with asynchronous tasks (e.g. tracking vendors such as ad networks). Even if some vendors provide asynchronous snippets to not block rendering, it’s better to load synchronously to avoid flickering issues, also called FOOC (Flash of Original Content).

Best Practice for flickering issues

Other best practice to solve this flickering issue include:

Make sure your solution uses vanilla JavaScript to render modifications. Some solutions still rely on the jQuery library for DOM manipulation, adding one additional network request. If you are already using jQuery on your site, make sure that your provider relies on your version rather than downloading a second version.
Optimize your code. For a solution to modify an element on your site, it must first select it. You could simplify this targeting process by adding unique ids or classes to the element. This avoids unnecessary processing to spot the right DOM element to update. For instance, rather than having to resolve “body > header > div > ul > li:first-child > a > span”, a quicker way would be to just resolve “span.first-link-in-header”.
Optimize the code auto generated by your provider.When playing around with any WYSIWYG editors, you may add several unnecessary JavaScript instructions. Quickly analyse the generated code and optimize it by rearranging it or removing needless parts.
Rely as much as possible on stylesheets. Adding a stylesheet class to apply a specific treatment is generally faster than adding the same treatment using a set of JavaScript instructions.
Ensure that your solution provides a cache mechanism for the script and relies on as many points of presence as possible (CDN)so the script can be loaded as quickly as possible, wherever your user is located.
Be aware of how you insert the script from your vendor. As performance optimization is getting more advanced, it’s easy to mess around with concepts such as async or defer, if you don’t fully understand them and their consequences.

Be wary of imported fonts

Unless you are using a Web Safe font, which many businesses can’t due to their branding, the browser needs to fetch a copy of the font so that it can be applied to the text on the website. This new font may be larger or smaller than the original font, causing a reflow of the elements. Using the CSS font-display property, alongside preloading your primary webfonts, can increase the change of a font meeting the first paint, and help specify how a font is displayed, potentially eliminating a layout shift.

Think carefully about the variation changes

When adding new HTML to the page, consider if you can replace an existing element with an element of similar size, thus minimising the layout shifts. Likewise, if you are inserting a brand-new element, do preliminary testing, to ensure that the shift is not exceeding the CLS threshold.

Technical CLS considerations

Always use size attributes for the width and height of your images, videos and other embedded items, such as advertisements, and iframes. We suggest using CSS aspect ratio properties for images specifically. Unlike older responsive practices, it will determine the size of the image before it is downloaded by the browser. The more common aspect ratios out there in the present day are 4:3 and 16:9. In other words, for every 4 units across, the screen is 3 units deep, and every 16 units across, the screen is 9 units deep, respectively.

screen size Knowing one dimension makes it possible to calculate the other. If you have an element with 1000px width, your height would be 750px. This calculation is made as follows:

height = 1000 x (3 / 4)

When rendering elements to the browser, the initial layout often determines the width of a HTML element. With the aspect ratio provided, the corresponding height can be calculated and reserved. Handy tools such as Calculate Aspect Ratio can be used to do the heavy lifting math for you.

Use CSS transform property

The CSS transform property is a CSS trigger which will not trigger any geometry changes or painting. This will allow changing the element’s size without triggering any layout shifts. Animations and transitions, when done correctly with the user’s experience in mind, are a great way to guide the user from one state to another.

Move experiment to the server-side

Experimenting with many changes at once is considered against best practice. The weight of the tags used can affect the speed of the site. It may be worth moving these changes to the server-side, so that they are brought in upon initial page load. We have seen a shift in many sectors, where security in optimal, such as banking, to experiment server-side to avoid the use of tags altogether. This way, once a testing tool completes the changes, layout shift is minimised.

Working hand in hand with developers is the key to running server-side tests such as this. It requires good synchronisation between all stockholders, from marketing to product to engineering teams. Some level of experience is necessary. Moving to server-side experiments just for the sake of performance must be properly evaluated.

Server-side testing shouldn’t be confused with Tag Management server-side implementation. Some websites that implement a client-side experimentation platform through tag managers (which is a bad idea, as described previously), may be under the impressions that they can move their experimentation tag on the server-side as well and get some of tag management server-side benefits, namely reducing the number of networks request to 3rd party vendors. If this is applicable for some tracking vendors (Goggle Analytics, Facebook conversions API…), this won’t work with experiment tags that need to apply updates on DOM elements.

Summary

The above solutions are there to give you an overview of real life scenarios. Prioritise the work to be done in your tech stack. This is the key factor in improving the site experience in general. This could include moving requests to the server, using a front-end or server-side library that better meets your needs. All the way to rethinking your CDN provider and where that are available versus where most of your users are located.

One way to start is by using a free web tool such as Lighthouse and get reports about your website. This would give you the insight to begin testing elements and features that are directly or indirectly causing low scores.

For example, if you have a large banner image that is the cause of your Largest Contentful Paint appearing long after your page begins loading, you could experiment with different background images and test different designs against one another to understand which one loads the most efficiently. Repeat this process for all the CWV metrics, and if you’re feeling brave, dive into other metrics available in the Lighthouse tools.

While much thought has gone into the exact CWV levels to strive for, it does not mean Google will take you out of their search ranking as they will still prioritise overall information and relevant content over page experience. Not all companies will be able to hit these metrics, but it certainly sets standards to aim for.

Written by Nelson Sousa, Chief Technology Officer, Creative CX

Nelson is an expert in the field of experimentation and website development with over 15 years’ experience, specialising in UX driven web design, development, and optimisation.